Sparse Attention Patterns
Pattern:
Full (dense)
Local window
Strided
Global tokens
BigBird (local+global+random)
Longformer
Seq len:
16
Window w:
3