Blog Post··6 min read
Positional Encodings: From Sinusoids to RoPE
Attention is permutation-invariant. Positional encodings break that symmetry. The choice of encoding method determines whether your model can generalize to longer sequences than it trained on.