Table of Contents

Feed “cat bites dog” and “dog bites cat” into a Transformer — without positional information, these two sentences are identical to the model: just the tokens “cat,” “bites,” “dog” in some order. Self-attention lets each token attend to all others, but that “fully connected” design loses the concept of sequence order entirely. Positional Encoding was introduced in the original Transformer paper as the fix, but from 2017 to today, the solution to this problem has evolved through several generations.

TL;DR

  • Sinusoidal absolute positional encoding (original Transformer): computes position vectors using sine/cosine functions, no training needed, but can’t extrapolate beyond training sequence length
  • Learnable absolute positional encoding (GPT-2, BERT): trains position vectors as parameters, some flexibility but equally unable to extrapolate
  • Relative positional encoding (T5, ALiBi): attention directly perceives relative distance between tokens, more friendly for long sequences
  • RoPE (LLaMA, Mistral, Qwen, DeepSeek, most modern LLMs): multiplies positional information into Query and Key using rotation matrices — parameter-free, naturally encodes relative distance, extendable via YaRN and similar techniques — currently the dominant approach

The Problem

Why Positional Encoding Is Needed

The self-attention computation is:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

This computation is permutation-invariant over the input token order. Shuffle the input sequence and each token’s output vector simply rearranges — the values don’t change. That’s fine for image patch classification or set problems, but in language, word order carries enormous semantic information.

Positional encoding’s task: inject position information into token representations without modifying the attention mechanism itself.

How Each Approach Works

Approach 1: Sinusoidal Absolute Positional Encoding (Vaswani et al., 2017)

The original Transformer paper’s method: for each position pos, each dimension i, compute:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

This vector is added directly to the token embedding, and the position information is carried implicitly through all subsequent computations.

Intuition: Different dimensions use sine waves of different frequencies, analogous to binary counting — low-frequency dimensions capture coarse position (is this in the first half or second half?), high-frequency dimensions capture fine position (which exact slot?).

Downside: Training only sees sequences up to a certain length. At inference time, positions beyond that length have no learned PE, and performance drops sharply.

Approach 2: Learnable Absolute Positional Encoding (GPT-2, BERT)

Instead of formulas, build a max_seq_len × d_model embedding table where each position’s vector is trained via backpropagation. BERT and GPT-2 both use this.

Advantage: The model can learn position representations suited to the task.

Disadvantages:

  1. Increases parameter count
  2. Still can’t extrapolate — no vectors beyond position 512
  3. The “relative relationship” between position 1 and position 2 isn’t explicitly modeled; the model has to learn it

Approach 3: Relative Positional Encoding

T5’s approach (Shaw et al., 2018): Instead of adding position to embeddings, directly add a relative position bias to the attention computation for each (query token, key token) pair. This makes the attention scores themselves carry relative distance information.

ALiBi (Press et al., 2021): A cleaner relative encoding — for each attention head, add a negative linear bias proportional to relative distance directly to the attention logit. No extra parameters needed; more distant tokens get a larger negative penalty (effectively decaying). ALiBi performs relatively robustly when extrapolating to longer sequences.

Approach 4: RoPE — Rotary Positional Embedding (Su et al., 2021)

RoPE is the most widely adopted positional encoding scheme today, used by LLaMA, Mistral, Qwen, DeepSeek, PaLM 2, and nearly all modern LLMs.

Core idea: Multiply positional information into Query and Key vectors, rather than adding it to token embeddings. Done via rotation matrices:

For a token at position m, rotate each pair of dimensions (q_{2i}, q_{2i+1}) of its Q vector:

[q_{2i}' ]   [cos(mθ_i)  -sin(mθ_i)] [q_{2i}  ]
[q_{2i+1}'] = [sin(mθ_i)   cos(mθ_i)] [q_{2i+1}]

where θ_i = 10000^(-2i/d_model) — a frequency design similar to sinusoidal.

Why does this work? When computing the dot product of Q at position m with K at position n:

Q_m^T · K_n = f(q, m)^T · f(k, n) = depends only on (q, k, m-n)

The dot product result depends only on the relative position m-n, not on the absolute position. This naturally encodes relative distance into the attention computation without modifying the attention formula itself.

RoPE’s engineering advantages:

  • Parameter-free: No additional learnable parameters
  • Naturally encodes relative distance: Dot product value depends only on relative position
  • Extendable: With techniques like YaRN (Yet another RoPE extensioN) and Positional Interpolation, the training context window can be extended severalfold — Llama 3.1 uses RoPE + long-context fine-tuning to reach 128K context
               Absolute Positional Encoding
               ┌──────────────────────────────┐
               │ Sinusoidal (additive)        │ ← original Transformer
               │ Learnable embedding (additive)│ ← BERT, GPT-2
               └──────────────────────────────┘

               Relative Positional Encoding
               ┌──────────────────────────────┐
               │ T5 Bias (attention)          │ ← T5
               │ ALiBi (linear decay)         │ ← BLOOM, MPT
               └──────────────────────────────┘

               Rotary Encoding (multiplicative)
               ┌──────────────────────────────┐
               │ RoPE                         │ ← LLaMA, Mistral,
               │                              │   Qwen, DeepSeek
               └──────────────────────────────┘

What About No Positional Encoding?

Some 2023 research explored whether Transformers without positional encoding could work. The conclusion: for specific tasks with few tokens (classification), models can infer position implicitly from causal masking. But for language generation, models without positional encoding have higher training loss and significantly worse generation quality. Non-Transformer architectures like Mamba and RWKV encode position implicitly through SSM (State Space Model) or RNN time steps — that’s a different path.

Summary

SchemeParametersExtrapolationRelative DistanceModern LLM Adoption
SinusoidalNonePoorIndirectRare
Learnable absoluteYesPoorIndirectRare (BERT era)
T5 BiasFewMediumDirectT5 family
ALiBiNoneGoodDirect (linear)BLOOM, MPT
RoPENoneGood (with help)Direct (rotation)LLaMA, Mistral, Qwen…

RoPE’s dominance isn’t accidental — it simultaneously satisfies “parameter-free,” “relative distance,” and “extendable,” and has been validated across a large number of LLM training runs. Understanding RoPE’s mathematical principle also helps explain why long-context extrapolation techniques like YaRN work: fundamentally, it’s adjusting θ frequencies so the model acts as if it’s still within its training position range.

References

Ask this article

Answers come from this article only. Click any prompt below or open the chat at the bottom right.

🇺🇸 English

Here's a fun little puzzle to start with. Take two sentences: "cat bites dog" and "dog bites cat." To you and me, those mean wildly different things — one of them is a bad day for the dog, the other is a bad day for the cat. But feed both into a raw Transformer with no sense of position, and the model literally cannot tell them apart. To it, both sentences are just the same bag of words: "cat," "bites," "dog," in some order it doesn't even register.

Why? Because the heart of a Transformer — self-attention — is what we call permutation-invariant. Every token gets to look at every other token, all at once, fully connected. And that "everyone talks to everyone" design is powerful, but it throws away order completely. Shuffle the words around, and the outputs just shuffle right along with them. The actual values don't change. For something like classifying patches of an image, that's fine. But for language? Word order carries a huge chunk of the meaning. So the whole game of positional encoding is this: how do we inject a sense of position into the model, without breaking that elegant attention mechanism? And the story of how we've answered that question runs from 2017 all the way to today's biggest language models.

Let's walk through it, generation by generation.

The first attempt came from the original Transformer paper, and it's called sinusoidal absolute positional encoding. The idea: for every position in the sentence — position one, two, three, and so on — you compute a little vector using sine and cosine waves of different frequencies. Then you just add that vector directly onto the word's embedding before it enters the model. No training required, it's all fixed math.

The intuition here is actually kind of beautiful. Think about how binary counting works — some digits flip fast, some flip slow. Same thing here. The low-frequency waves capture coarse position: are we in the first half of the sentence or the second half? The high-frequency waves capture fine position: which exact slot is this? Stack those frequencies together and each position gets a unique fingerprint.

The catch — and this becomes a recurring theme — is extrapolation. During training, the model only ever sees sentences up to a certain length. Go beyond that at inference time, and there's just no meaningful signal for those far-out positions. Quality falls off a cliff.

Second generation: learnable absolute positional encoding. This is what BERT and GPT-2 used. Instead of a fixed sine-and-cosine formula, you just make a big lookup table — one trainable vector per position — and let the model learn those position vectors through backprop, same as any other parameter. The upside is flexibility: the model can shape position representations to fit the task. But the downsides pile up. It adds parameters. It still can't extrapolate — if your table stops at position 512, there's simply nothing at position 513. And here's the subtle one: it never explicitly models the relationship between, say, position one and position two. The model has to figure out "these two are neighbors" entirely on its own.

That subtle problem is exactly what the third generation set out to fix: relative positional encoding. The insight is that what usually matters isn't "this word is at absolute position forty-two" — it's "this word is three tokens away from that one." So instead of tagging each word with an absolute address, you inject the relative distance directly into the attention scores.

T5 did this by adding a learned bias for each query-key pair based on how far apart they are. ALiBi took an even cleaner route: for each attention head, it just adds a negative penalty that grows linearly with distance. The farther apart two tokens are, the bigger the penalty — attention naturally decays with distance. No extra parameters, and it holds up surprisingly well when you push it to longer sequences than it trained on.

And that brings us to the star of the show: RoPE, Rotary Positional Embedding, from 2021. This is the one that basically won. LLaMA uses it, Mistral, Qwen, DeepSeek, PaLM 2 — nearly every modern large language model runs on RoPE.

So what makes it different? Every approach before it added position information. RoPE multiplies it in. And it does that through rotation. Here's the picture: take the query and key vectors, and split them into pairs of dimensions. For a token sitting at position m, you rotate each of those pairs by an angle proportional to m. Position five gets rotated a little, position five hundred gets rotated a lot. Different dimension-pairs rotate at different speeds — again, that same spread of frequencies we keep seeing.

Now here's the magic, and it's genuinely elegant. Attention works by taking dot products between queries and keys. When you take the dot product of a query rotated by position m and a key rotated by position n, the geometry works out so that the result depends only on the difference — m minus n. Not on the absolute positions themselves, just the gap between them. So by doing something as simple as rotating vectors, you get relative distance falling out of the attention computation for free, without touching the attention formula at all.

And that gives RoPE this trio of properties that's hard to beat. It's parameter-free — no lookup tables, nothing extra to learn. It naturally encodes relative distance. And it extends gracefully: with tricks like YaRN and positional interpolation, you can stretch the context window way beyond what it trained on. That's how Llama 3.1, for instance, reaches a 128,000-token context — RoPE plus some long-context fine-tuning.

Quick detour, because someone always asks: what if you use no positional encoding at all? Some research back in 2023 poked at this. Turns out for narrow tasks with just a handful of tokens — like classification — the model can actually sneak position information out of the causal mask, since each token can only look backward. But for real language generation? No positional encoding means higher training loss and noticeably worse output. Now, architectures like Mamba and RWKV do skip explicit positional encoding — but that's because they're not Transformers. They bake position into their sequential time steps, through state-space models or RNN-style recurrence. Different road entirely.

So let me tie the whole landscape together. If you line these schemes up: sinusoidal is parameter-free but extrapolates poorly and only handles distance indirectly — you rarely see it now. Learnable absolute encoding, the BERT era, adds parameters, still can't extrapolate, still indirect. T5's bias handles relative distance directly and does okay on longer sequences, but it lives mostly in the T5 family. ALiBi is parameter-free, extrapolates well, and handles distance directly through that linear decay — you'll find it in BLOOM and MPT. And then RoPE: parameter-free, extends well with a little help, encodes distance directly through rotation, and it's in basically every flagship model you can name.

Let me leave you with the three things worth remembering.

First: attention is orderless by nature. Self-attention, on its own, genuinely cannot tell "cat bites dog" from "dog bites cat." Positional encoding is the fix that gives the model a sense of sequence, and everything else is a debate about how best to do it.

Second: the field moved from absolute to relative, and from adding to multiplying. Early methods stamped each word with an absolute address and added it in. The winning idea was to encode the distance between words instead — and RoPE pulled that off by rotating vectors rather than adding anything.

And third: RoPE's dominance isn't hype, it's an engineering sweet spot. It's the one method that's simultaneously parameter-free, relative-distance-aware, and extendable to long contexts — validated across a mountain of real model training runs. And once you understand that it's all just rotation by angle, you understand why context-extension tricks like YaRN work too: they're quietly retuning those rotation frequencies so the model feels like it's still inside the range it was trained on. Same idea, stretched a little further.

🇹🇼 中文

「你打我」和「我打你」，同樣三個字、順序不同，意思卻完全相反。如果一個模型分不出這兩句話的差別，它根本沒辦法理解語言。而問題就在這裡——原始 Transformer 裡的 Self-Attention，還真的分不出來。今天要聊的，就是把「順序」補回去的技術：位置編碼，Positional Embedding。

先搞清楚，Self-Attention 為什麼看不到順序。

Transformer 的輸入是一串 Token，每個 Token 先變成一個向量，也就是 Embedding，然後送進一層一層的 Layer，每層裡面都有一個 Self-Attention 模組。它做的事情是：進去幾個 Token，出來一樣數目的 Token。

我們看它怎麼算。假設輸入四個 Token，變成四個 Embedding。每個 Embedding 各自乘上三個矩陣，得到 Q、K、V，也就是 query、key、value。要算最後一個位置的輸出時，先拿它的 query 去跟每個 Token 的 key 做內積，得到一組 Attention weight；做完 Softmax 正規化之後，再拿這些權重去對每個 Token 的 value 做加權總和。

關鍵就在最後這個「加權總和」。假設我把第一個和第三個 Token 對調，輸入從 ABCD 變成 CBAD，對最後那個輸出有沒有影響？答案是——完全沒有。因為 A、C 位置一換，它們對應的 Q、K、V、Attention weight 也跟著換，而加總這件事，先加 A 還是先加 C，結果一模一樣。

這就叫「排列不變」，permutation invariant。對「集合」類的問題無所謂，但對語言是致命的：「你打我」跟「我打你」，最後一個位置算出來的 Embedding 竟然一樣，模型當然分不出來。所以我們得額外餵它位置資訊。

第一個方案，Absolute Positional Embedding，絕對位置編碼。想法很直接：每一個位置都給一個專屬向量。位置零用 P 零、位置一用 P 一，然後直接加到該位置的 Token 上。這樣同樣一個 X_A，放在位置零加的是 P 零，放在位置二加的是 P 二，對 Self-Attention 來說就變成兩個不同的東西了，算出來自然不一樣。位置資訊就這樣被注入進去。

剩下的問題是：這些位置向量，到底長什麼樣？

Transformer 剛誕生的那個年代，用的是一種叫 Sinusoidal 的位置編碼，用 sin 跟 cos 來建構。規則是這樣：偶數維度用 sin，奇數維度用 cos，括號裡面是「位置 k，除以一個跟維度有關的分母」。分子 k 就是第幾個位置，k 越大角度越大；分母那一串，維度不同、分母不同，決定了角度變化的快慢。

視覺化來看，如果把所有位置在第零維的數值拉出來,沿位置軸看，會是一條 sine 波；第一維是 cosine 波；第十維又是 sine 波，但週期跟第零維不一樣，因為分母裡的維度變了。整體來看，靠近第零維的維度變化很快、頻率高，維度編號越大變化越慢、頻率越低。

有個很好的比喻：每一對維度，一個 sin 一個 cos，合起來其實就是二維平面上的一根指針，隨著位置往前而不斷旋轉。轉一圈要多少個位置？三角函數週期是 2π，算下來，最前面那對維度大概走六個位置就轉一圈，像秒針；中間的要走六百多個位置，像分針；最後那對要走五萬多個位置才轉一圈，像時針。一般時鐘三根指針，這裡呢？看你維度多少——一百二十八維，就是六十四根轉速全都不一樣的指針。我們希望 Self-Attention 看著這六十四根指針，就能判斷「現在在哪個位置」。

那問題來了，方法明明很多，2017 年作者為什麼偏偏選這個？論文正文只用一句話帶過，但微言大義：因為他們希望位置編碼能考慮「相對位置」。

什麼是相對位置？看「貓吃了魚」。「貓」跟「魚」隔了兩個 Token，假設處理「魚」的時候要 attend 回「貓」，分數是零點七。現在我在前面硬塞一堆字，「今天早上我看到貓吃了魚」，甚至塞一千個 Token，但「貓吃了魚」這個事件本身沒變，我們會希望那個 attention 分數還是零點七。反過來，如果貓在句首、魚在句尾，距離很遠，我們就希望分數小一點。換句話說，真正重要的往往是相對距離，不是絕對位置。

而 Sinusoidal 剛好藏了一個支撐相對位置的漂亮性質：位置 k 的 Embedding，乘上一個矩陣 M_R，就會變成位置 k 加 R 的 Embedding。而且這個矩陣只跟相對距離 R 有關，跟你在哪個絕對位置完全無關。所以 P 一乘 M 三等於 P 四，P 一百零一乘同一個 M 三，就等於 P 一百零四。

怎麼證？把位置 k 加 R 那一對維度，用高中的合角公式展開——sin(A 加 B) 跟 cos(A 加 B) 那兩條。展開之後你會發現，裡面冒出來的項，正好就是 P_k 的那兩個維度。整理成矩陣，就得到一個只跟 R 有關的二乘二旋轉矩陣。把每一對維度的小矩陣沿對角線排起來，就組成完整的 M_R。兩個位置的關係，只由相對距離決定。

那這個性質怎麼實際影響 attention？位置 n 當 query、位置 m 當 key，各自都是「Token Embedding 加上 Positional Embedding」再乘轉換矩陣。attention 分數是兩者內積，把括號展開，會得到四項：第一項只跟內容有關、完全不看位置；中間兩項是內容跟位置交互，比較複雜先擱著；第四項最有意思，只跟位置有關。

第四項單看，只跟絕對位置 m、n 有關，看不出相對性。但因為我們有剛剛那個性質，可以把 P_m 換成 M 乘 P_n，於是這一項就冒出一個「跟相對距離 m 減 n 有關」的成分，被加進 attention 分數裡，讓 attention 能感知相對距離。

不過要誠實講，這個影響相當間接。這一項裡除了相對位置，還混著絕對位置跟其他東西，四項裡面真正純粹跟相對位置有關的，也就這麼一小塊。Sinusoidal 是拐了個彎，才勉強把相對資訊塞進 attention。

所以整條演進線收斂成三個重點。第一，問題的根源是 Self-Attention 排列不變，天生分不出 Token 順序，所以位置資訊必須額外補進去。第二，從 Absolute 到 Sinusoidal，是用不同頻率的 sin 跟 cos，把每個位置編成一組轉速不同的指針。第三，Sinusoidal 那個「乘個矩陣就平移」的性質，讓 attention 能「間接」感知相對位置——但也就是間接而已。

既然大家真正想要的，是把相對資訊直接加進 attention，那何必拐彎去設計這麼神奇的絕對位置編碼？能不能乾脆跳過位置編碼、直接改 attention 本身？這，就是接下來 Relative Positional Embedding 時代要回答的問題了。

Tags

#transformer#rope#positional-encoding#nlp#machine-learning#deep-learning

Related Articles