A Taxonomy of Positional Embeddings from the Logic of Attention

Positional embedding in the Transformer begins with one question: how should a model receive information about token order?

The central operation in a Transformer is self-attention. By design, it reflects the elements of the context—the token or embedding information processed together in this article—but not each element's position.

Consider two sentences.

1. I love AI studying.
2. AI love I studying.

They clearly mean different things, and the second is grammatically incorrect. Even so, if both contain the same set of tokens, attention can perceive them as the same input.

At its simplest, a Transformer encoder receives input $X$, extracts $Q$, $K$, and $V$, performs attention, and then applies an MLP.

Suppose the input is a sequence of tokens, such as words.

$$X = [x_1, x_2, \cdots, x_n]$$

The tokens are transformed into queries, keys, and values for attention.

$$Q = XW_Q,~K = XW_K,~V=XW_V$$

Attention takes a weighted sum of token values, where the weights are relative probabilities derived from dot products between queries and keys. Because the raw score can scale with the embedding dimension—a dot product does not account for vector-norm magnitude—the widely used scaled dot-product formulation is

$$Attention (Q, K, V) = softmax\left(\frac{QK^\top}{\sqrt{d}} \right)V.$$

The important point is that the attention score between token $i$ and token $j$ is $Q_iK_j^\top$. It contains only a dot product of token embeddings; token positions do not enter the calculation.

Return to the earlier example.

1. I love AI studying. $\rightarrow$ [I, love, AI, studying]
2. AI love I studying. $\rightarrow$ [AI, love, I, studying]

The positions of I and AI change, but their embeddings remain the same, so the same attention scores appear. Position alone cannot distinguish the sequences.

Permutation Invariance in Equations

Consider a permutation matrix $P$. Multiplying an object by a permutation matrix rearranges its elements. A permutation matrix can be constructed by rearranging the entries of an identity matrix within each row.

$$X = [1, 2, 3, 4],~P = \begin{bmatrix} 0 & 1& 0 & 0 \newline 1 & 0 & 0& 0 \newline 0 & 0 & 0 & 1 \newline 0 & 0 & 1 & 0 \end{bmatrix}$$

For the vector $X$ and permutation matrix $P$ above, matrix multiplication gives the permuted $X^\prime$.

$$X^\prime = XP = [2, 1, 4, 3]$$

Now suppose $X$ is a token sequence of length $n$. An $n\times n$ permutation $P$ can produce $X^\prime$ by rearranging its elements. Calculating queries, keys, and values for the permuted sequence gives

$$\begin{aligned} Q^\prime =& P(XW_Q) = PQ \newline K^\prime =& P(XW_K) = PK \newline V^\prime =& P(XW_V) = PV. \end{aligned}$$

Applying the same attention operation,

$$Attention (PQ, PK, PV) = softmax\left(\frac{PQ(PK)^\top}{\sqrt{d}} \right)PV.$$

A permutation changes only positions, not the magnitude of the result, so it commutes with softmax in the required way. A permutation matrix is also orthogonal: $P^\top=P^{-1}$, and therefore $PP^\top=P^\top P=I$. Using these properties simplifies the equation to

$$Attention (PQ, PK, PV) = P \cdot Attention(Q,K,V).$$

Rearranging the token positions rearranges the attention outputs in the same way. A system in which an input transformation is reflected correspondingly in the output is called permutation equivariant.

Equivariance means that a change in the input appears correspondingly in the output. From the perspective of the Transformer's aggregate meaning, however, attention alone cannot use that rearrangement to distinguish token positions, so the model behaves invariantly to order.

To put the subtlety another way: the operation itself is equivariant, but semantically the model output is invariant. An attention-based Transformer without positional information can therefore be described as permutation invariant with respect to sequence meaning.

We Therefore Need Position Information

Attention calculates only similarity or association between tokens. Changing their positions does not itself change that relationship, but word order determines both grammar and meaning.

1. Dog bites man
2. Man bites dog

In the first sentence, [Dog] is the subject of [bite] and [Man] is its object. In the second, those roles are reversed. A Transformer therefore needs an added mechanism that expresses order.

1. Absolute Positional Embedding

Absolute positional embedding, or APE, adds a unique value for each absolute input position.

$$X + APE = [x_1+p_1,x_2+p_2,\cdots,x_n+p_n]$$

This was the earliest approach. Its simplest representative forms are learned positional embeddings and sinusoidal positional embeddings.

Suppose we learn one embedding for every position: $[0.42,-0.31,\cdots]$ at index 0, $[-0.14,0.82]$ at index $k$, and so on. The positional embedding acts like a key–value lookup table added to the token representation.

The moment the input exceeds the length seen in training, however, its positions become out of distribution. A model trained for at most 2,048 tokens cannot find embeddings for positions 2,049 onward in its lookup table when it receives 4,096 tokens.

The original Transformer paper therefore used sine and cosine rather than learning a separate embedding.

$$\begin{aligned} PE(pos, 2i) =& \sin\left(\frac{pos}{10000^{2i/d}}\right) \newline PE(pos, 2i+1) =& \cos\left(\frac{pos}{10000^{2i+1/d}}\right). \end{aligned}$$

Each embedding dimension uses a different frequency, while every position receives a unique embedding.

APE still has a fundamental problem: semantic and positional information become completely mixed before projection.

$$\begin{aligned} Q_iK_j^\top =& (x_i+p_i)^\top W_Q^\top W_K (x_j + p_j) \newline =& x_i^\top A x_j + x_i^\top A p_j + p_i^\top A x_j + p_i^\top A p_j. \end{aligned}$$

Because positional embeddings are added before queries and keys are calculated, the score mixes token–token, position–position, and token–position relationships. Later nonlinear operations must disentangle all of this, which is difficult to learn without additional modeling. With very long contexts, APE can also reduce numerical stability, while periodic functions may reach representational limits.

2. Relative Positional Embedding

APE adds a unique value according to absolute position. It therefore cannot naturally preserve the same relationship when an identical phrase begins at another location.

Consider

1. The cat sits on the box
2. I know the box where the cat sits usually

Within the local phrases, the distances between consecutive words such as the box and cat sits remain one. With APE, however, the added embedding changes with the absolute position, even for the same token. More importantly, the relationship between consecutive tokens should appear consistently in the attention map so that training can learn a consistent notion of token distance, but APE does not guarantee this.

Relative positional embedding therefore adds the relative distance between tokens to the attention score.

$$Q_iK_j^\top + Q_i R_{i-j}$$

The bias depends on the distance and order between tokens $i$ and $j$: it is negative in one direction and positive in the other. Attention can therefore include information about relative distance.

This method can greatly increase memory and computation. APE is added once to the token sequence. Relative embeddings enter the attention scores, so every score element needs a distance lookup. Both storage and computation grow accordingly.

3. Rotary Positional Embedding

Absolute embeddings have problems with learned representation and missing relative distance. Relative embeddings increase computation and memory. Rotary Positional Embedding, or RoPE, asks whether we can retain the advantages of both.

As modern LLMs adopted attention at scale, their context lengths increased and resource overhead grew with them. This made the efficiency and effect of positional representations during training much more important than they had been in the original Transformer.

RoPE argues that adding positional embeddings is itself problematic. Attention scores and token embeddings already contain representational features. Adding another vector can distort those relationships. RoPE instead approaches position as a rotation of the embedding.

The central idea is to rotate a multidimensional vector in embedding space by an angle determined by the token position. The paper's intuition is that representations in Euclidean space are constructed through inner products, or similarity, between tokens, and that angular difference matters more under rotation than raw vector distance.

This has a clear advantage. We do not need to look up the distance between every token pair and design an embedding for it, because rotation matrices have the property

$$R_i^\top R_j = R_{j-i}.$$

The intuition is straightforward. Suppose $R_i$ rotates a vector counterclockwise by $\theta_i$, while $R_j$ rotates it by $\theta_j$. A rotation matrix is orthogonal, so its inverse equals its transpose: $R^\top=R^{-1}$.

$R_i^\top$ therefore rotates clockwise by $\theta_i$. When the two rotations are multiplied, they apply sequentially, and the final result rotates the original vector counterclockwise by $\theta_j-\theta_i$. This is $R_{j-i}$.

Every vector is rotated by a unique angle determined by its absolute position, much like APE. The angular relation that remains between two rotated vectors naturally depends only on their index difference, giving a stable relative position at the same time.

Because RoPE uses predetermined absolute positions as rotation angles, it requires neither additional learned parameters nor a lookup table.

The angular representation $e^{i\theta}$ can still suffer frequency aliasing over long text and lose expressiveness at high frequencies. Even so, RoPE has been widely recognized as an effective combination of APE's and RPE's advantages. It is now used by many LLM families, including LLaMA, Qwen, and Gemma.