Rotary Position Embedding: The Clock Hands of Attention
Attention compares tokens with a dot product, a similarity score that has no idea which token came first. So how does a model tell "dog bites man" from "man bites dog"? RoPE's answer isn't to stamp a position number onto each vector, it's to spin the vector by an angle proportional to its position. Spin two vectors to a fixed angle apart and watch their dot product hold dead still, split one hand into four speeds to see a single position encoded at every resolution at once, then push a sequence past the length a model trained on and watch the fix that keeps the angles in familiar territory.
Rotary Position Embedding
Feed a transformer “dog bites man” and “man bites dog” and, on their own, attention can't tell them apart: it scores every pair of tokens with a dot product (a similarity score, the sum of each vector's matching components multiplied together), and a dot product only measures how alike two vectors are, never which one came first. So the position of every token has to be baked into its vector somehow before that comparison happens.
RoPE's move is to rotate each vector by an angle proportional to its position, rather than tagging a position number onto it. Below, drag two vectors to a fixed angle apart and watch the dot product between them hold still no matter where they point, then break the match and watch it move.
Why not just add a position number?
The obvious fix is the older one: compute a position vector for slot 0, 1, 2, … and add it to each token's embedding (its content vector) before attention ever runs. It works, and it is what the original Transformer paper did. But now the vector fed into the dot product carries the token's absolute index baked in—shift a whole sentence later in a document and every vector changes, even though nothing about the sentence itself did.
Start with one vector
Take a single toy vector, [1, 0.5]—the dashed arrow below, sitting at “position 0.” RoPE encodes a different position by rotating that same vector counterclockwise around the origin, by position × θ (a fixed angle per step, shown here as a friendly 30°). Drag the slider and the solid arrow sweeps away from the dashed one; the shaded arc between them is exactly how many degrees it turned. The dotted circle is the vector's length: rotation is a multiplication by a matrix that only ever spins a vector around that circle, so the tip can move along it but never toward or away from the center.
“CCW” is counterclockwise, the direction a positive angle turns in standard math (and the direction this page's rotation formula always uses). Position 0 means zero rotation, so the dashed and solid arrows overlap; each slider click adds one more 30° turn. Real RoPE uses a much smaller angle per position (see “One angle isn't enough” further down)—only the step size changes, not the idea.
The core trick: rotate, don't add
Take one query vector (what a token is looking for) and one key vector (what a token offers). Rotate the query by its position m times a fixed angle θ, and the key by its position n times the same angle. Because a rotation is a rigid turn that preserves length and angle, rotating both vectors and then comparing them is mathematically identical to leaving the query alone and rotating the key by (n−m)θ—the dot product literally cannot see m or n individually, only their difference.
The shaded arc is the actual angle between the two vectors right now—not just(m−n)×30°, since Q and K don't start out pointing the same way, but that plus their own fixed head start. Click “shift both” repeatedly: the arc, the distance box, and the dot product all hold still. Drag m or n alone and all three move together, because the actual gap between the vectors changed. Real RoPE uses a much smaller angle per position; the identity being demonstrated is the same either way.
One angle isn't enough
A single rotation angle can only tell two tokens apart up to how many degrees fit in a circle before it repeats. Real RoPE splits each embedding into pairs of numbers (dimensions), and spins every pair at its own frequency: θi = base−2i/d, a geometric sequence, so pair 0 spins fast and later pairs spin ever slower. It's a clock with several hands sharing one face: the fast hand distinguishes tokens one or two apart, the slow ones distinguish tokens dozens or hundreds apart, and together they cover both scales from a single position number.
| Pair i | θᵢ (rad/step) | Degrees/step | Full turn every |
|---|---|---|---|
| 0 | 1.0000 | 57.3° | 6.3 steps |
| 1 | 0.1000 | 5.7° | 62.8 steps |
| 2 | 0.0100 | 0.6° | 628.3 steps |
| 3 | 0.0010 | 0.1° | 6283.2 steps |
Base 10000 and 4 pairs (an 8-number embedding slice), matching the formula real models use. The slowest hand here barely creeps across 64 steps—that's deliberate: it's the one reserved for telling apart tokens hundreds of steps apart. Note the hands sweep clockwise as position increases, like an ordinary clock—the opposite visual direction from the counterclockwise vector plots above. Both draw the exact same rotation formula; this page just reads a single spinning hand as a clock and a single arrow as a math-class vector, and each reads more naturally its own way.
The same invariance, on real query/key vectors
The single clock hand above was a toy 2D vector. Real attention runs on 8, 64, or 128 numbers per token, rotated in pairs. Here are two fixed toy words, “quick” (key) and “fox” (query), with deterministic seeded vectors standing in for learned embeddings. Slide both of their positions together through a longer stretch of text and the RoPE-adjusted score won't budge; a raw dot product with no positional information at all never moved to begin with—proof that without something like RoPE, attention genuinely cannot see order.
The raw score is identical at every shift value because it never looks at position at all—the problem the intro opened with. The RoPE score is exactly the same at every shift too, as long as “break the match” stays at zero; move it and the score changes, because now the relative distance changed.
Past the trained length
Say a model only ever saw sequences up to 128 tokens during training. Every dimension pair only ever swept through the angle range that fits inside those 128 steps. The fast pairs wrap around the circle many times in that span, so any further rotation just lands on angles the model has already seen—wrapping is harmless for them. The slow pairs are the risk: over only 128 steps they barely move past a sliver of the circle, so pushing the sequence longer walks them into angles that are numerically valid but were never part of training data.
Turning rescaling on changes the frequency base to base · (L/L_train)^(d/(d−2)), which stretches the slow pairs back down so their angle at the new length matches what they topped out at during training, while leaving the fastest pair (θ₀ = base⁰ = 1, unaffected by any base) exactly as it was. This is the idea behind “NTK-aware” RoPE scaling; YaRN refines it further by blending old and new frequencies per dimension instead of one global rescale.