Interpretability

The Word It Has Already Chosen

I misspelled langauge and the model read it as language anyway. I set up a rhyme and it locked the last word in long before writing it. Decoding the model's internal guess at each layer with the paper's lens, I could see the correction and the rhyme were settled well before the output. The readouts are from a 4B model I ran myself.

interpretabilitytransformersjacobian-lensplanningtypos

LiveInteractive · drag, toggle, run it

Interpretability

The word it has already chosen

I wrote langauge, with two letters flipped, and the model still understood language. It didn't stumble on the typo the way a strict string match would. Somewhere inside, the messy characters on the page get read as the word I meant. Where does that happen, and how early?

The paper below documents both the typo reading and the rhyme planning, and the lens that shows them is theirs, the same one from the arithmetic walkthrough. Running it on a small model myself, I found the corrected word language is already the top pick by the lower middle of the network, and the literal misspelling never leads. The rhyming couplet shows the same early commitment: the model settles on death to rhyme with breath and then holds it for the rest of the stack.

The course required that every student learn a second langaugereading the model's mind at the highlighted position (what it is about to say next)

when each word lights up, layer by layer

Each line is one word. Reading left to right is going deeper into the network. Higher up means the model ranks that word closer to the top of what it might say next; a dot marks where it first breaks into the top few.

language  (what it reads)langauge  (what is on the page)

Layer 0 of 31

drag the slider or press play to sweep from the first layer to the last

auge

18%

whilst

2%

enje

1%

amongst

0%

orgeous

0%

但不限于

0%

eid

0%

麵

0%

Showing real words only. The raw top of the list is mostly punctuation and word fragments, filtered out here the same way the paper's own viewer does. Even filtered, the middle layers are messy before the answer settles.

what the model actually said next:.

Read at the right spot

I take both readouts at the last token of the prompt, the point where the model is deciding what comes next. For the typo that means it has already folded the garbled spelling into a clean sense of the word. For the rhyme it means the payoff word is chosen up front rather than improvised at the last instant. The paper shows a sharper version on larger models, where the rhyme for the next line is planned several words ahead. Here the effect is smaller, but the shape is the same.

More of the same lens at work:

Doing arithmetic : the sub-result, then the answer.
A silent reasoning step : reaching "Italy" without saying it.
The global workspace : the bigger claim behind all of these.

I computed these readouts on Qwen/Qwen3.5-4B with the lens neuronpedia/jacobian-lens@qwen-n1000. From Verbalizable Representations Form a Global Workspace in Language Models (Anthropic, 2026). Lens code: anthropics/jacobian-lens. I generated every number here with scripts/gen-lens-data.py in this repo; rerun it to reproduce.