Learning LabExplorable explanations
← All artifacts
Interpretability

Watch a Language Model Do Arithmetic

I gave a language model (2 + 3) * 4 and it wrote 20 without running a calculator. Using the paper's lens I read the model's half-formed answer out of each layer and watched it reach the sub-sum 5 partway up, then twenty near the top. The readouts are from a 4B model I ran myself.

interpretabilitytransformersreasoningjacobian-lensarithmetic
LiveInteractive · drag, toggle, run it
Interpretability

Watch a language model do arithmetic

I typed (2 + 3) * 4 into a language model and it answered 20. It didn't run a calculator or any code, it just did one forward pass, the same single sweep through the network it uses to guess the next word in a sentence. So where does the 20 come from?

This is one of the things Anthropic works out in the paper linked below. A transformer builds its answer in stages, one per layer, and they built a lens (the Jacobian lens) that decodes the half-formed guess sitting in the model at any layer into plain words, a bit like clipping a probe onto a chip and reading the bus while it computes. None of that is mine. What I did was run their lens on a small model I could host myself and wire it to a slider, so you can scrub through the same thing: the guess reaches the inner sum 5 partway up, then twenty near the top, before it writes a single digit.

(1 + 4) * 2 = 10↵ (2 + 2) * 3 = 12↵ (2 + 3) * 4 =
reading the model's mind at the highlighted position (what it is about to say next)
when each word lights up, layer by layer
Each line is one word. Reading left to right is going deeper into the network. Higher up means the model ranks that word closer to the top of what it might say next; a dot marks where it first breaks into the top few.
topburiedL0L15L30
5 (the sub-sum 2+3)twenty (the answer)
Layer 0 of 31
drag the slider or press play to sweep from the first layer to the last
e
0%
eza
0%
ações
0%
auf
0%
eled
0%
o
0%
iado
0%
in
0%
Showing real words only. The raw top of the list is mostly punctuation and word fragments, filtered out here the same way the paper's own viewer does. Even filtered, the middle layers are messy before the answer settles.
what the model actually said next:20
I give the model two worked examples first, then the expression. Without that nudge the 4B model reads a bare expr = as a quiz to echo back rather than solve, and computes nothing.

The answer shows up as a word, not a digit

The answer shows up as twenty or forty, not 2 or 4. Inside the network the result is a number as a concept, and only in the last few layers does it commit to the actual digit it will write. That is what makes the lens worth using to me: I can read the meaning the model is holding before it turns into output.

What the lens actually is

A transformer keeps a running vector for each token, the residual stream, and refines it layer by layer until the top layer turns it into a next-word guess. The natural way to peek at an early layer is to shove its vector straight into that final guess step (the logit lens), but early vectors live in a different internal coordinate system, so the readout is mush. The Jacobian lens first applies the average linear map from that layer to the top before decoding, which is why5 and twenty come out as real words instead of noise. It is a read-only probe: it changes nothing about the model, it just reports what a layer is leaning toward saying.

Next, the same lens on a different kind of thought:

I computed these readouts on Qwen/Qwen3.5-4B with the lens neuronpedia/jacobian-lens@qwen-n1000. The method and the original arithmetic example come from Verbalizable Representations Form a Global Workspace in Language Models (Anthropic, 2026). Lens code: anthropics/jacobian-lens. I generated every number here with scripts/gen-lens-data.py in this repo; rerun it to reproduce.