Interpretability

The Reasoning Step It Never Says Out Loud

I asked a model for the currency of the country shaped like a boot and it answered euro. To get there it first has to decide the country is Italy, but it never writes Italy. That step happens silently, inside. Using the paper's lens I caught Italy surfacing mid-network before it hands off to euro. The readouts are from a 4B model I ran myself.

interpretabilitytransformersreasoningjacobian-lensmulti-hop

LiveInteractive · drag, toggle, run it

Interpretability

The reasoning step it never says out loud

I gave a model "The currency used in the country shaped like a boot is..." and it finished with euro. That takes two hops: first work out that the boot-shaped country is Italy, then recall that Italy uses the euro. But the model never writes "Italy". It jumps straight from the riddle to the currency, so from the outside the middle step is invisible.

The paper below is where this hidden step was found, and the lens that reveals it is theirs, the same one from the arithmetic walkthrough. I ran it on the boot-currency example on a small model of my own, and could watch Italy rise out of the noise in the middle of the network, hold near the top for a while, then hand off to euro near the end. The model needed that thought and never said it.

Fact: The currency used in the country shaped like a boot isreading the model's mind at the highlighted position (what it is about to say next)

when each word lights up, layer by layer

Each line is one word. Reading left to right is going deeper into the network. Higher up means the model ranks that word closer to the top of what it might say next; a dot marks where it first breaks into the top few.

Italy  (the silent middle step)euro  (the answer)lira  (the old currency, a near miss)

Layer 0 of 31

drag the slider or press play to sweep from the first layer to the last

however

2%

the

1%

a

1%

an

1%

in

1%

one

1%

at

0%

there

0%

Showing real words only. The raw top of the list is mostly punctuation and word fragments, filtered out here the same way the paper's own viewer does. Even filtered, the middle layers are messy before the answer settles.

what the model actually said next:the Euro.

Why the buried step matters

There are two ways a model could answer this. It could pattern-match "boot currency" straight to "euro" from training text, or it could actually compose two separate facts. The lens shows Italy present as its own distinct thing before the currency forms, which points to the second: the country gets computed, held, and then used. In the Carnival case the buried country (Brazil) is fainter, which you can see in the softer orange line. Not every hop reads out this cleanly at 4B.