Attention, From the Ground Up
The word bank means a riverside in one sentence and a lender in another, yet the model stores just one meaning for it. Attention fixes that by letting each word borrow meaning from its neighbors: drag the weights around, read a hand-built head doing grammar, then watch a real one learn where to look.
Attention, From the Ground Up
Start with the problem attention solves, end by watching attention learn to solve it.
one word, two meanings
1. The Problem
Two sentences share the word bank. Below is a small map of meaning space: words about rivers cluster on the left, words about money on the right. Every word gets one fixed embedding, the list of numbers the model stores for it, so bank is stuck at a single point between the clusters no matter which sentence it appears in. Toggle the sentence and watch nothing happen. That is the problem.
bank′ = (1−t)·bank + t·mean(river, water)
= 1.00·(0.05, 1.15) + 0.00·(-3.05, 1.08) = (0.05, 1.15)
= 1.00·(0.05, 1.15) + 0.00·(-3.05, 1.08) = (0.05, 1.15)
Drag the slider and the dot lands in the right cluster, computed as a plain average of the context words. Attention's output is exactly this: a weighted average of the other tokens' vectors (a token is roughly one word, and its vector is that embedding, the same list of numbers). Everything that follows, the queries, the keys, the softmax, exists to answer one question: what should the weights be?