Word Embeddings and Analogy Arithmetic
King minus man plus woman lands on queen, and it sounds like a trick. Place words in a space where meaning is direction, then watch the arithmetic fall out of the geometry.
Word Embeddings and Analogy Arithmetic
King minus man plus woman lands on queen, and the first time you see it the result feels like sleight of hand. It stops being magic once each word is a point in space. Give every word a few coordinates so that similar words sit close together, and a relationship like male to female becomes a fixed step in one direction. Take that same step from king and you arrive next to queen.
Every word here is five numbers, one per labeled axis. A word is a vector, and two words mean similar things when their vectors point nearly the same way. The map further down is a flat snapshot of those five dimensions, so left to right tracks male to female and bottom to top tracks animal-or-thing up through person and royalty.
Read it as a is to b as c is to what. The step from b to a is one arrow; take the same arrow starting at c and its tip lands on the answer. When the two arrows come out parallel and the same length, the four points form a parallelogram, which is the picture behind the arithmetic a - b + c.
The three input words are held out of the ranking, since the answer should be a fourth word, not one you already named.
Add and subtract any words you like, then read off the nearest neighbor of the sum. Flip a term between plus and minus to feel how each one pushes the result toward or away from a region of the space.
No arithmetic here, just closeness. Pick a word and see which others point most nearly the same way. This is the raw similarity the analogies are built on top of.
This is an illustrative space with five named axes, and the 40 words were placed by hand so the textbook analogies close. The cosine similarities and the a - b + c arithmetic are computed straight from those coordinates, and the nearest word is found by ranking the whole vocabulary, never by special-casing the expected answer.
A learned embedding works the same way geometrically, with one honest difference: its dimensions are not assigned by a person and carry no labels. A model reads enormous amounts of text and discovers a few hundred directions that happen to separate words usefully. No single axis says gender, yet the gender step still exists as some consistent direction through the cloud, which is why the same arithmetic carries over.
Each result is the vector a - b + c over the five coordinates, and each ranking is the cosine between that vector and every word in the vocabulary, sorted high to low.