Linear Algebra

Dot Product, Cosine, and Projection

Two arrows from the origin, and one number that says how much they agree. Drag them apart and watch the dot product fade to zero where they cross at a right angle, then flip negative as they swing opposite.

linear-algebradot-productcosine-similarityprojectionembeddings

LiveInteractive · drag, toggle, run it

Linear algebra / Two vectors and one number

Dot Product, Cosine, and Projection

The dot product is usually taught as a formula: multiply matching coordinates and add. That hides what it measures. Drag the two arrows below and the same number reads two ways at once: the coordinate sum, and the length of a times the length of b times the cosine of the angle between them. It is large when the arrows point the same way, zero when they meet at a right angle, and negative when they oppose.

Drag the tip of a or b. The shadow of a falling onto b is the projection.

vector a

(2.50, 0.50)

vector b

(1.40, 1.70)

length |a|

2.55

length |b|

2.20

angle theta

39.2 deg

dot a . b

4.35

a . b = (2.50)(1.40) + (0.50)(1.70) = 4.35
|a||b|cos(theta) = (2.55)(2.20)(0.77) = 4.35

vector avector bprojection of a onto b

The projection: a casting its shadow on b

Shine a light straight down onto the line through b. The shadow of a is its projection, the closest point on that line. Its signed length is the dot product divided by the length of b, so the dot product is just this shadow length scaled back up by how long b is. When a and b sit at a right angle the shadow shrinks to nothing, which is the exact moment the dot product is zero.

projection length (a . b)/|b|

1.98

projection vector

(1.26, 1.52)

dot product

1.98 x 2.20 = 4.35

Right now the dot product is positive: the angle is under 90 degrees, so a leans the same way as b and its shadow points along b.

Cosine similarity: the angle alone, length thrown away

Divide the dot product by both lengths and the magnitudes cancel, leaving the cosine of the angle on its own. That is cosine similarity, a number from minus one to one: one when the arrows point the same way, zero at a right angle, minus one when they point opposite. It is exactly the dot product of the two unit vectors, so it sees direction and ignores how long the arrows are.

cosine similarity0.77

Rescaling both arrows to length one changes |a| and |b| and the raw dot product, yet cosine similarity holds at 0.77. Direction is all it ever measured.

Why embeddings rank by cosine

A search model turns each item into a vector. To find the closest match it compares the query's direction to every item's direction, not their lengths, because length often tracks something incidental like how often a word appears. Drag the query arrow and watch the ranking reshuffle by cosine. The top match is the one pointing most nearly the same way, even when a longer vector has a larger raw dot product.

ranked by cosine to query

1river1.00

2ocean1.00

3rain0.89

4desert0.48

5mountain-0.03

The bar is cosine similarity. Notice the longest item arrow is not always on top: raw dot product rewards length, cosine does not.

One number, three readings

The dot product is alignment weighted by size: take how much a points along b (its shadow), then scale by how long b is. That is why a longer b or a longer a both inflate it, and why two perpendicular vectors give zero no matter how long they are.

Cosine similarity is the same idea with the lengths divided out, the angle by itself. That makes it the natural way to ask whether two embeddings mean similar things rather than which one happens to be bigger, which is why search and recommendation systems reach for it first.

Every quantity here is computed from the live coordinates: the dot product as both the coordinate sum and |a||b|cos(theta), the projection as (a . b)/|b| along b, and each ranking as the cosine between unit vectors.