PCA vs t-SNE: What Each One Lies About
Both flatten high-dimensional data into a picture, and both lie. PCA keeps the global shape but cannot unbend a curve; t-SNE draws clean clusters whose sizes and spacing mean nothing. Run both and see where each one cheats.
PCA vs t-SNE: What Each One Lies About
High-dimensional data has to be squashed to two dimensions before you can see it, and every method distorts something on the way down. PCA finds the directions of greatest variance and projects onto them: linear, global, honest about distance, but unable to bend. t-SNE matches the neighbourhood structure instead, drawing crisp clusters that look definitive while quietly throwing away cluster size and the distances between groups. Run both on the same data below and you can see exactly where each one cheats.
Points sampled along a curved 1D path (an S laid out in 3D) then embedded in 10D with small noise. The data is intrinsically a line that has been bent. PCA can only rotate and flatten, so the two ends of the S collapse on top of each other; t-SNE follows neighbors along the curve and unrolls it.
The same points, two projections
Trustworthiness (k = 8 neighbours, higher is better) asks: of the points that look like near neighbours in the 2D picture, how many were genuinely close in the original space. It rewards t-SNE for preserving local structure, which is exactly what t-SNE optimises for.
t-SNE has a free parameter that reshapes the answer
Perplexity sets roughly how many neighbours each point tries to stay close to. It picks the Gaussian bandwidth per point through a binary search so the effective neighbour count matches this target. Changing it rebuilds the embedding from scratch (watch the plot above reset). Low perplexity fractures the data into many tiny islands; high perplexity blurs real clusters together. There is no single correct value, which is the point: the same data yields different pictures depending on a knob you set by hand. PCA has no such knob.
t-SNE is random; PCA is not
Press Reset embedding a few times. The PCA panel never moves: it is a deterministic eigen decomposition of the covariance matrix, so the projection is fixed by the data alone. The t-SNE panel lands differently each time because it starts from a random initial cloud and descends a non-convex objective. Cluster shapes, orientations, and which side each group ends up on are artifacts of that random start, not facts about the data. Reseed data + init draws a fresh dataset too, so you can confirm the structure is stable while the t-SNE rendering of it is not.
The 2D axes are the top two eigenvectors of the covariance matrix, the directions along which the data varies most. Distances and the overall layout are trustworthy projections of the real geometry. The cost: anything that lives along a low-variance direction gets squashed, and a curved manifold cannot be unbent by a linear map, so the S-curve folds onto itself and clusters that separate only along a thin direction can hide.
It minimises the KL divergence between high-D affinities p and low-D Student-t affinities q, so near neighbours stay near and everything else is free to move. That gives clean, well-separated clusters. The caveats are firm: cluster size is meaningless (dense and diffuse groups are inflated to similar blobs), the distance between two clusters is not proportional to their real separation, and the result depends on perplexity and the random seed. Read membership, never geometry.