Next-Token Prediction & Sampling
A language model writes one token at a time, drawing each from a probability distribution and feeding its own choice back in. Turn the temperature, top-k, and top-p knobs and watch the same distribution reshape under your hands.
Next-Token Prediction & Sampling
A language model never plans a sentence. It looks at the running text, produces a probability over every word that could come next, picks one, appends it, and then reads its own output back in to predict again. The knobs people argue about, temperature, top-k, and top-p, are just different ways of choosing from that one distribution. Run the loop by hand below and feel each knob bend the odds.
Temperature rescales confidence. The raw scores are logits, here the log of each count-based probability. Dividing them by a small T before the softmax stretches the gaps, so the top word dominates and the text turns rigid and repetitive. A large T squashes the gaps toward uniform, so rare words slip in and the text drifts into nonsense.
Top-k and top-p both prune the tail before sampling. Top-k keeps a fixed number of candidates; top-p keeps however many it takes to cover a fixed share of the probability mass, so it adapts: a confident step keeps few words, an uncertain step keeps many. After either prune, the survivors are renormalized so they sum back to one.
The whole game is a trade between coherence and diversity. Low temperature with tight pruning is safe and dull; high temperature with no pruning is surprising and often broken. Useful sampling lives in the middle.