Diffusion Language Models: All the Words at Once
A normal LLM writes one token per forward pass because each word has to wait for the last. A diffusion language model drafts the whole line and firms it up over a few passes. Race both decode orders on the same tiny model trained for this page, dissolve a sentence into masks, give the denoiser fewer steps and read the damage, then pin both ends of a story and let the middle fill itself in.
Diffusion Language Models: All the Words at Once
Every chatbot you have used writes like a typewriter: one token at a time (a token is the word or word piece an LLM reads and writes in), each waiting on the one before, because the network is only ever asked what comes next. Google's newest Gemma variant ignores that rule. It lays out 256 blank slots, drafts all of them in a single forward pass (one full run of the network), and spends a few more passes firming up whatever it was unsure about. The same trick, at toy scale, is running live in this page.
The race
Both panels below share one model, a small transformer trained on children's stories and embedded in this page. The left panel decodes autoregressively: predict a token, append it, run the whole network again, which is the loop inside GPT style models. The right panel treats the line as a canvas of masked slots (mask means "hidden, to be filled in"), fills every slot with a draft each pass, and locks in only the guesses it is most confident about.
Corrupt, then learn to reverse
Where does a model like this come from? Training never shows it clean text alone. Each example is corrupted first: sample a level t between 0 and 1, hide that fraction of the words behind mask tokens, then grade the model on restoring the originals. Do this at every level, from nearly intact to fully erased, and the model learns to repair any amount of damage. Drag the slider to see what its training data looks like.
Each position holds a fixed lottery ticket, so a word that disappears at t = 0.4 is still gone at every higher t. That is the absorbing state trick from D3PM: masking only ever accumulates, which keeps the training story consistent across corruption levels. Slide to t = 1 and the canvas is all mask, and that is exactly the input generation starts from.
The denoising playground
Generation is just the reverse walk: start from all mask, repair, repeat. How many repair passes to spend is a dial you choose at inference time, not a property baked into the weights. So is the order. The same model will fill slots by confidence, strictly left to right, or at random, and you can trade quality for passes right here. Temperature controls how the model picks each word: zero always takes its single most likely choice, higher values roll dice weighted by its probabilities.
The trick autoregression can't do
A causal model reads with blinders on: attention, the machinery that lets each position consult the others, is masked so a word can only look left. A diffusion LM drops that mask. Every slot attends in both directions, which means you can pin a beginning and an ending and ask for a middle that agrees with both. That is why these models are pitched at code editing, where the surrounding file is exactly a pinned prefix and suffix, and it is why LLaDA beat GPT-4o at completing a poem backwards from its last line: nothing in the architecture privileges left to right.
The pinned words never move; everything gray is regenerated to agree with both sides at once. Pin only the ending and watch the model write toward it, something a left to right decoder cannot do without tricks, because a causal model has no way to let early words depend on later ones.
The speed, and the fine print
The commercial pitch is a single number: tokens per second. Published figures, diffusion first, then autoregressive peers measured by the same third party where available:
Sources: Inception Labs and Artificial Analysis for Mercury and the autoregressive models, DeepMind and Google for the Gemini and Gemma numbers. Gemini Diffusion's figure is sampling speed, excluding some serving overhead.
Scores as a table
| Benchmark | Gemini Diffusion | Flash-Lite |
|---|---|---|
| HumanEval | 89.6 | 90.2 |
| MBPP | 76 | 75.8 |
| LiveCode v6 | 30.9 | 28.5 |
| GPQA | 40.4 | 56.5 |
| MMLU-Lite | 69.1 | 79 |
Code generation holds up and even edges ahead on LiveCodeBench. Knowledge and reasoning lag clearly. Google says the same about DiffusionGemma: it trades quality for speed against Gemma 4 of similar size. Source: DeepMind's Gemini Diffusion model page.
An autoregressive model finishes tokens in order, so it can memoize the attention state of everything already written (the KV cache) and never touch it again. In pure diffusion nothing is final until the last pass, so every pass recomputes attention over the whole block. Fixes exist: BD3-LM makes blocks causal with diffusion only inside each block, Fast-dLLM caches approximately and refreshes when needed, and DiffusionGemma's 256 token blocks are the block idea shipped at scale.
The headline speedup assumes the GPU has idle capacity to spend on one user, which is true on a laptop or a single stream agent. A busy serving cluster already keeps the GPU saturated by batching many users' autoregressive requests together, and there the advantage largely evaporates. Google states this limit itself: the win is biggest for local and single user inference.
Where this came from
Google shipped the biggest version, but the idea has a decade of lineage and none of it started there.
- 2021D3PM · Austin et al., GoogleDiscrete diffusion over tokens, including the absorbing [MASK] state this page's slider shows. The recipe that everything below refines.
- 2022Diffusion-LM · Li et al., StanfordRuns diffusion on continuous word embeddings instead of tokens, and shows the payoff is fine grained control over what gets generated.
- 2023SEDD · Lou et al., StanfordScore entropy training makes discrete diffusion competitive: it beat GPT-2 while using far fewer network evaluations per sample.
- 2024MDLM · Sahoo et al., CornellStrips masked diffusion to a weighted average of masked language modeling losses, plus semi autoregressive sampling past the context window.
- Feb 2025LLaDA · Nie et al., Renmin U. and Ant GroupAn 8B diffusion LM trained from scratch that holds its own against LLaMA3 8B. The training objective of the toy model on this page.
- Feb 2025Mercury · Inception LabsThe first commercial scale diffusion LLM, sold on exactly one number: tokens per second on code.
- May 2025Gemini Diffusion · Google DeepMindDeepMind's experimental demo at I/O, source of the benchmark chart above.
- Aug 2025Dream 7B · HKU and HuaweiShows you can warm start a diffusion LM from an autoregressive one instead of training from scratch.
- 2026DiffusionGemma · GoogleOpen weights, 26B mixture of experts (only a slice of the network runs per token, so capacity is cheap), generating in 256 token blocks.