EVOKE: Evicting and Recovering the KV Cache
An agent session outgrows its GPU memory budget fast. Watch every cache block earn a relevance score from how well it still matches the task in focus, see the lowest scorers evicted to host RAM, then spliced back by content identity when the history is re-sent, so almost nothing is recomputed.
EVOKE: Evicting and Recovering the KV Cache
An agent re-sends its whole conversation every turn, but the GPU can only hold so much attention state. EVOKE treats the KV cache like an OS treats memory. Every block carries a relevance score: how well it still matches the task in focus, blended with recency, with floors under conversation turns. At turn ends the lowest scorers are evicted to host RAM, and when the same bytes come back, the saved tensors splice in instead of being recomputed.
Run to the end at the current budget. The contest: prompt tokens re-decoded (left, lower is cheaper) against what stays resident on the GPU after the final turn (right, lower is leaner). Every policy spikes through the full working set mid-turn; the difference is what remains.
Discard respects the budget but re-decodes history every turn. No-eviction decodes cheaply (an intact cache is a perfect prefix cache) but its footprint grows with the session. EVOKE holds both: budget enforced, and recovery makes the resend nearly free.
The score bars above come from coherence and recency, the weights the headline benchmark arm uses. EVOKE can also read a second signal: the attention the model still pays each block. Each generated token runs one lookup over everything cached, its query vector is dotted against every cached key, and softmax turns the raw scores into a row of percentages that sums to 1. That row is the model's own record of where it looked while choosing the next token. Attention, From the Ground Up builds that machinery (queries, keys, values, softmax) interactively; this page only needs its output. Below, one frozen decode step over nine cached tokens:
The forked llama.cpp sums each decode step's row inside a block's token range, so the block gets its share of the model's attention. An attention-weighted config (attention 0.5, coherence 0.3, recency 0.2) blends that share into the score; the headline config leaves attention at zero weight and scores on coherence and recency alone. Either way, here the work is clearly centered on file: app.py, so that block stays while a block left cold for long enough becomes the next eviction.
- Task coherence (the dominant weight, 0.60): each block's embedding is compared against a running task-focus embedding, so a block earns its score by still matching what the session is working on. This is the signal driving the score bars and rescore phase above; switch tasks and the focus drifts, so the old task's blocks go cold within a turn or two.
- Recency (0.40): an exponential decay that keeps just-arrived blocks safe and acts as a stability prior, so one spike cannot thrash the cache.
- The model's own attention (off by default): the forked llama.cpp can copy each decode step's post-softmax attention row into a host buffer and hand each block its share. An attention-weighted variant blends it into the score; the headline arm leaves it at zero.
- Protections: the first tokens are attention sinks and never leave, user turns floor at 0.60 and assistant turns at 0.50 so the conversation backbone outlives tool output, and a freshly spliced block cannot be re-evicted in the turn it came back.
- Score bars under every chip: eviction takes the lowest score, not the oldest block. At the default budget, turn 3's first victim is a tool ack from the turn before, while the turn-1 user message outlives it, because the ack no longer matches the task in focus.
- Blue splices: an evicted block whose exact bytes reappear at the same position comes back as a tensor copy, not a forward pass. Identity match, never similarity search; that distinction is what keeps this from being retrieval-augmented generation.
- Orange decodes: the only blocks that cost compute. Under EVOKE they are the genuinely new content plus one echo per assistant turn.
- The system block is an attention sink: pinned at score 1.00, it never leaves.
- Each assistant block re-decodes once: the client echoes a re-templated copy of the reply, which never byte-matches the raw emit (whitespace and tool-call JSON re-rendering differ). After that one decode it is canonical and recoverable.
- Drop the budget to 3 and watch the discard arm pay for it; raise it to 12 and the policies converge because nothing needs to leave.
This simulation mirrors a live run: a real coding agent (opencode, 9 tools) built a notes webapp through an EVOKE server running Qwen3-8B on a 16 GB GPU at a 2,048-token budget. The agent session saw 17,397 prompt tokens but decoded only 9,719; all 59 evicted blocks spliced back recompute-free with zero identity mismatches. The eviction control re-decoded every prompt in full each turn, and the no-eviction control finished with 10,952 tokens resident and growing. The mechanism behind the blue splice is a pair of C++ primitives added to llama.cpp that copy a block's K/V tensors off-GPU at eviction and splice them back with positions re-anchored.