EVOKE: Benchmark Results

An LLM's KV cache is its working memory: every token it has read so far, kept around so it doesn't have to reprocess history on every turn. When that memory fills up, something has to go. EVOKE evicts to host RAM instead of discarding: it saves the exact K/V tensors for whatever gets evicted and splices them back in later with zero recompute. The four tabs below test that idea from different angles: does it beat just guessing what to keep, does it hold up across model architectures, does it save real compute in an actual agent session, and (a separate, complementary result) can eviction pick smarter victims in the first place.

The agent reads a long document, the document gets evicted to free KV memory, then a probe question asks for a detail from the evicted text. Accuracy is the share of probes answered correctly. Two kinds of baseline are on this chart: heuristic eviction (SnapKV and H2O pick what to keep by attention score, recency just keeps the newest tokens) drops whatever it evicts with no way back, while InfLLM offloads evicted text to CPU and retrieves the right passage by similarity search when it's needed. EVOKE takes a third path: it saves the exact K/V tensors for evicted blocks and splices them back in by content identity, without needing a retrieval model or paying any recompute cost.

At the tightest budget (512 blocks) InfLLM beats EVOKE, 81% vs. 51%: its similarity search can pull the right passage from anywhere in the offloaded history, while EVOKE only recovers blocks that get evicted and later re-requested, and fewer of those come back into play when the budget is this tight. That gap closes by 1024 blocks and flips by 2048 (63% vs. 56%), and EVOKE gets there without InfLLM's separate retrieval model, embedding index, or CPU-offload bookkeeping. Heuristic eviction never catches up at any budget tested: dropping content with no way back costs 30 to 60 points of accuracy against either recovery method.