RESEARCH · RESEARCH · #521
Contiguity, Not Importance — budgeted repair of stale KV caches after document edits (arXiv:2609.17983v1)
This paper formulates in-place repair of stale KV caches after document edits as budgeted recomputation and compares training-free position-selection policies on a factual RAG benchmark. Across three model families, an edit-local contiguous window at the primary budget recovers ≥0.94 of the post-edit answer margin, substantially outperforms attention-, KV-deviation-, and structural selectors on derived edits, and is 13–21× faster than a full re-prefill, though its advantage vanishes when answer-bearing text moves downstream.
KEY POINTS
- This paper formulates in-place repair of stale KV caches after document edits as budgeted recomputation and compares training-free position-selection policies on a factual RAG benchmark.
- Across three model families, an edit-local contiguous window at the primary budget recovers ≥0.94 of the post-edit answer margin, substantially outperforms attention-, KV-deviation-, and structural selectors on derived edits, and is 13–21× faster than a full re-prefill, though its advantage vanishes when answer-bearing text moves downstream.
- Because KV-cache reuse is widely used to cut inference cost, showing that unconditional edit-local repair cheaply preserves correctness when dependent text remains adjacent offers a practical, much faster alternative to costly full re-prefills.
WHY IT MATTERS
Because KV-cache reuse is widely used to cut inference cost, showing that unconditional edit-local repair cheaply preserves correctness when dependent text remains adjacent offers a practical, much faster alternative to costly full re-prefills.