RESEARCH · RESEARCH · #2703
When Retrieval Is Not Enough: obsolete retrieved facts can degrade agent accuracy
arXiv:2610.36308v1 analyzes the single-hop FactConsolidation task in MemoryAgentBench and shows that retrieving the newest statement is not sufficient for correct answers. With Qwen2.5-7B-Instruct on a 6K-history setup, BM25 accuracy falls from 45% to 32% when the retrieved evidence budget increases from 64 to 512 tokens despite a gold audit finding the newest statement in each retrieved set; substituting an unrelated statement with an obsolete competing statement drops accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B (all 50 harmful paired flips were classified as stale), and a 32K-history replication shows BM25-64 outperforming BM25-512 (69% vs 58%).
KEY POINTS
- arXiv:2610.36308v1 analyzes the single-hop FactConsolidation task in MemoryAgentBench and shows that retrieving the newest statement is not sufficient for correct answers.
- With Qwen2.5-7B-Instruct on a 6K-history setup, BM25 accuracy falls from 45% to 32% when the retrieved evidence budget increases from 64 to 512 tokens despite a gold audit finding the newest statement in each retrieved set; substituting an unrelated statement with an obsolete competing statement drops accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B (all 50 harmful paired flips were classified as stale), and a 32K-history replication shows BM25-64 outperforming BM25-512 (69% vs 58%).
- This shows agent memory systems must detect and resolve obsolete or competing statements, not just retrieve the newest evidence, to avoid being misled by stale facts.
WHY IT MATTERS
This shows agent memory systems must detect and resolve obsolete or competing statements, not just retrieve the newest evidence, to avoid being misled by stale facts.