kw-bench: benchmark for investigative knowledge work using real VAT reconstructions

KEY POINTS

1. kw-bench is a new benchmark (arXiv:2610.36405v1) built from real specialist engagements; its first task is a five-year VAT reconstruction across three generations of SAP delivered as raw system extracts with deterministic grading.
2. The authors ran four harnesses (Claude Code, Codex, OpenCode, micro-cc) on the same model (DeepSeek v4.1 Flash) for two hours — OpenCode 0.542 and micro-cc 0.528, while two lab harnesses detached from their home model scored 0.00; by contrast, micro-cc and Claude Code on Claude Opus 5.5 achieved 0.9986 and 1.000 in 22 and 25 minutes, showing frontier models collapse the task and non-frontier performance depends strongly on the harness.
3. This matters because kw-bench targets the costly investigative part of knowledge work with deterministic grading and shows that frontier models plus effective harnesses can reduce weeks-long engagements to minutes, changing how real-world capabilities should be evaluated and deployed.