kw-bench: benchmark for investigative knowledge work using real VAT reconstructions
kw-bench is a new benchmark (arXiv:2610.36405v1) built from real specialist engagements; its first task is a five-year VAT reconstruction across three generations of SAP delivered as raw system extracts with deterministic grading. The authors ran four harnesses (Claude Code, Codex, OpenCode, micro-cc) on the same model (DeepSeek v4.1 Flash) for two hours — OpenCode 0.542 and micro-cc 0.528, while two lab harnesses detached from their home model scored 0.00; by contrast, micro-cc and Claude Code on Claude Opus 5.5 achieved 0.9986 and 1.000 in 22 and 25 minutes, showing frontier models collapse the task and non-frontier performance depends strongly on the harness.