RESEARCH · RESEARCH · #3563
WordInferBench: benchmark tests agents' ability to infer Word document edits from context
WordInferBench is a new benchmark of 280 Word-document editing instances, each provided in three instruction conditions—descriptive, referential, and inferential—designed so target formatting can be recovered from an in-document exemplar. The authors evaluate six recent coding-agent systems (data from real Word files via Common Crawl) and report that the top system, Claude Code with Opus 4.6, scores 62.9% task pass rate on descriptive variants but only 40.7% on inferential variants, revealing a substantial drop when agents must infer formatting from context.
KEY POINTS
- WordInferBench is a new benchmark of 280 Word-document editing instances, each provided in three instruction conditions—descriptive, referential, and inferential—designed so target formatting can be recovered from an in-document exemplar.
- The authors evaluate six recent coding-agent systems (data from real Word files via Common Crawl) and report that the top system, Claude Code with Opus 4.6, scores 62.9% task pass rate on descriptive variants but only 40.7% on inferential variants, revealing a substantial drop when agents must infer formatting from context.
- It quantifies a concrete gap in current agents' ability to infer editing attributes from document context and provides a reproducible dataset to measure and drive improvements.
WHY IT MATTERS
It quantifies a concrete gap in current agents' ability to infer editing attributes from document context and provides a reproducible dataset to measure and drive improvements.
SOURCES & TIMELINE
1arXiv:2611.15189v1 Announce Type: new Abstract: Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consi…
arXiv:2611.15049v1 Announce Type: new Abstract: Humanity has long sought to explain natural phenomena. Reliable prediction of experimental outcomes is a central test of our understanding of the processes that produce them. If artificial general intelligence (AGI) is to surpass human scientific capabilities, it must also predict experimental outcomes more reliably than humans. Such capabilities could accelerate scient…
arXiv:2611.14949v1 Announce Type: new Abstract: Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts. Can large language model (LLM) agents automate this workflow accurately and efficiently? To investigate this qu…