Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #9372

SWE-bench Pro

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

RESEARCH · 1 SOURCE · arXiv cs.AI

ArXiv preprint 'Between Spoilers and Guesswork' on over-/under-specification in coding benchmarks

The arXiv preprint 'Between Spoilers and Guesswork' (arXiv:2610.22760v1) analyzes how task under-specification (causing guesswork) and over-specification (causing spoilers) harm repository-level coding benchmarks. The authors experiment with different specification levels, show that leaking ground-truth hints can introduce over-specification in benchmarks like SWE-bench Pro, FEA-bench, Doc2Feat-bench and Multi-SWE-bench, and propose techniques for producing parsimonious hints which they plan to release.

6.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Study identifies frequent cost-inefficient behaviors in coding agents and evaluates mitigations

arXiv:2609.30725v1 analyzes 1,200 trajectories from Claude Code and Mini-SWE-Agent on SWE-bench Verified, identifying three recurring cost-inefficient behaviors (subsumed retrieval, similar script generation, test re-execution) that affect 79–98% of tasks and can account for up to 22.75% of task cost. The paper evaluates three mitigations (structure-aware retrieval, agent-synthesized skills, developer-designed skills) over 10k held-out trajectories and reports that structure-aware retrieval can sometimes increase costs (up to 28.14%), agent-synthesized skills give limited, low-level gains, while developer-designed skills reduce costs by up to 41.73%, about twice the maximum gain from agent-synthesized skills.

7.0