Claude Opus

2026-10-23

RESEARCH · 1 SOURCE · arXiv cs.AI

Agent4Sci testbed (arXiv:2610.28763v1) evaluates frontier coding agents on 20 ML‑for‑science benchmarks

The paper introduces Agent4Sci, a long‑horizon testbed that measures how well current coding agents can improve published best models across 20 scientific ML benchmarks in 11 domains. Empirically, frontier agents (e.g., Claude Opus) improve published baselines (Claude Opus: 10.4% mean relative improvement), but gains mainly come from ML‑engineering tricks; simple plug‑in skills for ideation and experiment management boost Claude Opus to 20.1% and Codex to 8.1%. Code and data are available on GitHub and Hugging Face.

7.0

2026-10-05

RESEARCH · 1 SOURCE · arXiv cs.AI

When Terminal-Agent Training Stalls — arXiv:2610.02405v1

This arXiv paper studies failures in training terminal agents using frontier models (e.g., Claude Opus) as meta-agents to generate tasks and verifiers, identifying three failure modes: benchmark invalidity, harness brittleness, and reward misalignment. The authors show prompt redesign and context extension can increase baseline solvability 5.6×, report that a 9B model saturates at 81.3% mean pass@2 within 20 steps on Claude Opus–generated tasks, and that adding hard tasks can drop mean pass@2 to 20.6% without changing training configuration, arguing for solvability-band calibration, verifier audits, and infrastructure error accounting as first-class evaluation criteria.

7.0

2026-09-30

CODING · 1 SOURCE · InfoQ AI, ML & Data Engineering

CodeScene agents refactor 300K-line Street Fighter III codebase in three weeks

CodeScene published a case study in which coding agents refactored a 300,000-line C codebase (a decompiled Street Fighter III: 3rd Strike) over three weeks for roughly $4,000 in token costs, producing 2,903 commits across 726 files, modifying 252,055 lines and raising a Code Health score from 5.6 to 10.0. The project used a deterministic CodeHealth MCP Server score and a replay-trace harness to check behavioral correctness, favored Claude Opus (Claude Code) over Codex (Sol), produced a 22-recipe refactoring playbook, and prompted practitioner debate about scope, generalizability and real-world safety/performance implications.

7.0

2026-09-18

CODING · 1 SOURCE · InfoQ AI, ML & Data Engineering

DoorDash uses multi-agent LLMs to automate cleanup of 60,000 feature flags

DoorDash built a multi-agent LLM workflow (using Google’s Agent Development Kit and Claude models) to identify and remove stale feature flags across ~623 repositories and ~60,000 flags. In a 50-flag evaluation the system produced usable pull requests for 45 flags, averaging 13.8 minutes and $4.79 per cleanup, running agents in isolated Git worktrees, validating builds/tests/coverage, and integrating with Jira and the Model Context Protocol (MCP).

7.0