Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #8437

SWE-bench Verified

Related event timeline, sources and context from the news index.

EVENT TIMELINE

8

RESEARCH · 1 SOURCE · arXiv cs.AI

Termination lag in coding agents (arXiv:2611.14960v1)

The paper (arXiv:2611.14960v1) defines and measures “termination lag” — steps between an agent's first correct patch and the end of its run — by recording and testing every distinct repository patch across five models, two agent frameworks, and three SWE-bench datasets. It finds 38–82% of successful runs produce a correct solution on the initial edit, yet agents often continue working: median post-solution steps account for 35–37% for Qwen3-Coder and 54–60% for Laguna-XS.2 on SWE-bench Verified and Multilingual, and in 1–8% of cases agents later undo a previously correct patch.

6.5

RESEARCH · 1 SOURCE · The Decoder

Co-Design Bench finds coding agents fully complete only ~26% of real user tasks

Researchers (led by Alibaba and several Asian universities) released Co-Design Bench, a new benchmark using real user change requests for web apps; across ten models the best system (GPT-6 Astra) met about 70% of requirements on average but produced a fully correct result in only 26.4% of tasks, meaning human cleanup is still needed on nearly three out of four tasks. The study also shows performance depends strongly on task complexity and the agent harness (execution environment), so authors recommend reporting model and environment together.

8.0

RESEARCH · 1 SOURCE · InfoQ AI, ML & Data Engineering

Google open-sources EnvHarness to adapt training environments for self-evolving agents

Google researchers open-sourced EnvHarness, a framework that wraps static environments via the standard reset()/step() interface and applies composable setup, rule, and link mutations so environments adapt to an agent's weaknesses without changing the underlying environment or evaluators. The release also includes EnvRigger, an automated loop that analyzes agent rollouts to generate and validate targeted environment edits; Google reports consistent performance gains (≈2.9–3.7 percentage points) across benchmarks such as WebArena, SWE-bench Verified, and ALFWorld on models including Gemini, Qwen, and Claude Sonnet.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Behavioral Self-Rewarding (BSR) refines verifiable rewards for software-engineering agents

ArXiv preprint arXiv:2610.12598v1 introduces Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare debugging behavior among trajectories with the same verifier outcome, aggregating pairwise preferences into bounded auxiliary rewards while preserving pass/fail ordering. Experiments on SWE-bench Verified and SWE-bench-Live with Qwen3-14B and Qwen3-32B configurations report consistent improvements, providing additional supervision from collected trajectories without extra environment interactions.

6.0

RESEARCH · 1 SOURCE · Microsoft Research

Microsoft Research Asia open-sources Agent Lightning v1.0, a lightweight harnessed agentic RL framework

Microsoft Research Asia released Agent Lightning v1.0, a roughly 3,500-line open-source implementation of a Harnessed Agentic RL training paradigm that trains on the same agent harness used in deployment via an LLM proxy. The framework runs agents as native Kubernetes jobs, avoids reimplementing agent harnesses, and includes an end-to-end coding-agent example that raised Qwen3.5-9B’s Pass@1 on SWE-bench Verified from 41.8% to 56.4% using about 6,000 training samples.

7.0

RESEARCH · 1 SOURCE · Apple Machine Learning Research

SCLATE: a substrate for continual-learning agent training and evaluation

SCLATE is an execution substrate that unifies scheduling for benchmarks and unmodified agents via an open event scheduler and a hybrid simulated clock, compressing long multi-session scenarios into shorter runtime and recording every model call through an in-container proxy. The authors ported seven benchmarks and ran ten harness/memory configurations across ten models, finding that added memory systems do not consistently outperform native harness memory; they also post-trained Qwen3.5-4B using unmodified harnesses and memory, which reduced file reads, raised SWE-bench Verified pass rate by 16.7 points, and increased held-out MetaClaw accuracy by up to 11.8 points.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Study identifies frequent cost-inefficient behaviors in coding agents and evaluates mitigations

arXiv:2609.30725v1 analyzes 1,200 trajectories from Claude Code and Mini-SWE-Agent on SWE-bench Verified, identifying three recurring cost-inefficient behaviors (subsumed retrieval, similar script generation, test re-execution) that affect 79–98% of tasks and can account for up to 22.75% of task cost. The paper evaluates three mitigations (structure-aware retrieval, agent-synthesized skills, developer-designed skills) over 10k held-out trajectories and reports that structure-aware retrieval can sometimes increase costs (up to 28.14%), agent-synthesized skills give limited, low-level gains, while developer-designed skills reduce costs by up to 41.73%, about twice the maximum gain from agent-synthesized skills.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Privileged Self-Practice (PSP) for multi-turn LLM agents (arXiv:2609.29051v1)

This arXiv preprint identifies a failure mode of on-policy self-distillation (OPSD) in multi-turn agents—training yields overconfident behavior without the underlying information—and proposes Privileged Self-Practice (PSP). PSP keeps privileged information (PI) in the prompt/sampler rather than the loss: when a student fails rollouts, an analyzer model injects a short per-task instruction into the prompt, the task is re-sampled, and the result is trained with the same GRPO objective; across AppWorld and SWE-bench Verified and three student models PSP consistently outperforms OPSD and plain GRPO, improving task-goal completion by up to 65% on AppWorld and resolved rate by up to 61% on SWE-bench Verified.

7.0