RESEARCH · RESEARCH · #3118
ArXiv paper tests iterative writer–grader loops for end‑to‑end data‑science projects
arXiv:2611.05548v1 evaluates whether agentic language models can complete end‑to‑end data‑science projects by running a 'writer' model to create projects and a 'grader' model to evaluate them, repeating up to three revision rounds. Across 49 writer–grader model pairs (seven models from Anthropic, OpenAI, Google, DeepSeek, and Z.AI) the authors ran 147 initial submissions (107 executed and rendered); average scores rose from 59% to 76% after three rounds, with per‑model improvements varying widely (e.g., Anthropic Haiku 4.5 +46 percentage points, Opus 5.5 +3).
KEY POINTS
- arXiv:2611.05548v1 evaluates whether agentic language models can complete end‑to‑end data‑science projects by running a 'writer' model to create projects and a 'grader' model to evaluate them, repeating up to three revision rounds.
- Across 49 writer–grader model pairs (seven models from Anthropic, OpenAI, Google, DeepSeek, and Z.AI) the authors ran 147 initial submissions (107 executed and rendered); average scores rose from 59% to 76% after three rounds, with per‑model improvements varying widely (e.g., Anthropic Haiku 4.5 +46 percentage points, Opus 5.5 +3).
- Shows that iterative grading plus revision substantially improves multi‑model end‑to‑end data‑science performance and that gains depend heavily on which model plays which role.
WHY IT MATTERS
Shows that iterative grading plus revision substantially improves multi‑model end‑to‑end data‑science performance and that gains depend heavily on which model plays which role.