RESEARCH · RESEARCH · #2184
Author-in-the-loop agentic reproducibility audit deployed across 36 conference papers
arXiv preprint arXiv:2610.22465v1 presents an author-in-the-loop collaborative audit: authors provide a structured checklist of reproduction instructions which guides an AI agent in a two-stage GitHub workflow that requests clarifications, reassesses, and separates assessment of reproduction from requests-for-repair. The system was deployed across 36 conference papers and 133 experimental tables/figures, raised 20 issues after human review that were later closed by further checks, though full reproduction remained unverified in some cases; the paper also compares audits using complete versus blank checklists to show how author declarations shape agent investigations.
KEY POINTS
- arXiv preprint arXiv:2610.22465v1 presents an author-in-the-loop collaborative audit: authors provide a structured checklist of reproduction instructions which guides an AI agent in a two-stage GitHub workflow that requests clarifications, reassesses, and separates assessment of reproduction from requests-for-repair.
- The system was deployed across 36 conference papers and 133 experimental tables/figures, raised 20 issues after human review that were later closed by further checks, though full reproduction remained unverified in some cases; the paper also compares audits using complete versus blank checklists to show how author declarations shape agent investigations.
- Provides an empirical, practical workflow showing how combining author declarations and AI agents can scale and focus reproducibility checks in peer review, helping find and resolve reproducibility problems.
WHY IT MATTERS
Provides an empirical, practical workflow showing how combining author declarations and AI agents can scale and focus reproducibility checks in peer review, helping find and resolve reproducibility problems.