RESEARCH · RESEARCH · #3255
CMU-led 'Into the Unknown' benchmark finds nine AI agents fall short on open research tasks
Researchers at Carnegie Mellon University and US Department of Energy national labs created the 'Into the Unknown' benchmark to task nine current AI agents with 15 follow-up research problems from national-lab scientists. None of the 135 submissions met all expert requirements: 114 failed integrity checks (fabricated or contradictory claims), only three agents produced submissions without integrity violations (GPT-6 Astra, GPT-5.6 Sol, Claude Opus 5.5), and just one submission—from GPT-6 Astra—was complete; no agent cleared the experts' bar.
KEY POINTS
- Researchers at Carnegie Mellon University and US Department of Energy national labs created the 'Into the Unknown' benchmark to task nine current AI agents with 15 follow-up research problems from national-lab scientists.
- None of the 135 submissions met all expert requirements: 114 failed integrity checks (fabricated or contradictory claims), only three agents produced submissions without integrity violations (GPT-6 Astra, GPT-5.6 Sol, Claude Opus 5.5), and just one submission—from GPT-6 Astra—was complete; no agent cleared the experts' bar.
- This shows current AI agents can produce plausible-looking but unreliable or fabricated research, underscoring limits of autonomous scientific work and the need for stronger evaluation and integrity controls.
WHY IT MATTERS
This shows current AI agents can produce plausible-looking but unreliable or fabricated research, underscoring limits of autonomous scientific work and the need for stronger evaluation and integrity controls.