RESEARCH · RESEARCH · #2665
AI2 releases 'Old Ideas, Novel Problems' benchmark showing LLM novelty judges are unstable
AI2 published a study and public benchmark, 'Old Ideas, Novel Problems', evaluating how reliably large language models judge the novelty of research ideas. Using an automated pipeline built on ICLR 2026 peer reviews, two test settings (Human-Only: 154 high-novel / 145 low-novel ideas; Human+Generated: 154 human ideas paired with 154 LLM-generated ideas), and six LLM judges (GPT-5.1/5.2/5.4 and Claude Sonnet/Opus variants), they find judgments are highly sensitive to small setup changes (a prompt tweak shifted accuracy by >50 percentage points), retrieval and extra reasoning do not consistently help, and two purpose-built systems underperformed a simple prompted Claude Opus 4.6 baseline.
KEY POINTS
- AI2 published a study and public benchmark, 'Old Ideas, Novel Problems', evaluating how reliably large language models judge the novelty of research ideas.
- Using an automated pipeline built on ICLR 2026 peer reviews, two test settings (Human-Only: 154 high-novel / 145 low-novel ideas; Human+Generated: 154 human ideas paired with 154 LLM-generated ideas), and six LLM judges (GPT-5.1/5.2/5.4 and Claude Sonnet/Opus variants), they find judgments are highly sensitive to small setup changes (a prompt tweak shifted accuracy by >50 percentage points), retrieval and extra reasoning do not consistently help, and two purpose-built systems underperformed a simple prompted Claude Opus 4.6 baseline.
- If LLMs are to assist scientific discovery, we need dependable ways to tell whether proposed ideas are genuinely novel, and this benchmark shows current novelty-judging setups are far from reliable.
WHY IT MATTERS
If LLMs are to assist scientific discovery, we need dependable ways to tell whether proposed ideas are genuinely novel, and this benchmark shows current novelty-judging setups are far from reliable.