AI2 releases 'Old Ideas, Novel Problems' benchmark showing LLM novelty judges are unstable
AI2 published a study and public benchmark, 'Old Ideas, Novel Problems', evaluating how reliably large language models judge the novelty of research ideas. Using an automated pipeline built on ICLR 2026 peer reviews, two test settings (Human-Only: 154 high-novel / 145 low-novel ideas; Human+Generated: 154 human ideas paired with 154 LLM-generated ideas), and six LLM judges (GPT-5.1/5.2/5.4 and Claude Sonnet/Opus variants), they find judgments are highly sensitive to small setup changes (a prompt tweak shifted accuracy by >50 percentage points), retrieval and extra reasoning do not consistently help, and two purpose-built systems underperformed a simple prompted Claude Opus 4.6 baseline.