RESEARCH · RESEARCH · #3548
XYBench benchmark released to test models' handling of the 'XY problem' (EMNLP 2026 Best Resource Paper)
XYBench is an 8,115-question benchmark (from Stack Overflow, Stack Exchange, curated examples, and WikiHow-generated prompts) that tests whether assistants recognize when users ask about a mistaken approach (Y) instead of their actual goal (X). Developed with Carnegie Mellon, NVIDIA, and Johns Hopkins and judged automatically by Sonnet 4.5, XYBench evaluated models including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, finding they often answer the asked approach rather than redirecting to solutions for the underlying goal; the paper won the Best Resource Paper award at EMNLP 2026.
KEY POINTS
- XYBench is an 8,115-question benchmark (from Stack Overflow, Stack Exchange, curated examples, and WikiHow-generated prompts) that tests whether assistants recognize when users ask about a mistaken approach (Y) instead of their actual goal (X).
- Developed with Carnegie Mellon, NVIDIA, and Johns Hopkins and judged automatically by Sonnet 4.5, XYBench evaluated models including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, finding they often answer the asked approach rather than redirecting to solutions for the underlying goal; the paper won the Best Resource Paper award at EMNLP 2026.
- XYBench targets a practical failure mode for assistants—recognizing and correcting users' mistaken assumptions—providing a standard way to measure and drive improvements in model troubleshooting and pragmatic understanding.
WHY IT MATTERS
XYBench targets a practical failure mode for assistants—recognizing and correcting users' mistaken assumptions—providing a standard way to measure and drive improvements in model troubleshooting and pragmatic understanding.