RESEARCH · RESEARCH · #2157
Taste-Bench benchmark shows AI agents struggle with long-term decision forks
Researchers at Microsoft and City University of Hong Kong released Taste-Bench, a 502-question benchmark that evaluates an agent's 'taste'—the ability to choose between branching long-term paths by freezing 'decision forks' in agent trajectories. The best model tested (GPT-5.6 Sol) averaged 59.7% accuracy, most models scored far lower, and a distilled student from Qwen3.6-27B improved judgment accuracy from 30% to 47.9% and raised agent success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%.
KEY POINTS
- Researchers at Microsoft and City University of Hong Kong released Taste-Bench, a 502-question benchmark that evaluates an agent's 'taste'—the ability to choose between branching long-term paths by freezing 'decision forks' in agent trajectories.
- The best model tested (GPT-5.6 Sol) averaged 59.7% accuracy, most models scored far lower, and a distilled student from Qwen3.6-27B improved judgment accuracy from 30% to 47.9% and raised agent success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%.
- Taste-Bench exposes a core weakness—predicting delayed consequences of choices—providing a focused benchmark and a path (distillation/advice) to measure and improve agents' long-term planning.
WHY IT MATTERS
Taste-Bench exposes a core weakness—predicting delayed consequences of choices—providing a focused benchmark and a path (distillation/advice) to measure and improve agents' long-term planning.