Ask2Decide benchmark tests whether LLM agents request decision-relevant information
Ask2Decide is a new benchmark (arXiv:2611.13015v1) of 43 planning tasks that evaluates whether LLM agents ask for missing facts that change optimal plans. The authors collected 79 interaction sessions from 19 human participants, calibrated a simulated operator to match human reply patterns, and found that four frontier LLMs produced infeasible initial plans on 58–98% of tasks when information was missing and uncovered decision-changing facts only marginally more often than irrelevant facts (39–52% vs 32–41%).