RESEARCH · RESEARCH · #3394
Ask2Decide benchmark tests whether LLM agents request decision-relevant information
Ask2Decide is a new benchmark (arXiv:2611.13015v1) of 43 planning tasks that evaluates whether LLM agents ask for missing facts that change optimal plans. The authors collected 79 interaction sessions from 19 human participants, calibrated a simulated operator to match human reply patterns, and found that four frontier LLMs produced infeasible initial plans on 58–98% of tasks when information was missing and uncovered decision-changing facts only marginally more often than irrelevant facts (39–52% vs 32–41%).
KEY POINTS
- Ask2Decide is a new benchmark (arXiv:2611.13015v1) of 43 planning tasks that evaluates whether LLM agents ask for missing facts that change optimal plans.
- The authors collected 79 interaction sessions from 19 human participants, calibrated a simulated operator to match human reply patterns, and found that four frontier LLMs produced infeasible initial plans on 58–98% of tasks when information was missing and uncovered decision-changing facts only marginally more often than irrelevant facts (39–52% vs 32–41%).
- This matters because LLM planning in real-world tasks depends on eliciting the right missing information, and the benchmark exposes a key bottleneck where agents often fail to obtain decision-relevant facts from interaction.
WHY IT MATTERS
This matters because LLM planning in real-world tasks depends on eliciting the right missing information, and the benchmark exposes a key bottleneck where agents often fail to obtain decision-relevant facts from interaction.