Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3394

Ask2Decide benchmark tests whether LLM agents request decision-relevant information

Ask2Decide is a new benchmark (arXiv:2611.13015v1) of 43 planning tasks that evaluates whether LLM agents ask for missing facts that change optimal plans. The authors collected 79 interaction sessions from 19 human participants, calibrated a simulated operator to match human reply patterns, and found that four frontier LLMs produced infeasible initial plans on 58–98% of tasks when information was missing and uncovered decision-changing facts only marginally more often than irrelevant facts (39–52% vs 32–41%).

KEY POINTS

  1. Ask2Decide is a new benchmark (arXiv:2611.13015v1) of 43 planning tasks that evaluates whether LLM agents ask for missing facts that change optimal plans.
  2. The authors collected 79 interaction sessions from 19 human participants, calibrated a simulated operator to match human reply patterns, and found that four frontier LLMs produced infeasible initial plans on 58–98% of tasks when information was missing and uncovered decision-changing facts only marginally more often than irrelevant facts (39–52% vs 32–41%).
  3. This matters because LLM planning in real-world tasks depends on eliciting the right missing information, and the benchmark exposes a key bottleneck where agents often fail to obtain decision-relevant facts from interaction.

WHY IT MATTERS

This matters because LLM planning in real-world tasks depends on eliciting the right missing information, and the benchmark exposes a key bottleneck where agents often fail to obtain decision-relevant facts from interaction.

SOURCES & TIMELINE

1