RESEARCH · RESEARCH · #2182
Testing LLM-based agent planning against formal PDDL semantics
This arXiv paper evaluates claims that LLM-based agents can perform planning by grounding natural-language tasks in hidden PDDL models that formally determine action applicability and goal achievement. Across five domains, four models, six prompting strategies and three agent architectures, the best configuration solved 78% of solvable tasks offline but only 40% online; unsolvable tasks were recognised in under 10% of episodes, and heavier prompting used 5–22× more tokens without clear benefit.
KEY POINTS
- This arXiv paper evaluates claims that LLM-based agents can perform planning by grounding natural-language tasks in hidden PDDL models that formally determine action applicability and goal achievement.
- Across five domains, four models, six prompting strategies and three agent architectures, the best configuration solved 78% of solvable tasks offline but only 40% online; unsolvable tasks were recognised in under 10% of episodes, and heavier prompting used 5–22× more tokens without clear benefit.
- The paper shows that common LLM agent setups often fail to plan reliably under formal verification and that validated action traces and formal evaluation are needed to substantiate planning claims.
WHY IT MATTERS
The paper shows that common LLM agent setups often fail to plan reliably under formal verification and that validated action traces and formal evaluation are needed to substantiate planning claims.