Testing LLM-based agent planning against formal PDDL semantics
This arXiv paper evaluates claims that LLM-based agents can perform planning by grounding natural-language tasks in hidden PDDL models that formally determine action applicability and goal achievement. Across five domains, four models, six prompting strategies and three agent architectures, the best configuration solved 78% of solvable tasks offline but only 40% online; unsolvable tasks were recognised in under 10% of episodes, and heavier prompting used 5–22× more tokens without clear benefit.