RESEARCH · RESEARCH · #2327
LLM planning failures linked to predicate semantics; h_add predicts structural difficulty (arXiv:2610.26964v1)
This arXiv preprint evaluates seven STRIPS domains and finds that the zero-training delete-relaxation heuristic h_add predicts instance-level structural difficulty across held-out domains much better than learned alternatives (mean Spearman |ρ|=0.931 vs 0.707 for ridge regression and 0.400 for ARC). The authors also show that progressively renaming predicates (keeping states, transitions, and optimal plans unchanged) reduces valid LLM planning success from 67.5% to 40.0% without increasing refusals, indicating that semantic interpretability of symbols — not structural complexity — often drives LLM planning failures and that hybrid routing must model LLM semantic sensitivity.
KEY POINTS
- This arXiv preprint evaluates seven STRIPS domains and finds that the zero-training delete-relaxation heuristic h_add predicts instance-level structural difficulty across held-out domains much better than learned alternatives (mean Spearman |ρ|=0.931 vs 0.707 for ridge regression and 0.400 for ARC).
- The authors also show that progressively renaming predicates (keeping states, transitions, and optimal plans unchanged) reduces valid LLM planning success from 67.5% to 40.0% without increasing refusals, indicating that semantic interpretability of symbols — not structural complexity — often drives LLM planning failures and that hybrid routing must model LLM semantic sensitivity.
- The paper shows existing symbolic heuristics already estimate structural planning difficulty well and that improving hybrid LLM+planner systems requires modeling LLM sensitivity to predicate semantics rather than better structural predictors.
WHY IT MATTERS
The paper shows existing symbolic heuristics already estimate structural planning difficulty well and that improving hybrid LLM+planner systems requires modeling LLM sensitivity to predicate semantics rather than better structural predictors.