RESEARCH · RESEARCH · #2974
CIRCE predicts LLM task difficulty from constraint structure
CIRCE is a method that uses a cheap LLM to build a Constraint Intermediate Representation which deterministically compiles into a constraint graph, and then applies a GNN encoder to predict whether an LLM will solve a given problem. Evaluated across seven reasoning benchmarks (including SAT, logic, and math) and four LLMs, a randomly initialized GNN improved negative log-likelihood (0.465 → 0.423) and AUROC (0.691 → 0.727) versus the strongest baselines reported.
KEY POINTS
- CIRCE is a method that uses a cheap LLM to build a Constraint Intermediate Representation which deterministically compiles into a constraint graph, and then applies a GNN encoder to predict whether an LLM will solve a given problem.
- Evaluated across seven reasoning benchmarks (including SAT, logic, and math) and four LLMs, a randomly initialized GNN improved negative log-likelihood (0.465 → 0.423) and AUROC (0.691 → 0.727) versus the strongest baselines reported.
- Preemptively estimating task difficulty from constraint structure can improve model routing, test-time compute allocation, and indicate when additional tools are needed.
WHY IT MATTERS
Preemptively estimating task difficulty from constraint structure can improve model routing, test-time compute allocation, and indicate when additional tools are needed.