RESEARCH · RESEARCH · #2884
Seq@K: an evaluation protocol for sequential feedback-driven solving by LMs
This paper introduces Seq@K, a protocol that measures whether a language model can solve a problem within K sequential attempts when given feedback after each failed attempt. Evaluated on 7 benchmarks and 16 frontier models, Seq@K often yields different model rankings than Pass@K, changes Pareto frontiers for model/configuration trade-offs, and reveals behaviors in iterative refinement and sensitivity to feedback design.
KEY POINTS
- This paper introduces Seq@K, a protocol that measures whether a language model can solve a problem within K sequential attempts when given feedback after each failed attempt.
- Evaluated on 7 benchmarks and 16 frontier models, Seq@K often yields different model rankings than Pass@K, changes Pareto frontiers for model/configuration trade-offs, and reveals behaviors in iterative refinement and sensitivity to feedback design.
- Seq@K matters because it better matches agentic, feedback-driven workflows and can change which models and configurations are chosen for systems that iteratively refine outputs.
WHY IT MATTERS
Seq@K matters because it better matches agentic, feedback-driven workflows and can change which models and configurations are chosen for systems that iteratively refine outputs.