Seq@K: an evaluation protocol for sequential feedback-driven solving by LMs
This paper introduces Seq@K, a protocol that measures whether a language model can solve a problem within K sequential attempts when given feedback after each failed attempt. Evaluated on 7 benchmarks and 16 frontier models, Seq@K often yields different model rankings than Pass@K, changes Pareto frontiers for model/configuration trade-offs, and reveals behaviors in iterative refinement and sensitivity to feedback design.