Paper: Using model solution transcripts to assess capabilities after benchmark saturation
The paper (arXiv:2610.20667v1) studies whether full transcripts of correctly solved benchmark items can predict model capabilities after benchmark scores saturate. Using retrospective simulations, the authors evaluate methods (pairwise judging, single-transcript absolute scoring, deterministic transcript features/embeddings) and find transcripts retain capability signal beyond accuracy — pairwise judging reduced prediction error 5.3× versus using saturated scores — though some protocols fail to generalize across distribution shifts (e.g., underestimating early-2025 reasoning models).