Paper separates replication, measurement sensitivity, and persistence in hosted-LLM evaluation; Gemini 3.1 deficit replicates but can reverse under rebuilt configs
An arXiv preprint (arXiv:2609.22478v1) distinguishes three validation questions—replication, measurement sensitivity, and persistence—when evaluating hosted language models using the Regent Chess environment. The authors replicate a previously reported Gemini 3.1 Flash‑Lite deficit on fresh games but show that rebuilding the evaluation/inference configuration under the same public identifier and comparing across identifiers (Gemini 3.1 vs Gemini 3.7) can materially change or reverse endpoints; the potential contribution of the serving period remains unresolved. The paper recommends explicitly indexing behavioural claims by tested identifier, serving period, measurement instrument, and inference configuration.