RESEARCH · RESEARCH · #3040
Stanford study: 44% of 50 widely cited LLM claims have expired; mean lifespan 15.7 months
Stanford researchers Owen Queen and James Zou re-tested 50 widely cited empirical claims about large language models using 20 models released between July 2024 and September 2026 and found 44% of the claims have expired, 34% have weakened, 16% still hold, and 6% strengthened; among 22 findings with an estimated expiration date the average lifespan was 15.7 months. Reasoning-related failures expired fastest, behavior/preferences claims were more persistent, results varied widely across contemporary models (e.g., GPT-6.1 Sol and Qwen3.7-Flash showed opposite outcomes on citation fabrication), and the authors recommend reporting model, version, date and publishing prompts/code to enable retesting.
KEY POINTS
- Stanford researchers Owen Queen and James Zou re-tested 50 widely cited empirical claims about large language models using 20 models released between July 2024 and September 2026 and found 44% of the claims have expired, 34% have weakened, 16% still hold, and 6% strengthened; among 22 findings with an estimated expiration date the average lifespan was 15.7 months.
- Reasoning-related failures expired fastest, behavior/preferences claims were more persistent, results varied widely across contemporary models (e.g., GPT-6.1 Sol and Qwen3.7-Flash showed opposite outcomes on citation fabrication), and the authors recommend reporting model, version, date and publishing prompts/code to enable retesting.
- The finding shows that empirical claims about LLMs change rapidly across model releases, so evaluations must be time‑ and model‑specific and reproducible to remain reliable.
WHY IT MATTERS
The finding shows that empirical claims about LLMs change rapidly across model releases, so evaluations must be time‑ and model‑specific and reproducible to remain reliable.