Stanford study: 44% of 50 widely cited LLM claims have expired; mean lifespan 15.7 months
Stanford researchers Owen Queen and James Zou re-tested 50 widely cited empirical claims about large language models using 20 models released between July 2024 and September 2026 and found 44% of the claims have expired, 34% have weakened, 16% still hold, and 6% strengthened; among 22 findings with an estimated expiration date the average lifespan was 15.7 months. Reasoning-related failures expired fastest, behavior/preferences claims were more persistent, results varied widely across contemporary models (e.g., GPT-6.1 Sol and Qwen3.7-Flash showed opposite outcomes on citation fabrication), and the authors recommend reporting model, version, date and publishing prompts/code to enable retesting.