Stanford study finds 44% of LLM claims have expired

stanford researchers owen queen and james zou examined 50 widely cited claims about large language models against 20 models released between july 2024 and september 2026 by eight different developers. they found that 44 percent of the claims have already expired, with an average lifespan of just 15.7 months. claims about what models can't do expire fastest, especially those related to reasoning. in contrast, claims about preferences and tendencies prove longer-lived. the study warns against reading these results as an all-clear, as some issues like hallucinated references and social biases persist. the findings impact the reliability of llm research and development.