The shelf life of LLM findings

How long does a published finding about LLMs remain true? We revisited 50 widely cited findings about LLMs across 20 models. Within 15 months of publication, most of the LLM claims we tested either no longer held or showed substantially diminished effects. What we know about AI may have a surprisingly short shelf life.

In 2023, Walters and Wilder asked ChatGPT to write short literature reviews and then tried to find every work it cited. At the time, ChatGPT used GPT-3.5, and 55% of its citations were fake (C11). We sent the same prompts to recent models (GPT-6.1-Sol and Claude Sonnet 5.5) and found that they rarely output fabricated citations, though some models still do. We cannot assume claims made on yesterday’s LLMs hold on today’s more capable models.

We test a broad range of 50 LLM claims, from sycophancy and jailbreaks to long-context capabilities and cultural biases. By analyzing 20 models, released from July 2024 to September 2026 across eight different model developers, we see how claims change as a function of model release date. Our models span large and small, open- and closed-weight. For each claim, we run the same setup across our panel of modern models and fit trend lines to how the effect changes over time.

We classify claims as expired, weakened, persists, or strengthened, according to how much of the original effect remains by September 2026. A relative effect size (e) of 1 means the effect is as strong as in the paper, and 0 means it is gone. An expired claim describes an effect that has largely disappeared in newer models. We use a fifth of the paper’s effect or less as the expiration zone; a claim can expire even if the effect remains strong in some individual models. For expired claims, we estimate how long after publication the effect crossed into this expiration zone. The results below show which findings have faded, which hold up, and how long they last. Reproduction checks and metric definitions follow in How we measure claim persistence.

Results

Almost half (44%) of our tested claims are now expired in today’s models. Another 34% have weakened significantly, 16% persist, and 6% have strengthened relative to the original claim. Persistence varies by topic. Five of nine reasoning claims and six of eleven sycophancy-and-deception claims are classified as expired. Three of ten claims about models as judges, reviewers, and raters persist or strengthen. The three strengthened claims concern judges’ self-preference (C28), praise that follows a user’s preference (C17), and nonuniform random-number generation (C29).

Which claims expire, and when

How long do claims last before they expire?

Across all 50 claims, the estimated share that had expired was 4% six months after publication, 16% after one year, and 39% after two years. These estimates account for claims not yet expired at the end of observation. The share of claims still at full strength (e ≥ 0.8) falls from 64% six months after publication to 47% after one year and 31% after two years.

Among the 22 claims with an estimated expiration date shown in the timeline, time from publication to expiration averaged 15.7 months (median: 14.7 months). These summaries exclude claims that have not expired. Lifetimes vary widely, even within the same research area.

Areas that expire faster on average tend to be those optimized for in post-training pipelines, and those that take longer to expire are those more centered around bias and judgment. This roughly tracks the percentage of claims that are expired per research area. Note that the “Science, research ideas & peer review” area contains only one expired point, the fake citation claim (C11).

The life of LLM claims

The claim cards show which claims have expired and how fast, ordered by when the original paper was released. Each card pairs the original paper’s evidence with the same test across model releases. Select a paper visual to enlarge it, and use dropdowns to filter by research area and verdict.

How much of each effect remains today?

Claims in the same category can still have very different effect sizes. Among weakened claims, for example, the fitted effect is about one-quarter of its reference magnitude for basic prompt injection (C2) and implicit word-association bias (C48), about one-half for the reversal curse (C13), and nearly three-quarters for legal hallucinations (C20). All are weaker than the original finding, but they are far from equally diminished. To illustrate this, we show where each claim stands as of September 30, 2026 (the last date that any of our tested models were released).

A larger e means more of the original effect remains, but it does not necessarily mean the behavior is more common or more harmful.

What changes from model to model?

From GPT-4o mini (earliest model we tested, released in July 2024) to GPT-6.1 Sol (latest model we tested, released in September 2026), we see big shifts in the persistence of claims. The number of claims in the expiration zone rises from 5 to 30 out of 50, while the number that still hold or are stronger falls from 30 to 11. This gives a model-level view of the broad decline seen in the fitted trends. Each model’s claims are sorted from lowest to highest effect, so a claim’s horizontal position can change when you switch models; use its label or the category lists below the plot to follow it.

For each reliably scored model–claim pair, we apply the thresholds to that model’s measured effect e. Expired means e ≤ 0.2, without requiring its interval to lie below that threshold. Persists means 0.8 ≤ e < 1.5 and the lower bound of the 95% interval exceeds 0.2. Strengthened means e ≥ 1.5 with the same interval requirement. All remaining pairs are labeled weakened (“weaker” in the chart), including large point estimates whose intervals still reach the expiration zone. These labels describe one model on one claim, so they do not use the fitted release-date trend.

Release date alone does not determine the pattern. Among models released in July–September 2026, the expired count ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Qwen3.7-Flash has 14, GLM-5.3-Flash 19, and GPT-6 Luna 25. The patterns are also not perfectly aligned with model providers or model size. Hollow points are not scored reliably and are excluded from the category counts.

The totals hide the quirks unique to individual models. Claude Sonnet 5.5’s self-preference effect (C28) is close to the paper’s magnitude (e = 1.09), whereas GPT-6.1 Sol’s is larger (e = 2.88). The pattern reverses for preference for longer answers (C27): Sonnet retains that effect (e = 1.12), while Sol’s point estimate is near the expiration value (e = 0.09). Neither model has lost every tendency: both show stronger-than-reference praise following a user’s preference (C17) (e = 1.85), and both remain highly nonuniform when asked for random numbers (C29) (e = 1.87).

A claim’s panel-wide verdict can therefore differ from what you see on one model. Citation fabrication (C11) is classified as expired overall, yet Qwen3.7-Flash fabricated 69% of references in this test, compared with about 1% for Sonnet 5.5 and 0% for Sol. The plot is most useful for finding such exceptions and opening the corresponding claim card to inspect the original metric and protocol. It is a profile of measured behaviors, rather than a general-purpose model leaderboard.

Case Study on Hallucinated References

The citation example (C11) lets us examine what a declining effect looks like in the generated text. Here, the measured outcome is the share of references that cite nonexistent works. Errors in the bibliographic details of real publications are a separate outcome.

The original results show why fabricated references could be difficult to spot. Among the invented articles from both GPT-3.5 and GPT-4, only 2% named a nonexistent journal. The rest named real journals, giving readers plausible-looking citations to works that did not exist. Fabrication also varied by publication type: 73% of GPT-3.5’s cited journal articles were invented. GPT-4 improved on GPT-3.5 but still fabricated 18% of its references overall.

We repeated the paper’s literature-review prompts across 42 topics, using GPT-3.5 Turbo as the closest available reproduction model and each model in our panel. We checked the generated references against bibliographic databases. GPT-3.5 Turbo fabricated 59% of its 210 references, close to the paper’s 55%, giving us a comparable starting point for measuring change.

On the gap-year prompt, all 5 of GPT-3.5 Turbo’s references were invented: generic author names, real journals, and papers that do not exist. The recent models shown cite real studies, such as Martin (2010) in the Journal of Educational Psychology. These selected examples illustrate the difference in the outputs, and the plot below measures fabrication across all topics for each model.

The decline extends beyond the displayed examples. Across all topics, Claude Sonnet 5.5 fabricated 1% of its references and GPT-6.1 Sol 0%. The high fabrication rate in the original study has largely disappeared in most recent models under these prompts, although the model-level exceptions shown above remain.

This observation has motivated a number of policies where hallucinated references serve as a primary metric for detecting AI-generated work, a prominent concern in conferences and journals. ICLR 2026 reported desk-rejecting papers with hallucinated references, and ICML 2026 lists hallucinated references among its grounds for desk rejection. However, our results suggest that these phenomena might not occur at as high of a frequency as with older models, making such policies a poor direction to discourage AI-generated writing without human oversight. Beliefs formed around an earlier generation of models need to be tested again, and our investigation here is an example of how priors about LLM behavior must be updated with future model releases.

How we measure claim persistence

For the claims shown and cited in the Results section, we first take a key metric from each paper and put it on a common scale that measures change relative to the paper’s result. Second, we check that we can reproduce the paper’s central result within some tolerance. Third, we run the same setup across our panel of modern models and fit trend lines to how the effect changes over time. The sections below detail these steps.

Standardizing claims to a single scale

Validating reproduction

Fitting how each claim changes over time

Discussion

While preliminary, these results teach us a few things about the current state of LLMs and their capabilities as well as empirical research of LLMs writ large.

Claims about LLMs have a life span

In this work, we show that many LLM claims, even famous ones, disappear as LLM capabilities increase over time. Many claims made in empirical LLM literature are assumed to hold for a long time, and they often take root as perceptions the community has about LLMs. As with fabricated citations (C11), these perceptions often work their way into policies without consideration of how subsequent generations of LLMs might impact this claim. Similarly, lags in perception of LLM behavior influence AI trust and how often people use AI in their work, as explained in this tweet from Andrej Karpathy. By revisiting claims, we can more adequately shape our expectations for the failure modes and behaviors of LLMs.

Across the claims we tested, 16% had expired within a year of publication and 39% within two years. Among the 22 claims with an estimated expiration date shown in our plots, the time from publication to expiration averaged about 16 months, with individual lifetimes ranging from about one month to roughly three years. This average excludes claims that have not yet expired, so it is not an expected life span for all LLM findings. Even still, it shows how quickly published observations can lose relevance as models improve. On the contrary, even if a claim was made a long time ago, that doesn’t mean the claim is automatically invalid today. Four of the earliest claims we tested here (Press et al., 2022 C1; Perez & Ribeiro, 2022 C2; Li et al., 2023 C3; Wang et al., 2023 C4) are still not expired today. These aggregate persistence estimates have wide confidence intervals because the study includes only 50 claims.

These results exhibit how empirical LLM claims should be presented in the context of current models rather than as general truths about LLMs. A finding is an observation about particular models at a particular time, not a fixed property of language models. Because of this, papers should report the model, version and date with each result. Citing this work should carry these limitations and acknowledge that these claims might not hold up on newer models. Further, papers should include their exact provenance for reproducibility, including prompts and analysis code, in order to repeat these claims on future LLMs.

Capability failures fade; preferences stay

Most claims that we tested about what models cannot do have faded. Of the reasoning claims, five are expired, four have weakened, and none of them either persisted or strengthened. Some notable failures in other areas disappeared, including fabricated citations (C11), facts lost in the middle of long contexts (C7), and abandonment of correct answers when the user pushes back (C16). One hypothesis for this might be that these failures have a right answer that developers can measure and train against, accelerating progress against these effects. Importantly, many of these failures, especially high-profile ones, might have been specifically patched in post-training in recent models.

Claims about what models prefer have lasted longer. Of the 8 claims that persist and the 3 that strengthened, three concern models as judges or reviewers (favoring their own answers (C28), scoring accepted and rejected papers alike (C31), and rating spun trial abstracts more highly (C37)), three concern agreeing with or flattering the user (C17, C42, C43), two concern sameness (near-identical answers to open prompts (C45) and non-uniform random numbers (C29)), two concern answering when the model should decline (C10, C41), and one is a left political lean (C21). These are tendencies rather than errors on a test with one right answer, so they may be harder to notice and to train away by model developers.

Social bias is a mixed case. Of the seven claims about social bias, culture and simulated populations, only the political lean (C21) persists. Three have weakened, including gender-stereotyped pronoun resolution (C9) and implicit association bias (C48). These cases highlight clear biases that LLMs can be trained against, but these biases should continue to be traced in subsequent model developments as progress continues to be made. Three of these bias claims are classified as expired: harsher judgments of speakers of African American English (C24), stereotyped personas of Black women (C5), and more nonanswers in Arabic than in English (C50). The African American English effect reverses direction in the fitted trend, with its interval entirely below zero.

Weakened is not expired

This work also reveals some surprising failure cases in recently released models and shows the benefit of reproducing claims against LLMs over time. Some reasoning failures have shrunk without disappearing. On two-hop questions (C1), Claude Sonnet 5.5 still gets about one in ten wrong (10%) when it answers both single-hop sub-questions correctly; the paper reported 43% for ChatGPT. The reversal curse (C13) and contradictory answers to paired timeline questions (C47) have also weakened but remain measurable. Through revisiting old claims about LLMs, we can potentially derive insights on persistent problems that can help inform benchmark design or evaluation strategies for new model releases.

Agentic AI and the reproducibility era

In this work, we leaned on agents to reproduce papers at a rate much faster than humans could do manually. This allowed us to test many more claims in a much shorter period of time than would have been possible before coding agents. This is an example of how agentic AI can help us make science more rigorous and reproduce works at scale to gain new insights. We’re excited about future studies that take a similar approach: use agents to revisit old works and provide systematic reproducibility claims that help us advance tomorrow’s science.

We want your feedback!

Have any questions or thoughts about this work or how it can be extended? Think we did something wrong or left out some crucial details? Have suggestions for other claims you’d want us to test? Please reach out to oqueen@stanford.edu with any comments, questions, or concerns. We’d especially love to hear from you if you’re the author of one of the papers we listed above 😄.

Citation

If you use this work, please cite:

@misc{queen2026shelflife,
  title  = {The shelf life of {LLM} findings},
  author = {Queen, Owen and Zou, James},
  year   = {2026},
  url    = {https://zou-group.github.io/llm-claims-shelf-life/}
}