Findings
Researchers introduced ChronoEvo, a benchmark containing 1,709 recommendation-level revisions across 12 types of change in medical knowledge. The study was designed to separate two abilities: recognizing that clinical guidance had been revised and applying that revision fully when making a clinical recommendation. This work is a preprint and has not been peer reviewed.
For the primary model, 90.8% of responses recognized the revision and 80.2% were judged clinically correct, but only 58.5% fully incorporated the updated guidance. That produced a 32.3-percentage-point gap between recognition and incorporation. The researchers report that the broader pattern persisted across 14 models.
Two clinicians also adjudicated 300 responses in a validation sample, where the recognition–incorporation gap was 40.3 percentage points. Revisions involving recommendation strength and patient eligibility were associated with models retaining outdated guidance. This is an association reported by the researchers, not proof that those types of revisions caused the failures.
An intervention that highlighted the revision reduced retention of outdated content and improved clinical correctness, according to the abstract. However, it did not significantly improve full incorporation of the new guidance. The abstract does not supply the intervention’s effect sizes or enough statistical detail to assess the size and certainty of these changes.
Why it matters
The results suggest that access to updated information and apparent recognition of it may not be enough for an LLM to revise all parts of a medical recommendation consistently. If confirmed, this distinction could matter when evaluating systems intended to summarize evolving guidelines: a response may acknowledge a change while still preserving obsolete advice elsewhere.
The study evaluated model responses rather than patient care. It therefore does not show that these failures caused incorrect treatment, delayed care or other human health outcomes, nor does it establish that any tested model is safe or unsafe for clinical use.
What remains uncertain
The available abstract does not identify the models, medical specialties, prompting conditions, dates of the guideline revisions or detailed scoring criteria. It also does not report agreement between evaluators, uncertainty intervals for the main percentages or performance for each model and revision type. These omissions limit assessment of generalizability and robustness. Independent replication and peer review would be needed before drawing firm conclusions about clinical reliability.