Large language models have dramatically simplified prototyping AI-powered clinical experiences. Summarization, chart review, and search can now be generated by simply feeding patient context into a prompt.

This works well for demos. But in real healthcare settings, these one-shot, stateless approaches break down fast.

Clinical reasoning isn’t about answering a single question in isolation but about understanding how a patient’s story unfolds over time, across visits, exams, and medications. When each AI request treats that history as a fresh prompt, continuity is lost. For instance, to a doctor, today’s “cough” may mean something different if the patient started an ACE inhibitor the week before—AI, however, can miss that timing when interpreting symptoms.

Patient data is fragmented across notes, labs, radiology reports, PDFs, external records, and ad hoc uploads. Trying to “fix” this by stuffing more context into a single LLM call only increases both performance and governance risks.

Even Retrieval-Augmented Generation (RAG), which augments models with external knowledge, can still miss the broader longitudinal narrative clinicians rely on to understand care trajectories. In practice, it often becomes a brittle attempt to optimize context engineering rather than a real solution to longitudinal understanding.

That’s where a new problem quietly emerges: context rot.

What context rot actually is

The industry has named a core limitation of naive prompting context rot. It occurs when adding more information to a prompt actually reduces model performance, even if the relevant information is present somewhere in the input.

More context does not necessarily lead to better understanding. As context grows, noise can bury crucial facts. Attention mechanisms struggle to prioritize long-range dependencies, and the model’s ability to integrate information across multiple segments deteriorates. [1, 2]

This is especially problematic in medicine, where high-quality summarization and reasoning require integrating clinical trends, episodic events, medication changes, and outcomes over time.

Thus, this context deterioration has very concrete effects on the performance of AI systems in healthcare.

How one-shot and long-context AI models fall short in healthcare

1) Performance degradation with long context

In practice, models may miss connections, such as linking a lab value change to a prior intervention, not due to missing data but because the model cannot reason effectively across long context. Simply increasing context windows does not reliably improve reasoning across multi-step or semantic tasks. [3]

2) Rising costs and operational risk

Input tokens may be cheaper than before, but enterprise-scale healthcare workflows cannot rely on brute-force context stuffing, which increases computational and memory overhead and drives up latency. [4]

• Inconsistent outputs and hallucinations increase as context grows, creating factual drift even when relevant information exists. [5]

• Governance and traceability become harder. Larger prompts make it difficult to determine which parts of the input influenced the output, which is a critical issue for clinical compliance. [6]

Why RAG alone isn’t enough

RAG improves on naive prompting by retrieving relevant snippets from external sources before generation, helping models focus on pertinent information. But real clinical histories are complex narratives, not isolated retrieval targets. If key information is buried or misranked, models may still underweigh it or lose coherence. [3]

A systematic review of RAG in healthcare highlights that while RAG reduces hallucination and supports factual grounding, its effectiveness depends heavily on the quality and ranking of retrieved documents. If retrieval is imperfect—as it often is in real clinical contexts with noisy, overlapping records—the generative model may still produce incomplete or misleading answers. [7]

Final thoughts—and what comes next

One-shot LLMs implementations and naive RAG pipelines are excellent for prototypes but fall short of the reliability, scalability, and narrative continuity required for enterprise healthcare. Simply adding more tokens does not resolve underlying limitations and can exacerbate performance and governance risks.

Healthcare AI must move beyond stateless prompting toward models that understand patients as evolving stories, integrating structured and unstructured data across time with continuity and traceability.

Clinia addresses these limitations, maintaining structured, persistent patient context. This enables AI workflows that are more accurate, cost-efficient, and aligned with how clinicians think. (We’ll explore Clinia’s solution in detail in our next article.)

References

[1] Timothy B. Lee. “Context rot: the emerging challenge that could hold back LLM progress.” Understanding AI, Nov 10, 2025.

[2] Du Y, Tian M, Ronanki S, Rongali S, Bodapati S, Galstyan A, Wells A, Schwartz R, Huerta EA, Peng H. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. arXiv [preprint]. 2025. Available from: https://arxiv.org/abs/2510.05381

[3] Zhang G, Xu Z, Jin Q, et al. Leveraging long context in retrieval augmented language models for medical question answering. NPJ Digit Med. 2025;8:239. https://www.nature.com/articles/s41746-025-01651-w

[4] Alla CVK, Gaddam HN, Kommi M. BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models. arXiv [preprint]. 2025. Available from: https://arxiv.org/abs/2511.04919

[5] Liu S, Halder K, Qi Z, Xiao W, Pappas N, Htut PM, John NA, Benajiba Y, Roth D. Towards Long Context Hallucination Detection. arXiv [preprint]. 2025. Available from: https://arxiv.org/abs/2504.19457

[6] Asgari, E., Montaña-Brown, N., Dubois, M. et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digit. Med. 8, 274 (2025). https://www.nature.com/articles/s41746-025-01670-7

[7] Neha F, Bhati D, Shukla DK. Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review. AI. 2025; 6(9):226. https://www.mdpi.com/2673-2688/6/9/226