The Bottleneck in Healthcare AI Isn’t the Model. It’s the Patient.

At Health Datapalooza this fall, our team heard a health leader say something we hadn’t heard before: “We need to make a call on whether we own clinical context or buy it. But we have enough agents running that there isn’t a ‘later’ conversation.”

That’s where many healthcare organizations are now. The pilots worked. Agents are in production. And scaling them is exposing a problem no model upgrade will fix: the agents don’t understand the patients they’re acting on.

Data alone isn’t context. AI needs to see the patient, not just the prescriptions lips, lab results, and symptoms. That’s why AI context matters.

The bottleneck has moved from compute to context

For two years, the healthcare AI conversation centered on models: which one reasons best, which one costs least, and how many to run. Those questions matter less every quarter. Capability gaps between leading models are shrinking and swapping between them keeps getting easier.

What hasn’t gotten easier is giving those models the information they need. McKinsey found that nearly two-thirds of enterprises have experimented with AI agents, but fewer than 10% have scaled them to deliver real value. Eight in ten point to data limitations as the roadblock. Forrester now treats context as a production dependency for agents, not a supporting asset.

In healthcare, that knowledge is mostly about one thing: the patient.

How incomplete context leads to inconsistent, inaccurate action

When AI only answered questions, missing context produced a bad answer, and a clinician could catch it.

Agents don’t always wait for review.

Think of an AI agent as a digital staff member with one narrow job, like scheduling follow-ups or preparing prior authorizations. It pulls what it needs from your systems, makes a call, and acts, often without anyone checking each step. Now imagine that staff member is missing part of the patient’s story and has no way of knowing it.

Take a prior authorization agent preparing a request for a biologic. The patient has already failed two first-line drugs, but that history lives only in a scanned referral letter, not in the structured medication list. The agent submits without it, the payer denies the request for missing step therapy, and staff spend days redoing the work while the patient waits. The information existed. It just wasn’t anywhere the agent could use it.

Multiply that across every agent, and three problems follow:

• Wrong or incomplete answers. When the data underneath is broken, the agent’s output is too. In a care workflow, that error is dangerous if it reaches the patient.
• Inconsistent answers. When each run assembles the patient’s record differently, the same workflow can give two different answers. No clinical team can accept that.
• Too much access. Agents wired straight into source systems often see more than their task requires, including mental health or substance use history, and there’s no central place to audit who saw what.

A better model doesn’t fix any of these. Better context does.

Why context is harder in healthcare

Most AI context vendors cover business rules: workflows, policies, how a company handles its customers. Healthcare needs that too. But it also needs patient context: history, medications, home situation, communication preferences, what’s already been tried. That’s the harder half.

Consider one medication for a 70-year-old with a complex history. The hospital EHR lists metformin with a standard drug code. A scanned note from a community clinic calls it Glucophage, with no code at all. The claims feed carries a different code. The patient, in an undated statement, says he stopped taking it. And an imaging center has him on file under a first initial and date of birth that were never matched to his record.

Each record was accurate where it was written. Together, they disagree. An agent asked whether this patient is on metformin has five answers and no way to know which one is current. (We covered why common approaches fail to resolve this in our piece on retrieval and contradictory clinical data).

A few more factors raise the bar:

• The subject is the patient, not the user. Generic AI memory tools remember the person typing. Healthcare agents have to understand the person being cared for, who usually isn’t there to correct the record.
• Context goes beyond the chart. Claims, patient-reported data, social determinants, clinician preferences, and organizational protocols all shape the right action.
• Governance is stricter. A scheduling agent needs a refill date, not a mental health history. Each agent should see only what its task requires, and every access should be logged.
• Conflicts have higher stakes: In business rules, a conflict might just cause a temporary reduction in productivity. In healthcare, it could lead to an essential prescription not being renewed.
• The story is more complex: Patient context needs to represent a cohesive, longitudinal story based on complex, conflicting medical facts. Agents need the clinical knowledge to know what a specific condition is and the context of how that condition affects a specific patient over time.

Most agents can’t assemble patient context that meets a clinical standard. The few that can have to assemble it all on every task and the costs compound. Token spend repeats on every run. Two runs of the same workflow can return different answers, which is unacceptable in a healthcare context. Access control ends up split across every team that wired its own connection. What works in a pilot becomes too expensive and too hard to govern at scale.

What it takes to scale

Context can’t be solved one agent at a time. Forrester argues that enterprises need shared context services and common governance rather than siloed projects. Healthcare needs the same shift, built for clinical data.

One context layer that every agent reads from and writes back to delivers four things:

1. One understanding of the patient, instead of five versions that don’t reconcile.
2. Less repeated compute. Context is managed continuously instead of being built from scratch each time an agent runs.
3. Faster agent development, because builders start from an understanding of the patient that’s already in place.
4. Freedom to switch models and vendors without losing what you know about your patients.

To deliver that, the layer has to do health-specific work. It should resolve records by clinical meaning, so one medication is one fact. It should reconcile conflicts and show where every value came from. It should connect clinical relationships, like which drug treats which condition. And it should give each agent only the slice its task requires.

Rent the models. Own the context.

Models will keep improving and keep getting easier to swap. Your organization’s understanding of its patients won’t rebuild itself every time you change vendors. That’s the asset worth owning.

Before deploying the next agent, ask whether it has the patient context to act responsibly. If the answer depends on which agent you ask, that’s the constraint to solve first.