A new physician faces a long, hard road to being trusted with patients. In addition to exams, there are years of closely supervised training in clinical settings. Even with decades of experience under her belt, a physician faces regular review of her performance. AI systems should be held to the same standard of rigor. Large language models now score near-perfectly on medical licensing exams, but this is only the first step to proving that they can be trusted in a complex clinical environment. Process evaluations don’t just test whether a model can get to the right answer; they test if models match the work that clinicians do to reach one: gathering information, ordering the right tests, following guidelines, and fitting into a crowded workflow to arrive at a defensible and evidence-based conclusion.

Hager et al. (Nature Medicine, 2024) tested the gap between outcome and process evaluations directly. They had models work through 2,400 real emergency cases step by step, and diagnostic accuracy fell well below that of physicians. The models often diagnosed without ordering imaging, didn't follow treatment guidelines, and sometimes did worse when given all the available information. Newer systems are only beginning to close that gap. In 2026, Ferber et al. (Nature) tested an agent on a similar set of emergency cases, scoring not just its diagnoses but its test orders, prescriptions, and guideline adherence. While it succeeded in some areas, it ordered about twice as many blood tests as the physicians it was measured against.

As AI agents move into hospitals, how a model works matters as much as what it concludes. As a recent Lancet Digital Health viewpoint notes, Epic announced a platform for building agentic workflows inside its electronic health record in March 2026, while evaluation standards lag behind.

Evaluating clinical AI is less like grading a licensing exam and more like the way a hospital decides whom to trust with its patients. In the US, passing the national licensing exams and, typically, the specialty exams known as “boards” is only the starting point. Before a physician can treat patients at a particular hospital, the hospital verifies her training and track record, a process called credentialing. It then grants her privileges: a specific list of what she may do there, such as performing a particular surgery or interpreting a particular type of scan. Every new privilege starts with a period of focused review, in which colleagues observe or audit the physician’s early cases, with no exemption for board certification, experience, or reputation. After that, performance is reviewed on an ongoing basis for as long as the physician holds privileges, and the results decide whether those privileges continue, are limited, or are revoked. At many hospitals, especially teaching hospitals, errors are discussed candidly at morbidity and mortality (M&M) conferences, regular meetings where clinicians walk through cases with negative outcomes and discuss what could have prevented them. The system isn’t perfect, but it rests on a principle clinical AI lacks: no one is trusted on a test score alone, and trust is never permanent.

That same Lancet Digital Health viewpoint argues that clinical agents should meet “residency-level” assessment before they reach patients, the standard applied to new doctors during their supervised training after medical school. Medicine’s own practice suggests going further: scrutiny of clinical AI should be scoped to what a system is allowed to do, focused whenever it is deployed in a new setting or for a new use, and continuous for as long as it is in use.

Applying that scrutiny to clinical AI is hard, for a few reasons:

How did it work up the case? An agent works up a case through tool calls: pulling a patient’s record, ordering a lab, checking an insurer’s coverage policy. Every call it makes, and every call it skips, shapes the decision, so evaluation has to track the full sequence, not just the final answer. That means defining the evidence a sound decision depends on, and the shortcuts that undermine it, such as ruling out a condition because no note mentions it rather than confirming the test came back negative. Nor can we simply ask the models to explain their reasoning: a March 2026 preprint (Afolabi et al.) found that the reasoning that leading models write out is often not a reliable record of how they decided.

Experienced physicians don’t all practice the same way. Clinicians don’t always agree on what constitutes best practice, even within one institution. Further, they often don’t follow the same steps to reach a diagnosis: when Mercan et al. tracked 87 pathologists reading digital breast biopsies, they found two distinct strategies: “scanners,” who pan across the slide at a constant magnification, and “drillers,” who zoom in and out on specific regions. If experienced clinicians reach their conclusions by measurably different routes, which route should a model be held to? The independent evaluations used in medical malpractice reviews could provide a framework for distinguishing between diversity of clinical opinion and clinical error, but this does not help us distinguish good practice from great practice.

No one works alone. The real unit of evaluation is the productive relationship between a clinician and an AI system. Yu et al. (Nature Medicine, 2024) measured how AI assistance changed the performance of 140 radiologists across 15 chest X-ray tasks. The effect varied widely, and experience and familiarity with AI didn't predict who would benefit. 

Privileges earned at one hospital don’t carry over to the next. A physician with decades of experience would still go through a new round of credentialing when joining a new hospital, in part because the policies, staff, and equipment she is used to have all changed. The same is true of models. A model trained on US records will struggle in places it hasn't seen, including many places where good training data is hard to come by. A baby with diarrhea in Nigeria is far more likely than one in the US to have an infection spread through unsafe water, so a US-trained model can misdiagnose. Models may also assume information exists when it doesn't, and algorithmic bias is often embedded as uneven outcomes, such as denial rates that differ by language or insurer. In a large city, an AI might reasonably recommend an MRI. In a small town without one, a better plan may start with locally available tests before sending the patient on an expensive trip.

Mistakes need to reach M&M. Health systems already review errors through clinical quality management, and drugs and devices are monitored through risk-based surveillance focused on moderate and severe adverse events. Clinical AI should build on these systems rather than replace them, and scale oversight to the stakes, the way privileges are scoped to specific procedures: a search tool and an autonomous decision-maker need different scrutiny. But those channels were built for discrete, attributable failures. AI often fails silently and gradually, and an error a supervising physician catches in a quality review rarely reaches the deployment team or model builder. Device reporting systems weren't designed to capture AI-specific failures, such as uneven performance across patient groups, that degrade care without counting as a malfunction.

Without process evaluation, a hospital system or a small-town clinic can't judge the risk of deploying a promising model. Under-resourced settings, which stand to gain the most, are the least able to absorb that risk. Monitoring after deployment is also largely unsolved: a 2024 review of 692 FDA-authorized AI devices found that only nine percent of their public summaries included a prospective post-market surveillance study. 

A July 2026 PLOS Digital Health study followed four deployed clinical AI systems and found that performance degraded even though no one changed the models. The earliest warnings were workflow signals, such as missing inputs and delayed data. And there is no agreed answer to a basic question: when a model shows too much risk, who can turn it off? A hospital’s medical staff can limit or revoke a physician’s privileges when review turns up a problem. No equivalent authority exists for a model.

One meaningful step toward realistic assessments involves running tests with novel, anonymized patient data. For example, Xlue AI recently partnered with Protege to independently test HealthSAW, its general clinical intelligence model, on predicting how a patient’s health changes over time. The benchmark contains real patients no model has ever seen, making memorization and overfitting impossible. Results were scored separately across 16 disease-risk workflows in cancer, heart disease, and neurodegenerative conditions, rather than as a single number. HealthSAW outperformed leading frontier models on 14 of them, at less than 0.1% of the cost per patient.

The upside of clinical AI is immense, both in extended healthspan and in the efficiency of a sector that makes up 18% of the US economy. Realizing that value depends on evaluating process as well as outcome. Patients can trust physicians they’ve never met in part because those physicians are credentialed, limited to specific privileges, and reviewed for as long as they practice. Clinical AI needs the same thing: a trusted, widely used way to judge quality, from the first case it touches to the thousandth.

If today's stories caught your eye, join us live at 9 AM PT on The AI Collective Podcast. Each morning we dig into the AI news that actually matters for you, what's happening inside and around The AI Collective, and the experts leading the field.

Our hosts, AJ Green and Catherine McMillan, will be interviewing these guests today:

The AI Collective is built by volunteers across 180+ chapters in 40 countries.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.