Get ready: Hack for Humanity kicks off in just ONE WEEK.

This past June, we celebrated Humans in AI Week, when 250,000 people across 100+ events named the challenges AI is creating in the places they live. 

Next Tuesday, September 1, Soul Summer culminates in Hack for Humanity: a global civic hackathon running through October 31 across 130 local AI communities in 50+ countries, our largest coordinated civic build to date. Chapters will build AI tools around three major themes: job displacement, technical literacy, and accessibility and set additional challenges based on their own cities' priorities.

Visit our microsite to learn how you can get involved in your community.

At Protege, our DataLab research team tackles frontier data challenges across domains. In healthcare, we provide data to AI model builders, with a data partner network spanning hundreds of millions of patients, billions of patient encounters, almost a trillion clinical notes, and tens of millions of multi-modal patient records. In the United States alone, we estimate this covers ~20% of the country’s economic activity.

This gives us a unique lens into how AI is being deployed in healthcare and what the pitfalls are. We are seeing a huge clinical readiness gap - between how AI performs in test settings and how it performs inside a real clinical workflow. 

Why does that scare me? We’re letting AI into healthcare based on tests that are flawed. It’s like letting a doctor practice on patients simply because they did well on the MCAT.

Let’s start with what we already know. The cat is out of the bag. AI in healthcare is here. More than 300 million people use ChatGPT weekly for healthcare questions. Almost every physician in the United States has referred to AI for clinical guidance this year - and data shows that doctor’s offices are increasingly using AI to document patient visits.

The question is no longer whether healthcare will adopt AI. The question is now: “Will this AI actually make healthcare better?”

Healthcare is betting that AI will save time, reduce costs, and improve care. But right now, we are choosing which systems to deploy based on a static test. Instead, healthcare needs to measure effectiveness: did the system reduce mistakes, lower costs, give clinicians meaningful time back, or help patients live longer?

Today, organizations are using benchmarks to evaluate healthcare AI. Think of a benchmark like a standardized test that AI models take to measure how well they perform on set tasks. 

The problem in healthcare is that current benchmarks are static: they are a test with the same pre-defined questions or cases, with a predetermined answer key. This is useful for testing a very specific capability under controlled conditions… but not for showing how AI performs during messy, complex, back-and-forth healthcare situations. Which is exactly how medicine is practiced.

The test is also becoming unreliable. In June, a Nature Medicine paper concluded that general-purpose models such as ChatGPT, Claude, and Gemini provided better clinical guidance than specialized healthcare AI models like OpenEvidence (used by 65%+ of US physicians). Two weeks later, another study reached the opposite conclusion. In a blinded evaluation, 149 medical experts across more than 30 specialties rated OpenEvidence more highly on accuracy, usefulness, source quality, and verifiability than general-purpose-models.

Both studies appeared in respected publications, with opposite results. How?

It’s not scientific misconduct or bad faith. It’s because medical AI evaluations — the tests — are full of consequential choices. What cases do you include? How do you phrase the prompt? Which model settings do you use? How much reasoning should you include? What goes into the test rubric?

What we ultimately care about is effectiveness: whether that performance improves care in the messy environment where patients, clinicians, incentives, workflows, and costs collide. Getting a perfect MCAT score does not mean you’re a good health practitioner. In the same way, acing a static healthcare benchmark does not mean the AI model is ready for hospital use.

I’m not the only person concerned. Leading AI researchers from OpenAI, Google, Microsoft, and Amazon, alongside clinicians from Stanford, Oxford, Scripps, Mass General, and ARPA-H, have called for a “rigorous, task-based framework” to define and measure medical AI “superintelligence.”

But why is this so hard? The quick answer: building fair, unbiased healthcare benchmarks is much harder than it sounds.

You can divide AI tasks into two types: verifiable and non-verifiable. Coding is instantly verifiable - the code runs or it doesn’t. This is why models have improved so quickly; you can run experiments, check the answers, iterate, and try again.

Not so in healthcare. There’s a whole chasm of tasks that can’t be verified instantly. Sometimes confirming the answer requires a lab experiment. Or expert adjudication. Or the outcome hasn’t happened yet. Or there’s no pre-determined “right” answer that’s agreed upon.

Here’s one example: knee replacements. If you need surgery for this, you can either get a “partial” or a “total” knee replacement. Both are clinically defensible - there is no automatic “right” answer. 

What we found in the data is that a huge part of the choice… comes down to who your surgeon is. 

Patient characteristics, comorbidities, facilities, and year only explain 3.4% of the variation in the choice to perform partial or full replacement. But if you add the surgeon’s identity, it immediately becomes 14.8%. In other words, 77% of the explained variation in partial vs full can be answered by simply asking: “Who was the surgeon?”

This is an example of how “taste” is reflected in medical data, and we see it across many other complex healthcare situations. Who’s right when there isn’t a “right” answer among the best doctors in the world?

This is why training AI for medicine — especially the most difficult, judgment-based decisions — is very, very hard. Each decision requires coordination, context, and most importantly: human judgment. 

Current benchmarks can test whether a model can retrieve the correct lab value. But actual care would require AI to interpret that lab value, incorporate new clinical information, communicate uncertainty, and influence the decisions of the patient, nurse, and physician. And all this across many interactions.

As we’ve parsed through terabytes of data, we’ve come to the same conclusion again and again: real medicine is never a clean question paired with one clean response. The data beneath those decisions spans time and modalities. All the data needs to be connected to provide a clear picture: clinical notes, labs, imaging, genomics, claims, patient messages…all that went into clinical decisions.

Another often overlooked part is privacy and context. We must ensure that sensitive patient data is privacy-protected from start-to-finish, without losing the important signal in the process. That’s difficult. 

Consider the example above: on the left, you have an illustrative patient note that’s been anonymized by replacing every name and date with another. However, this destroys the note’s value – you need this information to understand what happened. Imagine this happening across different data modalities — the note needs to still connect to MRI scans, pathology reports, and other health systems. 

Standard de-identification processes produce incoherent records that an AI model cannot learn from. The chart on the right illustrates our context-preserving de-identification approach, which shifts dates and changes names without compromising the value or privacy. All this is part of the data quality bar set for creating realistic evaluations.

Creating healthcare benchmarks requires human judgment that gets imputed into the benchmark itself. Someone chooses the cases, prompts, settings, and what constitutes success or failure. That’s why it’s so important for the benchmark designer to understand both how healthcare is practiced and the nuances of the source data. 

Otherwise, healthcare providers evaluate AI models based on either static tests, or trust the testing done by the model-builders themselves. That’s like a chef grading their own food. Right now, stakeholders have no reliable, independent way to compare different AI systems. 

When the benchmark is a test, there is an incentive to beat the test. Our testing shows that even something as simple as the answer choice ordering can change the outcome. 

In a diagnosis task where models were asked to choose the best answer given the data provided, we gave the model four random permutations of the exact same answer choices. Same case, same answers - different order. And the models frequently changed their answers!

Even when the data, conditions, and prompts are constructed thoughtfully, models can become better at passing the test without improving outcomes. There is a very real danger of healthcare AI companies optimizing for static benchmarks that do not reflect the true point of care.

Building a better healthcare benchmark has to begin in the opposite direction. Start with the outcome that matters, then work backward to the task, then the data, and then finally the test.

This means moving beyond static questions and toward realistic, multi-turn scenarios built on real-world healthcare data. This also means having an independent evaluator of how AI models are performing.

But well-designed benchmarks are only the first step. The real test is what happens at the point of care, when AI interacts with clinicians and patients over time. Did the system reduce medical errors? Did it give clinicians meaningful time back? Did it lower costs or improve outcomes?

With the healthcare data we have access to at Protege, we’re connecting privacy-protected, multimodal, longitudinal healthcare data to specific real-world tasks so model builders and healthcare institutions can better evaluate AI. We hope this continues to build the bridge between static benchmarks and real-world healthcare outcomes.

We’re looking forward to building alongside other research-driven organizations that care deeply about this challenge. If you’re building in this space, we’d love to hear from you!

Read more at https://withprotege.ai/model-builders/healthcare, and reach out at https://withprotege.ai/contact.

The AI Collective is built by volunteers across 180+ chapters in 40 countries.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.