News / Press release / October 21, 2026

Protege Launches Independent Blind Evaluations for Healthcare AI

Clinically-grounded assessments built with held out real-world data help AI builders prove and improve performance

New York, October 21, 2026 — Protege, the data and evaluation platform for AI in healthcare, today announced the launch of independent blind evaluations for healthcare AI. The new offering gives healthcare AI builders private, clinically-grounded assessments that test AI models against new real-world data for key use cases. Protege is launching the initiative with its first benchmark alongside independent evaluations in health world models and oncology with healthcare AI companies Xlue AI and Standard Model Biomedicine.

81% of physicians now use AI in their day-to-day work, but measuring AI performance has lagged behind deployment. Public benchmarks are saturated, private benchmarks built by model builders can suffer from bias when the benchmark-builder is also selling the model, and other benchmarks often lack the patient data that is representative of real-world situations. Existing benchmarks also commonly suffer from leakage, where the benchmark data leaked into the model training. When this happens, the model has been exposed to the test answers in advance, which renders the results inaccurate.

As a result, healthcare organizations looking to introduce AI in care settings and administrative workflows are currently limited in their ability to independently and fairly evaluate the effectiveness of these models.

Protege addresses that gap with the release of independent blind evaluations that are built exclusively with real-world healthcare data that existing AI models have never seen before. The offering combines the healthcare data expertise of DataLab, the company’s research arm, with Protege’s extensive healthcare data provider network and expertise in preparing data for AI model training and evaluation. The result gives healthcare AI buyers additional evidence when deciding which AI products to adopt, while helping model builders understand where their models succeed, fail, and can improve. These blind evaluations complement Protege’s existing evaluation datasets and benchmarks for healthcare AI released last year.

“Healthcare AI is moving quickly, but the way we evaluate it hasn’t kept pace,” said Bobby Samuels, co-founder and CEO of Protege. “When builders test their own models on data they control, healthcare organizations are being asked to trust performance without an independent measure of whether it will hold up against real patient scenarios. We think healthcare AI needs a credible third party that can provide that evidence, and Protege’s real-world data and healthcare data expertise combine to produce independent, relevant, and accurate evaluations of AI model performance.”

The first of Protege-designed healthcare benchmarks, the Disease Odyssey benchmark evaluates whether AI can recognize a serious illness developing before the diagnosis occurs. This tests how the model reasons through an ambiguous clinical record, and is designed for teams developing clinical AI that reviews patient histories, supports diagnostic reasoning, or identifies patients for further assessment.

Initial Protege DataLab testing demonstrates that leading AI models consistently score between 52.7% and 69.3% accuracy on free-response questions that asked models to correctly diagnose conditions such as sleep apnea and anemia. This indicates that there is significant opportunity for leading AI model builders to train and fine-tune their models for one of the most common yet complex parts of a clinician’s job.

“Healthcare AI models must be measured on real world cases, rather than saturated test questions based on data it may have already seen,” said Engy Ziedan, Ph.D., Protege co-founder and Chief Scientific Officer at Protege. “Designing these benchmarks requires scientific rigor: pre-defining what we’re testing, selecting uncontaminated data, and using blind testing to reduce bias. That requires understanding not only where models do well, but also where they fail and whether the performance applies to real-world healthcare settings.”

In addition to DataLab-constructed benchmarks, Protege is also working with partners to test emerging healthcare models with real-world patient data.

Protege partnered with Standard Model to evaluate its biomedical world model on overall survival and time-to-next-treatment predictions for breast and lung cancer. In a double-blind test using EHR and DICOM data, the specialized model performed competitively with frontier models at 36 to 170 times lower cost per prediction, delivering results in seconds rather than minutes with zero outside network calls. Protege provided a held-out test set with net-new data and controlled the testing and scoring.

“We were pleased to see the Standard Model outcompete frontier model performance, with an order of magnitude decrease in operational costs,” said Kevin Brown, co-founder and CEO of Standard Model Biomedicine. “Protege’s double-blind evaluation shows that a compact, specialized model can beat the world’s largest frontier models on real patient histories at significantly lower costs, and that combination makes patient simulation practical at the scale required for trial design and execution. We’re excited to have early access to Protege’s benchmarking platform, as its double-blind evaluation offers the kind of rigorous comparison the industry has been waiting for.”

In its partnership with Xlue AI, Protege measured Xlue’s healthcare world model HealthSAW, a model that learns how a patient’s health evolves in response to care and predicts what happens next. The Protege-designed benchmark tested HealthSAW on how well it could use longitudinal patient histories to predict cancer, cardiovascular incidents, and neurodegenerative conditions — all using data that AI models had never seen before.

Protege independently validated that Xlue’s HealthSAW was better than the other frontier models at identifying which patients were most likely to develop a range of diseases in the future, ranking highest on 14 of 16 disease tests ranging from cancer (bladder, breast, colorectal, kidney, liver, lung, pancreatic) to cardiovascular diseases (acute coronary syndrome, coronary artery disease, cerebral aneurysm, heart failure, stroke), and neurodegenerative conditions (Alzheimer’s, dementia, Parkinson’s disease, vascular dementia).

In addition, the costs were significantly lower than existing frontier models: at scale, Xlue AI estimates that screening 1 million patients would cost ~$36 with HealthSAW versus ~$32,000 using GPT-6 Astra. This indicates the potential of HealthSAW to help predict who will respond well within a clinical trial based on future disease risk and treatment outcomes, at much more scalable costs.

“Our ambition is to build models that understand how a patient’s health evolves and help anticipate what comes next and how patients react to treatments,” said Chenyan Xiong, co-founder and CEO of Xlue AI. “Protege gives us something essential to that work: an independent test on patient histories our model has never seen. That lets us measure progress credibly and focus development on the capabilities that help physicians treat patients better and pharmaceutical companies develop drugs more efficiently and effectively.”

The launch follows Protege’s Evaluating Healthcare AI: From Benchmarks to Real-World Trust event, where healthcare and AI leaders discussed the need to move beyond one-size-fits-all evaluation toward testing that reflects specific clinical workflows, real-world complexity, and datasets models have not been trained on. Protege plans to expand its evaluation work across additional healthcare applications, including clinical diagnosis, pathology, dermatology, oncology, and healthcare administration.

Healthcare AI builders interested in independently evaluating their models or applications, as well as enterprises looking for independent measurement of healthcare AI before deployment can contact Protege.