Reported performance does not establish clinical reliability. Epic reported AUROC of 0.76–0.83 for its original sepsis model, but external validation found a hospitalization-level AUROC of 0.63; at threshold 6, sensitivity was 33% (Wong et al., 2021). On MedQA, answer-format disruption reduced standard LLM accuracy by 26.5–38.2 percentage points (Bedi et al., 2025). Clinical decision support therefore requires independent evaluation on the intended task and local population.
Match Evidence to the Claim
Clinical AI evidence is multidimensional, not a single ladder. Study design should be judged against the question it is meant to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE applies certainty ratings to a body of evidence for a specified outcome, not to a journal, vendor, or single paper (CDC ACIP GRADE Handbook, 2024).
| Does the model perform as specified? |
Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance |
TRIPOD+AI for prediction models, STARD-AI for diagnostic accuracy, and PROBAST+AI for risk of bias and applicability (Collins et al., 2024; Sounderajah et al., 2025; Moons et al., 2025) |
| Does performance transfer? |
Independent temporal, geographic, and multi-site evaluation in populations and data systems relevant to intended use |
External validation with explicit assessment of case mix, prevalence, measurement, calibration, and operating thresholds |
| Does the system work in the clinical workflow? |
Prospective evaluation of data pipelines, clinician interaction, adoption, workload, failure modes, and subgroup effects |
DECIDE-AI for early-stage live clinical evaluation (Vasey et al., 2022) |
| Does using the system improve care? |
Comparative evidence on decisions, processes, patient-important outcomes, and harms |
Randomized trials when feasible, or a justified quasi-experimental design; CONSORT-AI for reports of AI intervention trials (Liu et al., 2020) |
| Does benefit persist after deployment? |
Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, and unintended consequences |
Risk-based surveillance across the product lifecycle (FDA, 2025) |
Reporting guidelines improve transparency, but complete reporting does not by itself establish low risk of bias or clinical benefit. Risk-of-bias tools must also match the design: PROBAST+AI for prediction models, QUADAS-3 for diagnostic accuracy studies, RoB 2 for randomized trials, and ROBINS-I for nonrandomized intervention studies (Moons et al., 2025; Whiting et al., 2026; Cochrane RoB 2; Sterne et al., 2016). GRADE then assesses certainty across a body of evidence for a specified outcome. Likewise, an RCT is suited to a causal claim about using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, or long-term drift. Evidence requirements should increase with the consequence of error, degree of autonomy, and difference between the study setting and intended use.
Clinical utility examples: The CONCERN early warning system was evaluated in a pragmatic cluster-randomized trial across 60,893 hospital encounters in 74 clinical units; the published article reported lower mortality risk and shorter length of stay with the intervention (Rossetti et al., 2025). A 2026 author correction revised the length-of-stay effect estimate after a modeling error but did not change its statistical significance (Rossetti et al., 2026). GOLDEN BRIDGE II, a cluster-randomized trial of 21,603 patients with acute ischemic stroke in 77 hospitals, found fewer new vascular events at 3 months with an integrated clinical decision support system than with usual care (2.9% vs. 3.9%; adjusted HR 0.74) (Zhang et al., 2026). These trials support causal claims about the evaluated interventions and settings; they do not validate every AI system or guarantee transportability elsewhere.
Common Validation Pitfalls
These patterns can inflate development performance or reduce applicability in deployment.
Selection Bias
Problem: Training only on patients who received gold standard test.
Example: Biopsy-confirmed cancer AI trained only on lesions suspicious enough to biopsy.
Result: Misses spectrum of disease severity in real practice.
Temporal Bias
Problem: Train on old data, validate on old data.
Result: Medical practice evolves; algorithm becomes obsolete before deployment.
Site-Specific Overfitting
Problem: Works at Institution A, fails at Institution B.
Cause: Different EHRs, imaging equipment, patient populations, documentation practices.
Evaluation response: Use multi-site external validation when the claim requires transportability across institutions.
Label Leakage
Problem: Training labels contain information not available at prediction time.
Example: Sepsis prediction using antibiotics administered (clinician already diagnosed sepsis).
Evidence of prevalence: A literature review found 40.2% of MIMIC-based same-admission prediction models used ICD codes as features, even though these codes are finalized only after discharge. Models using only ICD codes achieved AUROCs of 0.97–0.98 for mortality prediction, with the most “predictive” codes being brain death, cardiac arrest, and palliative care encounter (Ramadan et al., 2025).
Result: Inflated performance that may not reproduce prospectively.
Publication Bias
Problem: Only positive results published.
Result: True performance lower than literature suggests.
Outcome Definition Shifts
Problem: Training outcome differs from deployment outcome.
Example: Train to predict ICD codes, deploy to predict actual clinical deterioration.
Benchmark vs. Reasoning (LLM-specific)
Problem: High accuracy on medical benchmarks may reflect pattern matching, not clinical reasoning.
Evidence: In a 68-question MedQA test, answer-format disruption reduced standard LLM accuracy by 26.5–38.2 percentage points (Bedi et al., 2025).
Interpretation: Answer-format sensitivity limits what benchmark accuracy alone can establish.
Why External Validation Matters
Clinical AI evaluation still relies heavily on retrospective accuracy measured on curated datasets, with less attention to workflow fit, adoption, safety controls, or downstream care effects (Azad et al., 2026). Internal validation cannot establish transportability because performance may change with case mix, prevalence, measurement practices, equipment, and data pipelines.
FDA authorization does not guarantee validation quality. An analysis of 130 FDA-authorized AI medical devices (2015–2020) identified important evidence gaps (Wu et al., 2021):
- 126 of 130 devices (97%) were evaluated using only retrospectively collected data
- Only 13% of the available summaries reported evaluation across demographic subgroups
- In the study’s accompanying chest-radiograph case analysis, AUROC declined by 0.03–0.12 when site-trained models were evaluated at external sites
- Public summaries often lacked information needed to judge multi-site testing, sample size, and subgroup performance
In this 2015–2020 cohort, prospective clinical evaluation was uncommon and important information about performance across settings and patient populations was often unavailable.
Recent regulatory guidance on adaptive AI: The FDA finalized its Predetermined Change Control Plan (PCCP) guidance in August 2025. A PCCP can describe specified modifications and the methods used to develop, validate, and implement them as part of a marketing submission (FDA PCCP Guidance, August 2025). The FDA’s January 2025 lifecycle and marketing-submission guidance for AI-enabled device software functions remains a draft and is not for implementation (FDA Draft Guidance, January 2025).
Version-Specific Evaluation: Epic Sepsis Model
The original Epic Sepsis Model and a later version produced materially different results across studies and sites. Evaluation must therefore name the model version, outcome definition, threshold, population, timing, and alert burden. The full evidence comparison is maintained in the Epic Sepsis Model evidence review.
Performance Metrics for Clinicians
Understanding these metrics helps you interpret vendor claims and identify misleading statistics.
Accuracy (Often Misleading)
Formula: (TP + TN) / Total
Problem: Disease prevalence affects interpretation dramatically.
Example: - Cancer prevalence: 1% - Algorithm always predicts “no cancer”: 99% accuracy but clinically useless
Never use accuracy alone for rare outcomes.
Sensitivity (True Positive Rate)
Formula: TP / (TP + FN)
What it measures: % of actual positives correctly identified
When critical: Screening tests, rule-out situations (don’t miss cancers)
Threshold trade-off: Lowering a decision threshold to increase sensitivity usually increases false positives
Specificity (True Negative Rate)
Formula: TN / (TN + FP)
What it measures: % of actual negatives correctly identified
When critical: Avoiding unnecessary workups, rule-in tests
Threshold trade-off: Raising a decision threshold to increase specificity usually increases false negatives
Positive Predictive Value (PPV)
Formula: TP / (TP + FP)
What it measures: If test is positive, what’s probability patient actually has disease?
Critical insight: PPV depends on disease prevalence in the intended population.
Example showing prevalence impact (with 90% sensitivity, 90% specificity):
| 50% |
90% |
Excellent |
| 10% |
50% |
Half of positives are false |
| 1% |
8% |
92% of positive results are false positives |
Ask for PPV at the intended operating threshold and expected local prevalence. Also examine sensitivity, false negatives, alert volume, and the consequences of each error type.
AUC-ROC (Area Under Curve)
What it measures: Overall discrimination across all possible thresholds
Range: 0 to 1. A value of 0.5 indicates chance discrimination; a value below 0.5 indicates ranking opposite to the specified outcome direction.
Interpretation: A higher AUROC indicates better ranking of positive versus negative cases across thresholds, but no universal band establishes clinical adequacy. Acceptability depends on the comparator, intended use, case mix, operating threshold, and consequences of error.
Limitations: - Doesn’t tell you performance at specific clinical threshold - Can be high even when PPV is poor at low prevalence - Doesn’t capture calibration
Calibration (Often Overlooked)
What it measures: Do predicted probabilities match observed frequencies?
Example: - Good calibration: AI predicts “30% mortality risk” for 1000 patients → ~300 actually die - Poor calibration: Predicted 30%, but 50% actually die (underestimates risk)
Why it matters: Poorly calibrated models produce misleading probabilities, hampering clinical decisions.
Study Design and Clinical Impact
Match Study Design to the Question
Retrospective evaluation: - Appropriate for model development, initial validation, temporal or geographic external validation, and failure analysis - Main concerns include selection bias, data leakage, unstable labels, missingness, and mismatch with intended use - Cannot establish the causal effect of introducing the system into care
Prospective silent evaluation: - Tests the locked system and data pipeline on new patients without exposing clinicians to its output - Measures live performance, alert burden, data failures, and transportability - Cannot establish how clinicians will use the output or whether patient outcomes will improve
Prospective live evaluation: - Measures clinician interaction, workflow fit, adoption, overrides, errors, and safety in the intended setting - Early-stage studies can establish feasibility and reveal implementation failure modes - Without a concurrent comparator, changes in outcomes may reflect secular trends or confounding
Comparative impact evaluation: - Randomized designs provide the clearest causal estimate when randomization is feasible - Stepped-wedge, interrupted time-series, or controlled before-after designs may be justified when randomization is impractical, but their assumptions and residual confounding must be explicit - Outcomes should match the claimed benefit and include harms, workload, equity, and patient-important effects where relevant
When an observational study is used to support a causal claim, its protocol should specify the target trial it seeks to emulate: eligibility criteria, intervention strategies, assignment procedure, time zero, follow-up, outcomes, causal contrast, and analysis plan. Target-trial emulation makes the intended comparison and its assumptions inspectable; it does not remove residual or unmeasured confounding (Hernán and Robins, 2016).
Subgroup Analysis (Essential for Equity)
Algorithm performance can vary across clinically relevant subgroups.
Essential subgroups to evaluate: - Demographics: Age, sex, race/ethnicity - Clinical: Disease severity, comorbidities - Socioeconomic: Insurance status, ZIP code - Technical: Different imaging equipment, EHR systems
Documented example: A commercial healthcare risk algorithm systematically underestimated illness burden for Black patients because healthcare costs were used as a proxy for health need (Obermeyer et al., 2019).
Evaluate clinically relevant subgroups before deployment and during monitoring.
Post-Deployment Monitoring
Algorithm performance drifts over time. Continuous monitoring is non-negotiable.
NIST’s CAISI proposed six monitoring categories (functionality, operational, human factors, security, compliance, large-scale impacts) to help teams structure post-deployment monitoring beyond ad hoc dashboards (Rao et al., 2026).
Causes of Drift
- Patient population changes
- Clinical practice evolution
- EHR updates
- Equipment changes
- Seasonal variation
Risk-Based Monitoring Plan
Monitoring frequency should be predefined from clinical risk, use volume, expected drift, feedback latency, and the speed with which harm could accumulate. A high-volume triage system may require near-real-time operational surveillance, while reliable outcome labels may mature only after weeks or months.
At minimum, the plan should specify:
- Data-pipeline integrity, missingness, downtime, and out-of-distribution inputs
- Threshold-specific errors, calibration, alert burden, and subgroup performance
- Exposure, clinician response, overrides, and downstream actions
- Patient-important outcomes and adverse events at a clinically meaningful interval
- Named owners, escalation thresholds, stopping rules, and criteria for recalibration or retirement
For patient-facing or diagnostic AI, patient-reported outcome and experience measures can supplement post-deployment monitoring when the selected instrument is valid for the intended population and use. Dukhanin et al. mapped candidate domains to reports after a specific encounter and cross-sectional reports spanning the diagnostic journey, but emphasized that many domains still require instrument development and validation (Dukhanin et al., 2024). These measures should complement clinical outcomes, record-based surveillance, and adjudicated safety events.
Observability for Generative and Agentic Systems
Generative and agentic systems should be monitored as clinical workflows, not only as base models. Treat system instructions, retrieval sources, connected tools, permissions, and interfaces as components of the deployed workflow, and subject material changes to re-evaluation (Rao et al., 2026).
For an auditable workflow, retain the version identifiers for the model, system instructions, retrieval corpus, tools, and permissions; relevant retrieval and tool-failure signals; clinician review, override, and escalation outcomes; and associated incident records. Logging should follow applicable privacy, security, and retention requirements, with access sufficient for incident reconstruction but without indiscriminate duplication of clinical content.
Define a response for material events before deployment:
- A material model, prompt, retrieval, tool, permission, or workflow change requires the specified re-evaluation before the changed system returns to clinical use.
- A tool action outside the approved scope, unresolved retrieval or tool failure, or safety-related incident triggers review and, where indicated, a pause of the affected workflow.
- Recurrent clinician overrides or disagreements should be reviewed by the accountable clinical owner to determine whether the system, workflow, training, or operating threshold requires change.
LLM-Specific Evaluation: Beyond Benchmark Accuracy
Large language models can achieve high scores on medical benchmarks such as MedQA, but benchmark accuracy does not establish reliable performance in clinical use. One useful stress test asks whether performance persists when superficial answer patterns change.
Evaluation Controls Supported by 2026 Evidence
Five recent studies sharpen the evaluation design for clinical LLMs:
- Separate process quality from patient outcomes. A pragmatic cluster-randomized trial in Kenyan primary care found improvements in several expert-rated process measures with generative AI decision support, while its prespecified 14-day treatment-failure outcome did not differ significantly. The study supports claims about the measured care processes, not improved patient outcomes (Agweyu et al., 2026).
- Pair static tests with valid perturbations. Dynamic red-teaming changed many initially correct answers under adaptive, health-specific stress tests. The attack results depend on mutation validity, search budget, and judging, so they demonstrate brittleness under the tested conditions rather than the prevalence of routine clinical error (Pan et al., 2026).
- Treat evaluators as part of the measurement system. Ratings of responses to real clinical cases varied across physician seniority and practice setting, with changes in relative model ranking. Report evaluator composition and rank stability rather than one pooled expert score (Shi et al., 2026).
- Allow automated judges to abstain. MedQADE, a 2026 preprint, found that automated judges could approach clinician-level agreement while failing to reproduce physicians’ increasing caution on difficult items. Automated evaluation should include uncertainty thresholds, escalation, and model-family bias checks (Philipp et al., 2026, preprint).
- Distinguish contextual relevance from deployment evidence. IyawoBench uses locally grounded Nigerian primary-care triage cases, but its synthetic cases and reference-label process require independent relabeling, prompt-sensitivity analysis, geographic validation, and prospective testing before deployment claims (Gabriel et al., 2026, preprint; Gabriel and Olawuyi, 2026, preprint).
The evaluation protocol should define the clinical claim first, then prespecify outcomes, stress tests, evaluator panels, abstention rules, and monitoring. A strong score on a narrower component cannot substitute for the missing component.
Why This Matters for Clinical Deployment
Novel presentations are common: Clinical medicine constantly presents unfamiliar patterns. A patient with atypical STEMI presentation, a rare medication interaction, or an unusual disease constellation requires reasoning beyond memorized patterns.
Benchmark scores do not establish performance under distribution shift: High MedQA performance may partly reflect familiarity with recurring question and answer patterns.
Model classes differed: The models labeled as reasoning-focused in the study had smaller accuracy reductions, but their performance still changed after answer-format disruption.
Clinical Implications
Benchmark accuracy answers a narrow question - High MedQA scores do not establish performance for a clinical use case
Test with novel scenarios - Evaluate LLM performance on cases that differ from training patterns
Compare model classes empirically - Do not infer clinical reliability from an architecture label
Maintain human oversight - LLMs should support, not replace, physician clinical reasoning
Require stress testing - Ask vendors how performance changes with unfamiliar presentations, altered formats, missing information, and conflicting evidence
Limitations and Context
The study had limitations: small sample size (68 questions), 0-shot evaluation only, and no comparison to human performance on NOTA questions. NOTA-style questions don’t directly simulate clinical practice, where physicians generate differential diagnoses rather than select from predefined options.
The study supports a narrower conclusion: benchmark performance can be sensitive to answer format. Deployment decisions require task-specific clinical evaluation beyond this stress test.
Script Concordance Testing: A Validated Reasoning Benchmark
The Script Concordance Test (SCT) offers a more clinically grounded complement to NOTA-style disruption studies. Adapted from medical education assessment, SCT presents ambiguous clinical scenarios requiring probabilistic reasoning under uncertainty rather than retrieval of established facts. A 2025 study in NEJM AI validated SCT as a benchmark for comparing LLM clinical reasoning against reference panels of practicing physicians (McCoy et al., NEJM AI, 2025).
Why SCT adds a different test: Standard multiple-choice benchmarks emphasize well-defined problems. Clinical practice often requires reasoning under uncertainty, with incomplete information and competing hypotheses. SCT compares responses with the distribution of judgments from an expert panel rather than a single fixed answer key.
Practical implication: For clinical decision support, SCT and answer-format stress tests can complement standard benchmark accuracy. They still do not replace evaluation in the intended workflow.
Human Baselines and Real EHR Second Opinions
Brodeur et al. show what stronger LLM evaluation can look like: generated differential diagnoses, diagnostic reasoning documentation, management reasoning, probabilistic reasoning, physician-adjudicated scoring, human baselines, and a blinded emergency department second-opinion study using unstructured EHR data from 79 randomly selected patients (Brodeur et al., 2026). This is stronger than MedQA-style multiple choice testing because it compares model output against physician performance on tasks closer to clinical reasoning.
The study still does not answer the deployment question. It tested text-based second opinions, not live physician-AI collaboration, triage decisions, disposition, treatment execution, liability, cost-effectiveness, equity, or patient outcomes. A linked Science Perspective framed the next step appropriately: strong text-based reasoning performance should trigger prospective clinical trials and governance work, not autonomous deployment (Hopkins & Cornelisse, 2026).
The Interaction Gap: Benchmarks Don’t Predict Real-World Use
Pattern matching is not the only reason benchmarks overstate clinical utility. A 2026 randomized trial (n=1,298) found that LLMs correctly identified medical conditions in 94.9% of cases when tested alone, but participants using those same LLMs identified relevant conditions in fewer than 34.5% of cases, no better than a control group using internet search. MedQA scores above 80% corresponded to real-world participant scores below 20% for the same clinical topics in several cases. Simulated patient interactions (LLMs role-playing as patients) also failed to predict human performance, producing higher scores and less variability than real users (Bean et al., 2026). Benchmark accuracy, simulated testing, and real-world human interaction measure fundamentally different things.
A 2026 LLM-assisted systematic review of clinical LLM studies found the same evidence gap at field scale: 4,609 peer-reviewed clinical-medicine LLM studies were identified from January 2022 through September 2025, but only 19 were prospective randomized trials. Most studies evaluated simulated scenarios or exam-style tasks rather than patient-centered outcomes, reinforcing that publication volume should not be confused with deployment readiness (Chen et al., 2026).
Evaluation on authentic oncology notes provides a more direct view of reasoning failures. Across two retrospective cohorts spanning breast, pancreatic, and prostate cancer, GPT-4 produced reasoning errors in 23.1% of note interpretations; confirmation bias, anchoring bias, and omission errors were most strongly associated with potentially harmful outputs. GPT-5.1 reduced the overall error rate but retained structured failure patterns, and automated LLM evaluators detected error presence without reliably classifying subtypes (Kenaston et al., 2026). This supports clinician-reviewed error taxonomies on representative clinical notes rather than endpoint accuracy alone.
HealthBench: OpenAI’s Health-Specific Evaluation Framework
In May 2025, OpenAI released HealthBench, an open-source benchmark specifically designed to evaluate LLM performance on health-related tasks (OpenAI et al., 2025, preprint).
Development methodology:
HealthBench was created in collaboration with 262 physicians who have practiced in 60 countries, are proficient in 49 languages, and have training in 26 medical specialties. Unlike traditional medical benchmarks based on multiple-choice exam questions, HealthBench uses conversation-specific rubrics that reflect how clinicians actually judge response quality.
Benchmark structure:
- 5,000 multi-turn conversations between a model and user (patient or healthcare professional)
- 48,562 unique rubric criteria describing attributes of responses that should be rewarded or penalized
- Criteria range from specific facts (medication dosages) to behavioral dimensions (asking clarifying questions, appropriate care escalation)
- Conversations average 2.6 turns, with 58.3% being single-turn (emphasizing complete initial responses)
Evaluation dimensions:
HealthBench stratifies performance across seven themes:
- Expertise-tailored communication (adjusting complexity for lay vs. professional audiences)
- Response depth (appropriate level of detail)
- Emergency referrals (recognizing when to escalate)
- Health data tasks (interpreting lab values, vital signs)
- Global health (cross-cultural and resource-limited contexts)
- Responding under uncertainty (appropriate hedging and caveats)
- Context seeking (asking clarifying questions when needed)
Benchmark variations:
- HealthBench Consensus (3,671 examples): Filtered subset with high physician agreement, useful for studying high-confidence failure cases
- HealthBench Hard (1,000 examples): Challenge subset identified as especially difficult for frontier models
Model performance (2025):
| o3 |
60% |
| GPT-4o |
32% |
| GPT-3.5 Turbo |
16% |
The OpenAI-authored preprint reported that GPT-4.1 nano outperformed GPT-4o on HealthBench and was 25 times cheaper at the cited API prices; this vendor benchmark and pricing comparison does not establish clinical superiority (Arora et al., 2025, preprint).
Human baseline comparison:
Physicians were asked to produce responses both with and without model assistance. The researchers found that recent models produce higher quality responses than physicians unless physicians are assisted by the same models, suggesting LLMs may be most valuable as augmentation tools rather than standalone systems.
Vendor-developed benchmark: HealthBench was created by OpenAI. Models may be optimized specifically for this evaluation, and the benchmark may not capture failure modes that competitors would identify.
Rubric-based evaluation uses LLM grading: Responses are evaluated by GPT-4.1 as the grader, introducing potential bias toward OpenAI model outputs.
Benchmark ≠ clinical validation: High HealthBench scores do not substitute for prospective clinical trials demonstrating patient benefit.
Not FDA-recognized: HealthBench is not an FDA-accepted validation framework for medical device clearance.
When vendors cite HealthBench performance, ask: (1) What is their performance on independent benchmarks? (2) Have they conducted prospective clinical validation? (3) How does performance generalize to your patient population?
HealthBench Professional: Clinician-Specific Evaluation
OpenAI extended the HealthBench framework with HealthBench Professional (OpenAI, HealthBench Professional, 2026), an evaluation specifically designed for clinician-facing AI tasks. Unlike the general HealthBench (5,000 synthetic health conversations), HealthBench Professional uses real conversations written by physicians testing ChatGPT for Clinicians during its development.
Benchmark structure:
- 525 physician-authored tasks
- Three clinician-facing use cases: care consult, writing and documentation, medical research
- Rubric-graded by GPT-5.4 at low reasoning
Performance results:
| ChatGPT for Clinicians (GPT-5.4) |
59.0 |
51.0 |
64.1 |
67.0 |
| Base GPT-5.4 |
48.1 |
N/A |
34.6 |
58.1 |
| Specialty-matched physicians with web access and unlimited time |
43.7 |
42.7 |
32.1 |
56.3 |
The report describes ChatGPT for Clinicians as a GPT-5.4-based system used with clinician-facing product tools. Its scores should be interpreted as results for that system and benchmark configuration, not for base GPT-5.4 or clinical care generally.
The benchmark conversations were written by physicians testing ChatGPT for Clinicians during product development. That development dependence limits benchmark independence and may align the task distribution with the product. The results require corroboration on independent data and in the intended workflow.
MedHELM: Multi-Dimensional Evaluation for Medical LLMs
MedHELM, formally named “Holistic Evaluation of Large Language Models for Medical Tasks,” evaluates LLMs across multiple clinical task categories rather than only medical exam questions (Bedi et al., 2026).
Task taxonomy:
MedHELM organizes evaluations into clinical note generation, patient communication, medical research assistance, clinical decision support, and administration and workflow. Performance varied across tasks and models, so an aggregate benchmark score should not be treated as evidence for every clinical use.
Evaluation method:
The published evaluation used an LLM panel as judges and compared those ratings with clinician ratings. Reported agreement was modest (ICC 0.47 for LLM-judge versus clinician ratings and 0.43 between clinicians), which supports scalable screening but not replacement of clinical review (Bedi et al., 2026).
Stanford Health Care has also described using MedHELM for initial model selection in its ChatEHR program (Shah et al., 2026, preprint). The implementation report notes error patterns that benchmark selection did not resolve, including temporal confusion and numeric-value errors.
Safeguards for LLM-as-a-Judge
An LLM judge may support scalable screening, but it should not be the sole release criterion for a high-stakes clinical use. In a global-health evaluation, the highest-performing LLM judge matched human evaluators on only 4 of 11 criteria (Williams et al., 2026). A clinical LLM evaluation framework used randomized, blinded, task-specific rubric review by practicing physicians and measured reviewer agreement and reproducibility after a washout period (Kocaman et al., 2025). Those controls support local calibration of an LLM judge against clinical review, rather than treating a judge score as a safety determination.
- Pre-specify the rubric, clinically material error categories, decision threshold, and sample of outputs that will receive independent clinician ratings.
- Compare judge and clinician ratings blindly where feasible, report agreement overall and by error category, and investigate clinically consequential disagreement.
- Version-lock the judge model, system instructions, rubric, and threshold. Recalibrate after a material change to any of them or to the evaluated workflow.
- Do not use a judge to assess outputs from the same exact model without documenting and testing for self-preference; record shared developer or model relationships.
- Escalate clinically consequential outputs and unresolved judge-clinician disagreement to qualified human review. A judge score does not authorize autonomous clinical action.
Request results for the tasks that match the intended use, then reproduce the evaluation on independent, representative data. Task-stratified benchmark performance can identify where additional testing is needed, but it does not authorize deployment or establish clinical benefit.
The framework is open source and available at crfm.stanford.edu/helm/medhelm.
MAST: Task-Specific Evaluation of Medical AI
Goh and colleagues propose a task- and context-specific framework for evaluating progress toward medical AI superintelligence. The framework compares systems with strong clinicians or clinical teams across diagnostic reasoning, management reasoning, multimodal interpretation, safety and harm avoidance, and calibration under uncertainty. It gives particular attention to longitudinal, multiturn, and agentic tasks, and identifies patient-important outcomes as the eventual standard for assessing real-world impact (Goh et al., 2026). The supplementary evidence tables place current results in context: the cited head-to-head studies largely use structured vignettes, simulated consultations, retrospective data, or sandboxed EHR tasks. These studies provide evidence about specific capabilities, while clinical safety, effectiveness, and patient outcomes require additional forms of evaluation.
The literature cited by Goh and colleagues illustrates how different study designs answer different questions. A randomized virtual OSCE tested longitudinal disease management across three visits per scenario (Liévin et al., 2026); MedAgentBench tested multi-step actions in a FHIR-compliant virtual EHR (Jiang et al., 2025); and the population-based MASAI trial evaluated AI-supported mammography using interval cancer rate, sensitivity, and specificity (Gommers et al., 2026). Each provides evidence for a defined task and setting, rather than an interchangeable measure of medical capability.
The Medical AI Superintelligence Test (MAST) puts this framework into practice through an ARISE-run benchmark suite and public leaderboard. Its current methodology combines diagnostic reasoning, management reasoning, safety, radiology, and medical-image performance into composite scores. Agentic EHR tasks are reported separately and are not yet included in those composites (ARISE MAST methodology). The suite includes First Do NOHARM for management safety (Wu et al., 2025, preprint), Script Concordance Testing for reasoning under uncertainty, and virtual-EHR benchmarks for agentic tasks. A public repository provides submission tooling and runnable subsets. MAST is currently in preview, and its methodology states that human baselines remain under development and that composite weighting necessarily simplifies clinical performance. Its results are therefore most informative when task-level findings, comparator conditions, judge design, data provenance, and failure modes are considered alongside the chapter’s intended-use evidence framework.
Evidence Gaps in Clinical LLM Evaluation
A systematic review of 519 health care LLM studies published from January 2022 through February 19, 2024 found that 44.5% evaluated medical knowledge through question-answering tasks, while 5.0% used real patient care data. Fairness, bias, or toxicity was evaluated in 15.8% of studies, and calibration or uncertainty in 1.2% (Bedi et al., 2025). These findings describe the studies included in that review; they should not be treated as current prevalence estimates for all clinical AI research.
Evaluation should therefore move beyond exam-style accuracy while remaining clear about what stronger simulations establish. CRAFT-MD uses simulated AI patient agents, automated scoring, and expert review to evaluate conversational diagnostic reasoning across 12 medical specialties (Johri et al., 2025). That design tests a more realistic interaction than a fixed multiple-choice item, but it still does not establish performance in live clinical workflow or effects on patient outcomes.
Clinical implication: Ask whether the evaluation matches the intended clinical task, uses representative data, measures uncertainty and subgroup performance, and tests effects in the intended workflow.
Evaluating Explainability: Beyond the Black Box
Many clinical AI systems are not directly interpretable. Explainable AI (XAI) methods aim to characterize factors associated with model outputs, but a post hoc explanation is not necessarily a faithful account of how a model produced a prediction. For background on XAI techniques, see the AI Basics chapter; this section focuses on evaluating XAI claims in vendor offerings.
Why Explainability Matters for Evaluation
Safe use requires information proportionate to the intended user, context, and risk. Joint guiding principles from the FDA, Health Canada, and the UK Medicines and Healthcare products Regulatory Agency call for transparency that is appropriate to the intended users, risks, and context of use (FDA, 2024). These principles are not binding FDA guidance and do not require the same form of explanation for every system.
Evaluation implications:
- Trust calibration: Appropriate transparency and validated explanations may help users calibrate reliance, but explanation alone does not prevent automation bias
- Error investigation: Feature attributions and other explanations can generate hypotheses about inappropriate model behavior; confirmation requires analysis of data provenance, timing, and model performance
- Regulatory assessment: Evaluate transparency against the intended use, users, risks, and applicable regulatory requirements rather than treating explainability as a universal checkbox
Dominant XAI Methods in Clinical AI
A systematic review of 62 studies published from 2018 through 2025 found that SHAP and LIME were the most frequently used XAI methods in clinical decision support, with SHAP prominent for tabular data and saliency-based methods for imaging (Abbas et al., 2025).
SHAP (SHapley Additive exPlanations): Estimates feature contributions to a prediction under the assumptions of the selected explainer and background data. One study used SHAP to examine a mortality-prediction model among hospitalized patients with sepsis (Zhang et al., 2024).
LIME (Local Interpretable Model-agnostic Explanations): Approximates complex models with simple, interpretable models around specific predictions. Produces intuitive visual explanations for image-based AI.
Saliency maps and attention visualization: Highlight which image regions or text segments influenced predictions. However, attention weights may not faithfully reflect model reasoning (Jain & Wallace, 2019).
For clinical examples of XAI in dermatology (saliency maps revealing spurious correlations with rulers and skin markers), see the Dermatology chapter.
Evaluating Vendor XAI Claims
- What explanation method do you use? (SHAP, LIME, saliency, inherently interpretable model?)
- Are explanations validated? Do they reflect actual model behavior, or are they post-hoc rationalizations?
- Can clinicians act on explanations? Does knowing a feature importance score change clinical decision-making?
- How stable are explanations? Same patient, same input, same explanation every time?
- What are explanation limitations? Honest vendors acknowledge what explanations cannot reveal.
For a structured checklist, the CLIX-M framework provides 14 clinician-informed items for evaluating XAI in clinical decision support (Brankovic et al., NPJ Digital Medicine, 2025). See the Vendor Evaluation appendix for procurement integration.
Red Flags in XAI Evaluation
Explanations highlight implausible features: Patient identifiers, time of day, bed number, or non-clinical metadata suggest spurious correlations rather than clinical reasoning.
Explanations highlight treatment variables: Treatment variables may reflect label leakage, post-outcome information, or a legitimate contemporaneous signal. Investigate feature timing, intended prediction time, and data generation before drawing a conclusion. See Common Validation Pitfalls.
Explanations are unstable: Materially different explanations for the same input should trigger investigation of the explanation method, baseline, stochasticity, and model behavior. Instability alone does not prove that the underlying prediction is unreliable.
Vendor claims “proprietary explainability”: Insufficient disclosure of the method, inputs, assumptions, or validation prevents independent assessment.
Evaluation Conclusion
Explanations are evaluation tools, not proof of model reasoning. Require risk-appropriate transparency and validate any explanation method for the intended task, user, and decision context.
Post Hoc Interpretability Audit of a Reinforcement-Learning Policy
A retrospective study of one reinforcement-learning policy for Medicaid care coordination combined attention analysis, SHAP attribution, and sparse autoencoders to examine divergent recommendations (Basu et al., 2026). Investigators adjudicated 200 divergent cases and grouped them into premise, calibration, and contextual categories. Those categories describe this study and should not be presented as a general taxonomy of clinical AI failure.
The study also reported retrospective estimates from a conformal safety layer. Such estimates depend on the evaluated data, policy, outcome definition, and exchangeability assumptions; they do not establish prospective safety or transportability to another health system. Agreement across attention, SHAP, and sparse-autoencoder analyses can strengthen a hypothesis about model behavior, but it does not prove that a post hoc explanation faithfully represents the model’s internal computation.
Evaluation implication: Use interpretability analyses to identify testable failure hypotheses. Confirm those hypotheses with temporal data review, subgroup and error analyses, prospective monitoring, and evaluation in the intended setting.
Algorithmic Auditing: A Practical Guide
Clinical teams should not rely solely on vendor-reported performance metrics. Algorithmic auditing provides systematic methods for detecting hidden failures, bias, and performance disparities before deployment.
The Medical Algorithmic Audit Framework
The Lancet Digital Health framework (Liu et al., 2022) recommends three core audit components:
- Exploratory error analysis: Identify patterns in model failures (are errors random or systematic?)
- Subgroup testing: Evaluate performance across clinically relevant patient groups
- Adversarial testing: Probe model behavior under edge cases and unusual inputs
Bias Detection Through Comparison Testing
A five-step audit framework developed for LLMs in specific clinical applications illustrates the elements needed for a defensible bias evaluation; it should not be applied unchanged to every clinical AI system (Templin et al., 2025).
The comparison protocol:
- Engage affected stakeholders to define the audit purpose, questions, outcomes, methods, and risk tolerance
- Select the LLM and evaluation design for the intended population, expected effect size, calibration needs, and statistical power
- Test clinically relevant scenarios; use matched inputs only when the counterfactual comparison remains clinically coherent
- Interpret effect sizes and uncertainty against non-AI-assisted clinical decisions, accounting for repeated sampling or multiple comparisons where relevant
- Monitor the deployed system for drift and recurrence of identified disparities
- What is the worst-performing subgroup? Demand vendor disclosure of performance floor, not just average
- How was subgroup performance tested? Single demographics or intersectional combinations?
- What fairness metrics were applied? Equal accuracy, demographic parity, equal opportunity?
- Is there performance monitoring for drift? Thresholds and review frequency should be predefined from clinical risk and statistical uncertainty
- Can we conduct our own audit? Access to model outputs on local data enables institutional validation
Statistical Rigor in Benchmark and Audit Comparisons
The benchmark tables throughout this chapter report accuracy as a single number: 91.1 percent, 86.4 percent, 97.4 percent. A 2024 statistical framework for language model evaluations argues that a bare percentage hides the question a reader actually needs answered: is a reported gap a genuine capability difference, or could it plausibly be sampling noise from the specific question set used (Miller, 2024, preprint)? The framework treats eval questions as a sample drawn from a larger, unseen pool of possible questions, not as a fixed, complete test, and makes four practical recommendations that extend the audit protocol above.
Report a standard error alongside the point estimate. For a simple accuracy score from n independent questions, the standard error is approximately the square root of p(1-p)/n, where p is the observed accuracy. Applied to the NOTA test above (n = 68 questions), GPT-4o’s reported 85.3 percent original accuracy carries a standard error of roughly 4 percentage points, and its 48.5 percent NOTA-modified accuracy carries a standard error of roughly 6 points. A 68-question benchmark score reported without an error bar is missing several points of uncertainty in either direction. This is an illustrative recomputation from the published aggregate percentages, not a reanalysis of the underlying study, and it assumes the 68 questions behave independently; if several questions shared a clinical vignette or answer pattern, as commonly happens in question banks, the true uncertainty would be larger.
Cluster the standard error when questions are correlated. Many benchmarks group several questions under the same passage, case, or vignette. Treating grouped questions as independent systematically understates uncertainty; Miller reports naive standard errors under-stating clustered standard errors by roughly threefold on some public benchmarks. Before citing a benchmark table in a procurement decision, ask whether the underlying question set clusters by case and whether any reported uncertainty accounts for that.
Use paired analysis when the same questions are reused. The NOTA table above is a paired design: each model answered the identical 68 questions twice, once in the original format and once with the correct answer replaced. A model’s accuracy drop is the difference between two measurements on the same items, not two independent samples. Matched-pairs analysis, which uses the correlation between a model’s original and NOTA-modified performance on the same question, is substantially more statistically powerful than treating the “before” and “after” scores as unrelated samples, which is the standard rationale for matched-pairs tests such as McNemar’s test in this kind of same-item, before-and-after design. The same reasoning applies whenever two models are compared on one shared question set, as in the HealthBench and Vishwanath et al. head-to-head results above: a paired comparison of the same questions is the statistically stronger analysis, not two separately reported percentages.
Size the eval to the gap that matters. How many questions an eval needs depends on how small a capability gap it must reliably detect: Miller’s own worked example shows that resolving a 3-percentage-point gap with conventional error rates requires roughly 1,000 questions, far more than most published clinical LLM benchmarks use. The 68-question NOTA study was well suited to the 9 to 38 percentage-point drops it measured, since large effects are detectable with fewer questions, but the same 68-question design would be underpowered to detect a small, single-digit-point difference between two models. Institutions running their own local audits (see “Can we conduct our own audit?” above) should size the question set to the smallest performance gap they need to catch, not to whatever sample is convenient to assemble.
Post-Deployment Surveillance
Algorithm performance can change as data, populations, workflows, and care patterns change. The STANDING Together recommendations primarily address transparency about health data diversity, representativeness, and subgroup evaluation; that information helps institutions judge applicability and plan monitoring (Alderman et al., 2025). Post-deployment controls should follow a risk-based monitoring plan such as the one described earlier in this chapter.
Monitoring checklist:
- Performance and error metrics by clinically relevant subgroup
- Distribution-shift tests selected for the data type and deployment context
- Predefined investigation thresholds that account for uncertainty and clinical consequence
- Clear escalation, pause, recalibration, and retirement pathways
- Periodic independent review at an interval proportionate to risk
Operational example: A 2026 Mayo Clinic review describes 17 internally developed radiology AI algorithms in live clinical use, with monitoring at different cadences for utilization, failure modes, data drift, and end-user feedback through automated alerts and dashboards (Cook et al., Mayo Clinic Proceedings: Digital Health, 2026). This illustrates one institutional implementation, not a universal monitoring schedule. Each organization should define cadence, thresholds, ownership, and response actions from the system’s risk and local workflow.
When to Reject an Algorithm
Auditing may reveal that an algorithm should not be deployed. Red lines include:
- Performance below a predefined safety or utility threshold in a clinically relevant subgroup
- Evidence of harm in validation data (missed diagnoses concentrated in vulnerable populations)
- Vendor refusal to provide subgroup performance data
- No mechanism for local validation or ongoing monitoring
- Validated evaluation demonstrates clinically inappropriate feature reliance (see XAI Red Flags)
The audit is not a checkbox. It is a gatekeeping function that should prevent harmful deployments. A rigorous audit that leads to rejection is more valuable than a superficial audit that enables deployment of a flawed tool.