Evaluating AI Clinical Decision Support Systems

Reported performance does not establish clinical reliability. Epic reported AUROC of 0.76–0.83 for its original sepsis model, but external validation found a hospitalization-level AUROC of 0.63; at threshold 6, sensitivity was 33% (Wong et al., 2021). On MedQA, answer-format disruption reduced standard LLM accuracy by 26.5–38.2 percentage points (Bedi et al., 2025). Clinical decision support therefore requires independent evaluation on the intended task and local population.

Learning Objectives

After completing this chapter, clinicians should be able to:

  • Apply systematic evaluation frameworks to medical AI
  • Distinguish retrospective validation from prospective clinical trials
  • Assess AI performance metrics critically (beyond accuracy)
  • Identify common validation pitfalls and biases
  • Match evidence requirements to the clinical claim and intended use
  • Conduct local pilot testing before full deployment
  • Implement continuous post-deployment monitoring
  • Recognize red flags indicating inadequate validation

The Problem: A high score can answer a narrow performance question while leaving transportability, workflow effects, clinical benefit, and safety unresolved. In a systematic review of 81 diagnostic deep learning studies, 58 were at high risk of bias and only nine were prospective (Nagendran et al., 2020). Final-answer accuracy can also hide pathway failure: MEGA-CDP is a preprint example, not a clinical outcome (Chen et al., 2026, preprint).

Evidence must match the claim:

  1. Technical performance: discrimination, calibration, threshold performance, and error analysis
  2. Transportability: independent temporal, geographic, and multi-site evaluation
  3. Live clinical performance: prospective evaluation of data pipelines, workflow, human factors, and subgroup effects
  4. Clinical utility: comparative evidence that use changes decisions or patient-important outcomes
  5. Lifecycle safety: monitoring for drift, failures, inequitable effects, and unintended consequences

Conditions that should pause deployment:

  • No independently reviewable evidence beyond vendor marketing
  • No evaluation independent of the development data
  • No adequate analysis of clinically relevant subgroups
  • Headline accuracy without the threshold, prevalence, denominator, and error distribution
  • The vendor will not provide the evidence needed to assess the intended use

Core measures:

  • PPV at the expected local prevalence and proposed operating threshold
  • Subgroup performance (age, sex, race, insurance)
  • Calibration of predicted probabilities
  • Clinical impact, including benefits, harms, workload, and patient-important outcomes

Common staged approach, adapted to intended use and risk:

  1. Retrospective testing on local data
  2. Silent prospective testing in the intended workflow
  3. Limited clinical pilot with predefined success and stopping criteria
  4. Full deployment with continuous monitoring

Decision rule: Match the evidence to the claim and deployment risk. Appraise risk of bias and applicability, test in the intended setting, and monitor throughout the product lifecycle. Clinician-driven EHR interactions need live-session monitoring, not benchmark selection alone (Shah et al., 2026).

Introduction

AI Evaluation Decision Tree

Use this flowchart to systematically evaluate any clinical AI tool before deployment:

Clinical AI evaluation decision treeStart: clinical AI use proposed. Ask what claim the evidence must support. Model performance: assess reporting, risk of bias, thresholds, and external evaluation. Workflow performance: run prospective live evaluation with human-factors measures. Patient benefit or harm: use a comparative design suited to causal inference. All three pathways lead to three sequential decisions. Decision 1: population, setting, data, and threshold match intended use? If no, generate local or otherwise directly applicable evidence, then reassess. Decision 2: regulatory status and risk controls appropriate? If no or unclear, resolve regulatory, safety, and governance requirements, then reassess. Decision 3: deployment and monitoring plan is predefined? If no, define workflow, accountability, metrics, and stopping rules, then reassess. If yes to all three, proceed with controlled deployment and surveillance.Clinical AI use proposedWhat claim must the evidence support?Model performanceAssess reporting, risk ofbias, thresholds, andexternal evaluationWorkflow performanceRun prospective liveevaluation withhuman-factors measuresPatient benefit or harmUse a comparative designsuited to causal inferenceDECISION 1Population, setting, data, and thresholdmatch intended use?Generate local or otherwisedirectly applicable evidenceNoReassessYesDECISION 2Regulatory status andrisk controls appropriate?Resolve regulatory, safety,and governance requirementsNo or unclearReassessYesDECISION 3Deployment and monitoringplan is predefined?Define workflow,accountability, metrics,and stopping rulesNoReassessYesProceed with controlled deploymentand surveillanceEvidence pathwayDecisionProceedCorrective action (returns to the same decision)
Clinical AI evaluation decision treeStart: clinical AI use proposed. Ask what claim the evidence must support. Model performance: assess reporting, risk of bias, thresholds, and external evaluation. Workflow performance: run prospective live evaluation with human-factors measures. Patient benefit or harm: use a comparative design suited to causal inference. All three pathways lead to three sequential decisions. Decision 1: population, setting, data, and threshold match intended use? If no, generate local or otherwise directly applicable evidence, then reassess. Decision 2: regulatory status and risk controls appropriate? If no or unclear, resolve regulatory, safety, and governance requirements, then reassess. Decision 3: deployment and monitoring plan is predefined? If no, define workflow, accountability, metrics, and stopping rules, then reassess. If yes to all three, proceed with controlled deployment and surveillance.Clinical AI use proposedWhat claim must the evidence support?Model performanceAssess reporting, risk of bias,thresholds, and external evaluationWorkflow performanceRun prospective live evaluationwith human-factors measuresPatient benefit or harmUse a comparative designsuited to causal inferenceDECISION 1Population, setting, data, and thresholdmatch intended use?Generate local or otherwisedirectly applicable evidenceNoReassessYesDECISION 2Regulatory status andrisk controls appropriate?Resolve regulatory, safety,and governance requirementsNo or unclearReassessYesDECISION 3Deployment and monitoringplan is predefined?Define workflow, accountability,metrics, and stopping rulesNoReassessYesProceed with controlled deploymentand surveillanceEvidence pathwayDecisionProceedCorrective action (returns to the same decision)
Figure 27.1: Clinical AI Evaluation Decision Tree

Reading the flowchart: The first question is the claim, not the study label. A retrospective external validation can support transportability claims, a prospective live study can test workflow performance, and a comparative trial can test whether using the system changes outcomes. Regulatory authorization is a separate assessment and does not substitute for evidence in the intended setting.

Match Evidence to the Claim

Clinical AI evidence is multidimensional, not a single ladder. Study design should be judged against the question it is meant to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE applies certainty ratings to a body of evidence for a specified outcome, not to a journal, vendor, or single paper (CDC ACIP GRADE Handbook, 2024).

Evidence question Evidence needed Appropriate appraisal or reporting tools
Does the model perform as specified? Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance TRIPOD+AI for prediction models, STARD-AI for diagnostic accuracy, and PROBAST+AI for risk of bias and applicability (Collins et al., 2024; Sounderajah et al., 2025; Moons et al., 2025)
Does performance transfer? Independent temporal, geographic, and multi-site evaluation in populations and data systems relevant to intended use External validation with explicit assessment of case mix, prevalence, measurement, calibration, and operating thresholds
Does the system work in the clinical workflow? Prospective evaluation of data pipelines, clinician interaction, adoption, workload, failure modes, and subgroup effects DECIDE-AI for early-stage live clinical evaluation (Vasey et al., 2022)
Does using the system improve care? Comparative evidence on decisions, processes, patient-important outcomes, and harms Randomized trials when feasible, or a justified quasi-experimental design; CONSORT-AI for reports of AI intervention trials (Liu et al., 2020)
Does benefit persist after deployment? Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, and unintended consequences Risk-based surveillance across the product lifecycle (FDA, 2025)

Reporting guidelines improve transparency, but complete reporting does not by itself establish low risk of bias or clinical benefit. Risk-of-bias tools must also match the design: PROBAST+AI for prediction models, QUADAS-3 for diagnostic accuracy studies, RoB 2 for randomized trials, and ROBINS-I for nonrandomized intervention studies (Moons et al., 2025; Whiting et al., 2026; Cochrane RoB 2; Sterne et al., 2016). GRADE then assesses certainty across a body of evidence for a specified outcome. Likewise, an RCT is suited to a causal claim about using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, or long-term drift. Evidence requirements should increase with the consequence of error, degree of autonomy, and difference between the study setting and intended use.

A 2026 npj Digital Medicine framework sequences medical-AI evaluation into five gated phases (technical validation, operational robustness, controlled interaction, clinical evidence, and real-world integration), with fall-back and re-entry triggers when drift, version change, or safety signals appear (Ye et al., 2026). Architecture proposal, not a regulatory checklist or evidence that any deployed system has cleared all five phases.

A parallel NEJM AI Perspective maps clinical-AI claims to five evidence phases from model development through post-deployment monitoring, and warns that strong technical performance or regulatory authorization alone does not establish clinical readiness or patient benefit (Nenadic et al., 2026). Perspective framework: use it to size the claim to the phase, not as proof any tool has cleared the ladder.

For conversational and LLM systems, two 2026 commentaries sharpen the same claim-to-evidence rule. Prospective evidence for conversational medical AI is hard, but non-negotiable: trust cannot be benchmarked into existence and must be earned in real-world clinical settings where the hardest lessons often concern the humans and workflows around the model (Schaekermann et al., 2026). In parallel, Mahajan, Shah, and Powell argue that clinical LLMs break static evidence timelines because models iterate on weeks-to-months cycles while trials take years, so evaluation should emphasize standardized prospective studies, validated benchmarks linked to clinical endpoints, and post-deployment monitoring of defined tasks and human-AI systems, without lowering evidentiary standards (Mahajan et al., 2026).

Clinical utility examples: The CONCERN early warning system was evaluated in a pragmatic cluster-randomized trial across 60,893 hospital encounters in 74 clinical units; the published article reported lower mortality risk and shorter length of stay with the intervention (Rossetti et al., 2025). A 2026 author correction revised the length-of-stay effect estimate after a modeling error but did not change its statistical significance (Rossetti et al., 2026). GOLDEN BRIDGE II, a cluster-randomized trial of 21,603 patients with acute ischemic stroke in 77 hospitals, found fewer new vascular events at 3 months with an integrated clinical decision support system than with usual care (2.9% vs. 3.9%; adjusted HR 0.74) (Zhang et al., 2026). These trials support causal claims about the evaluated interventions and settings; they do not validate every AI system or guarantee transportability elsewhere.

20 Questions for Evaluation

Before deployment, obtain sufficiently detailed answers to the following questions.

Questions for Developers and Vendors

About the Data:

  1. How many patients or encounters were included in the development data, and from how many institutions?
  2. What calendar period did the data cover?
  3. How representative are the data of the intended population and setting?
  4. What inclusion and exclusion criteria were applied?
  5. How were labels or reference standards established and adjudicated?
  6. What is known about label error or interrater agreement?

About Validation:

  1. Was the locked model evaluated at institutions independent of development?
  2. Was temporal validation performed on a later patient cohort?
  3. Was the system evaluated prospectively in the intended workflow?
  4. What inclusion and exclusion criteria were used for each evaluation?
  5. What performance and uncertainty estimates were reported for clinically relevant subgroups?

About Performance:

  1. What are sensitivity and specificity at the proposed operating threshold?
  2. What is PPV at the expected prevalence in the intended population?
  3. How calibrated are probability predictions?
  4. What alert volume and false-alert burden are expected locally?
  5. What comparative evidence supports an effect on decisions, workflow, or patient-important outcomes?

About Deployment:

  1. How does the system change the clinical workflow, workload, and allocation of responsibility?
  2. How does the system detect and respond to data outside its validated range?
  3. How is performance monitored post-deployment?
  4. What is the update, change-control, and maintenance plan?

Common Validation Pitfalls

These patterns can inflate development performance or reduce applicability in deployment.

Selection Bias

Problem: Training only on patients who received gold standard test.

Example: Biopsy-confirmed cancer AI trained only on lesions suspicious enough to biopsy.

Result: Misses spectrum of disease severity in real practice.

Temporal Bias

Problem: Train on old data, validate on old data.

Result: Medical practice evolves; algorithm becomes obsolete before deployment.

Site-Specific Overfitting

Problem: Works at Institution A, fails at Institution B.

Cause: Different EHRs, imaging equipment, patient populations, documentation practices.

Evaluation response: Use multi-site external validation when the claim requires transportability across institutions.

Label Leakage

Problem: Training labels contain information not available at prediction time.

Example: Sepsis prediction using antibiotics administered (clinician already diagnosed sepsis).

Evidence of prevalence: A literature review found 40.2% of MIMIC-based same-admission prediction models used ICD codes as features, even though these codes are finalized only after discharge. Models using only ICD codes achieved AUROCs of 0.97–0.98 for mortality prediction, with the most “predictive” codes being brain death, cardiac arrest, and palliative care encounter (Ramadan et al., 2025).

Result: Inflated performance that may not reproduce prospectively.

Publication Bias

Problem: Only positive results published.

Result: True performance lower than literature suggests.

Outcome Definition Shifts

Problem: Training outcome differs from deployment outcome.

Example: Train to predict ICD codes, deploy to predict actual clinical deterioration.

Benchmark vs. Reasoning (LLM-specific)

Problem: High accuracy on medical benchmarks may reflect pattern matching, not clinical reasoning.

Evidence: In a 68-question MedQA test, answer-format disruption reduced standard LLM accuracy by 26.5–38.2 percentage points (Bedi et al., 2025).

Interpretation: Answer-format sensitivity limits what benchmark accuracy alone can establish.

A 27 August 2026 preprint, MEGA-CDP, scored 16 LLMs on 42,353 synthetic cases built from 2,274 English and Chinese guidelines (2020–2025) and measured pathway consistency separately from the terminal recommendation (Chen et al., 2026, preprint). Multi-turn outcome accuracy fell by 40.51 percentage points on average versus single-turn vignettes, and final-answer ranks did not match pathway-consistency ranks. A correct terminal answer does not show that the model gathered the right information or followed the supplied guideline; the cases are synthetic and the pathways were extracted automatically.

Why External Validation Matters

Clinical AI evaluation still relies heavily on retrospective accuracy measured on curated datasets, with less attention to workflow fit, adoption, safety controls, or downstream care effects (Azad et al., 2026). Internal validation cannot establish transportability because performance may change with case mix, prevalence, measurement practices, equipment, and data pipelines.

FDA authorization does not guarantee validation quality. An analysis of 130 FDA-authorized AI medical devices (2015–2020) identified important evidence gaps (Wu et al., 2021):

  • 126 of 130 devices (97%) were evaluated using only retrospectively collected data
  • Only 13% of the available summaries reported evaluation across demographic subgroups
  • In the study’s accompanying chest-radiograph case analysis, AUROC declined by 0.03–0.12 when site-trained models were evaluated at external sites
  • Public summaries often lacked information needed to judge multi-site testing, sample size, and subgroup performance

In this 2015–2020 cohort, prospective clinical evaluation was uncommon and important information about performance across settings and patient populations was often unavailable.

Training-cohort geography. Even multi-site US studies may draw patients from a narrow geography. A JAMA research letter mapped geographically identifiable US cohorts in 74 peer-reviewed image-based deep learning studies (2015–2019) across radiology, ophthalmology, dermatology, pathology, gastroenterology, and cardiology (Kaushal et al., 2020). Among 56 studies with identifiable locations, California appeared in 22 (39%), Massachusetts in 15 (27%), and New York in 14 (25%); 40 studies (71%) used at least one cohort from these three states, while 34 states contributed no identifiable cohorts (Kaushal et al., 2020). This mapping describes training-cohort geography, not proof that every model fails outside those states. US-trained or multi-site performance claims still require external validation in the deployment setting.

Recent regulatory guidance on adaptive AI: The FDA finalized its Predetermined Change Control Plan (PCCP) guidance in August 2025. A PCCP can describe specified modifications and the methods used to develop, validate, and implement them as part of a marketing submission (FDA PCCP Guidance, August 2025). The FDA’s January 2025 lifecycle and marketing-submission guidance for AI-enabled device software functions remains a draft and is not for implementation (FDA Draft Guidance, January 2025).

Case Study: Epic Sepsis Model

Vendor-reported performance for ESM v1: AUROC 0.76–0.83 (Wong et al., 2021).

External validation of ESM v1: A single-center retrospective study at Michigan Medicine evaluated 38,455 hospitalizations among 27,697 patients and found a hospitalization-level AUROC of 0.63; at threshold 6, sensitivity was 33% and PPV was 12% (Wong et al., 2021).

Updated evidence for ESM v2: A multicenter prospective validation across four U.S. health systems found encounter-level AUROC of 0.82–0.92; at site-specific thresholds selected for 60% sensitivity, PPV was 13%–26%, with substantial site variation and high alert burden (Wong et al., 2026).

Lesson: Validate both discrimination and clinical burden locally before deployment.

Performance Metrics for Clinicians

Understanding these metrics supports interpretation of vendor claims and identification of misleading statistics.

Accuracy (Often Misleading)

Formula: (TP + TN) / Total

Problem: Disease prevalence affects interpretation dramatically.

Example: - Cancer prevalence: 1% - Algorithm always predicts “no cancer”: 99% accuracy but clinically useless

Never use accuracy alone for rare outcomes.

Sensitivity (True Positive Rate)

Formula: TP / (TP + FN)

What it measures: % of actual positives correctly identified

When critical: Screening tests, rule-out situations (do not miss cancers)

Threshold trade-off: Lowering a decision threshold to increase sensitivity usually increases false positives

Specificity (True Negative Rate)

Formula: TN / (TN + FP)

What it measures: % of actual negatives correctly identified

When critical: Avoiding unnecessary workups, rule-in tests

Threshold trade-off: Raising a decision threshold to increase specificity usually increases false negatives

Positive Predictive Value (PPV)

Formula: TP / (TP + FP)

What it measures: If test is positive, what’s probability patient actually has disease?

Critical insight: PPV depends on disease prevalence in the intended population.

Example showing prevalence impact (with 90% sensitivity, 90% specificity):

Prevalence PPV Interpretation
50% 90% Excellent
10% 50% Half of positives are false
1% 8% 92% of positive results are false positives

Ask for PPV at the intended operating threshold and expected local prevalence. Also examine sensitivity, false negatives, alert volume, and the consequences of each error type.

AUC-ROC (Area Under Curve)

What it measures: Overall discrimination across all possible thresholds

Range: 0 to 1. A value of 0.5 indicates chance discrimination; a value below 0.5 indicates ranking opposite to the specified outcome direction.

Interpretation: A higher AUROC indicates better ranking of positive versus negative cases across thresholds, but no universal band establishes clinical adequacy. Acceptability depends on the comparator, intended use, case mix, operating threshold, and consequences of error.

Limitations: - Does not tell you performance at specific clinical threshold - Can be high even when PPV is poor at low prevalence - Does not capture calibration

Calibration (Often Overlooked)

What it measures: Do predicted probabilities match observed frequencies?

Example: - Good calibration: AI predicts “30% mortality risk” for 1000 patients → ~300 actually die - Poor calibration: Predicted 30%, but 50% actually die (underestimates risk)

Why it matters: Poorly calibrated models produce misleading probabilities, hampering clinical decisions.

Study Design and Clinical Impact

Match Study Design to the Question

Retrospective evaluation: - Appropriate for model development, initial validation, temporal or geographic external validation, and failure analysis - Main concerns include selection bias, data leakage, unstable labels, missingness, and mismatch with intended use - Cannot establish the causal effect of introducing the system into care

Prospective silent evaluation: - Tests the locked system and data pipeline on new patients without exposing clinicians to its output - Measures live performance, alert burden, data failures, and transportability - Cannot establish how clinicians will use the output or whether patient outcomes will improve

Prospective live evaluation: - Measures clinician interaction, workflow fit, adoption, overrides, errors, and safety in the intended setting - Early-stage studies can establish feasibility and reveal implementation failure modes - Without a concurrent comparator, changes in outcomes may reflect secular trends or confounding

Comparative impact evaluation: - Randomized designs provide the clearest causal estimate when randomization is feasible - Stepped-wedge, interrupted time-series, or controlled before-after designs may be justified when randomization is impractical, but their assumptions and residual confounding must be explicit - Outcomes should match the claimed benefit and include harms, workload, equity, and patient-important effects where relevant

When an observational study is used to support a causal claim, its protocol should specify the target trial it seeks to emulate: eligibility criteria, intervention strategies, assignment procedure, time zero, follow-up, outcomes, causal contrast, and analysis plan. Target-trial emulation makes the intended comparison and its assumptions inspectable; it does not remove residual or unmeasured confounding (Hernán and Robins, 2016).

Technical Performance ≠ Clinical Impact

The gap: High technical performance does not guarantee clinical benefit.

First-generation mammography CAD: - Rapid clinical adoption preceded strong evidence of benefit - In an observational analysis of 625,625 digital screening mammograms interpreted by 271 radiologists at 66 facilities, CAD was not associated with improved sensitivity, specificity, or cancer detection (Lehman et al., 2015)

Require evidence that directly supports the claimed clinical benefit.

Subgroup Analysis (Essential for Equity)

Algorithm performance can vary across clinically relevant subgroups.

Essential subgroups to evaluate: - Demographics: Age, sex, race/ethnicity - Clinical: Disease severity, comorbidities - Socioeconomic: Insurance status, ZIP code - Technical: Different imaging equipment, EHR systems

Documented example: A commercial healthcare risk algorithm systematically underestimated illness burden for Black patients because healthcare costs were used as a proxy for health need (Obermeyer et al., 2019).

Evaluate clinically relevant subgroups before deployment and during monitoring.

Local Validation: Before You Deploy

External validation at other sites does not guarantee performance in a new institution. The need for local evidence should be determined by intended use, consequence of error, autonomy, data-system differences, and the directness of existing evidence. High-impact systems should not influence care until performance and workflow have been assessed in the intended setting.

Phase 1: Retrospective Local Testing

  • Test the algorithm on local historical data
  • Measure performance metrics
  • Identify failure modes
  • Calculate expected false positive rate

Phase 2: Silent Mode Prospective Testing

A “silent trial” runs the AI model on live patient data in real time while keeping outputs invisible to the treating team, so predictions cannot influence care. This phase tests whether retrospective performance holds under real-world conditions.

A scoping review of 75 silent evaluations across 16 countries reported a common pattern of lower performance in live or prospective evaluation, alongside substantial heterogeneity in methods and reporting (Tikhomirov et al., 2026). Most studies emphasized technical metrics, while fewer addressed clinical ground-truth verification, stakeholder engagement, or human-computer interaction. This supports silent evaluation as a check on local performance and data pipelines, but it does not establish that every model will decline.

Minimum silent trial components:

  • Algorithm runs in background (outputs not shown to clinicians)
  • Compare AI predictions to actual clinical outcomes
  • Assess performance on real-time data
  • Measure potential alert burden (false positive rate at operating threshold)
  • Test subgroup performance across demographics (race, sex, age, comorbidities): most silent trials omit this
  • Monitor data pipeline stability, model downtime, and data drift
  • Verify clinical ground truth labels with independent expert review, not just automated comparison to EHR codes

Common pitfall: Treating the silent phase as purely technical. The CANAIRI collaboration advocates “translational trials” that also assess stakeholder readiness, workflow fit, and implementation barriers before exposing patients to AI-influenced care (McCradden et al., 2025).

Phase 3: Limited Clinical Pilot

  • Deploy to small user group
  • Close monitoring
  • Collect user feedback
  • Track clinical impact

Pilot design must measure use, not only availability. The PRIMA-AI randomized trial embedded a machine-learning 1-year graft-loss risk prediction model into the EHR for kidney transplant recipients with advanced graft dysfunction. The intervention did not increase patient-reported conversations about treatment options after graft loss during 12 months (39% intervention vs. 40% control), and no significant secondary clinical or shared-decision-making benefits were observed. Post-study feedback showed low and variable clinician uptake with workflow barriers (Osmanodja et al., 2026).

Evaluation lesson: An AI prediction placed in the EHR is not an intervention by itself. Clinical trials should measure whether clinicians see, trust, discuss, and act on the output, and whether the interface directly supports the intended workflow.

Phase 4: Full Deployment

  • Gradual rollout
  • Continuous monitoring
  • Performance review at a predefined, risk-based interval

Red Flags and Stop Signs

Pause Deployment If

No independently reviewable evidence beyond vendor marketing

No evaluation independent of the development data when required to support the intended use and risk

Vendor refuses to share performance evidence needed to assess the intended use

No adequate analysis of clinically relevant subgroups

Headline accuracy without the operating threshold, prevalence, denominator, or error distribution

No prospective evaluation when the claimed benefit depends on workflow, clinician use, or clinical impact

Material mismatch between the validation data and intended population, setting, or data pipeline remains unresolved

No plan for performance monitoring post-deployment

Available information about intended use, inputs, limitations, and outputs is insufficient for safe use and oversight

Required regulatory authorization is absent or does not cover the intended use

Poor customer references (other physicians had bad experiences)

Vendor pressures rapid deployment (no time for proper evaluation)

Post-Deployment Monitoring

Algorithm performance drifts over time. Continuous monitoring is non-negotiable.

NIST’s CAISI proposed six monitoring categories (functionality, operational, human factors, security, compliance, large-scale impacts) to help teams structure post-deployment monitoring beyond ad hoc dashboards (Rao et al., 2026).

A Cell perspective catalogs recurring reliability failures in predictive and generative AI for health and medicine, including erroneous outputs, clinically unjustified performance differences across groups or settings, and degradation after deployment (Salaudeen, Zhang, Kim, Singh, and Ghassemi, 2026). The authors argue that familiar technical patches alone are insufficient and call for lifecycle-aware evaluation, continuous monitoring, and institutional governance rather than one-time model scores. Failure taxonomy and governance argument, not a prospective outcomes trial or proof that a named vendor tool fails.

Causes of Drift

  • Patient population changes
  • Clinical practice evolution
  • EHR updates
  • Equipment changes
  • Seasonal variation

Risk-Based Monitoring Plan

Monitoring frequency should be predefined from clinical risk, use volume, expected drift, feedback latency, and the speed with which harm could accumulate. A high-volume triage system may require near-real-time operational surveillance, while reliable outcome labels may mature only after weeks or months.

At minimum, the plan should specify:

  • Data-pipeline integrity, missingness, downtime, and out-of-distribution inputs
  • Threshold-specific errors, calibration, alert burden, and subgroup performance
  • Exposure, clinician response, overrides, and downstream actions
  • Patient-important outcomes and adverse events at a clinically meaningful interval
  • Named owners, escalation thresholds, stopping rules, and criteria for recalibration or retirement

For patient-facing or diagnostic AI, patient-reported outcome and experience measures can supplement post-deployment monitoring when the selected instrument is valid for the intended population and use. Dukhanin et al. mapped candidate domains to reports after a specific encounter and cross-sectional reports spanning the diagnostic journey, but emphasized that many domains still require instrument development and validation (Dukhanin et al., 2024). These measures should complement clinical outcomes, record-based surveillance, and adjudicated safety events.

Observability for Generative and Agentic Systems

Generative and agentic systems should be monitored as clinical workflows, not only as base models. Treat system instructions, retrieval sources, connected tools, permissions, and interfaces as components of the deployed workflow, and subject material changes to re-evaluation (Rao et al., 2026).

For an auditable workflow, retain the version identifiers for the model, system instructions, retrieval corpus, tools, and permissions; relevant retrieval and tool-failure signals; clinician review, override, and escalation outcomes; and associated incident records. Logging should follow applicable privacy, security, and retention requirements, with access sufficient for incident reconstruction but without indiscriminate duplication of clinical content.

Define a response for material events before deployment:

  • A material model, prompt, retrieval, tool, permission, or workflow change requires the specified re-evaluation before the changed system returns to clinical use.
  • A tool action outside the approved scope, unresolved retrieval or tool failure, or safety-related incident triggers review and, where indicated, a pause of the affected workflow.
  • Recurrent clinician overrides or disagreements should be reviewed by the accountable clinical owner to determine whether the system, workflow, training, or operating threshold requires change.

A 2026 Nature Medicine on-premise clinical-agent study couples institutionally governed local deployment with multi-perspective decision-time reliability (internal likelihood, language cues, and behavioral consistency). On MIMIC-IV MIRA-v2, the best open-weight on-prem model (Qwen-3.5) reached about 90.0% diagnostic accuracy versus 90.7% for a cloud GPT-5.2 baseline under a matched agent architecture; on CDM, Qwen-3.5 reached 83.8% (Zhang et al., 2026). Diagnostic behavioral consistency (ConsistencyDx) best discriminated correctness (AUC 0.860) and remained informative under informational-scarcity stress (AUC 0.875). At a consistency threshold of 0.90, the workflow retained 49.4% of cases at 98.9% accuracy for selective autonomy, at roughly five-fold token cost versus single-pass inference (Zhang et al., 2026). This is a retrospective simulation with residual high-consistency errors and a descriptive age-accuracy gradient, not a prospective outcomes trial; absolute thresholds require local calibration.

Triggers for Immediate Review

  • Sudden performance drop
  • User complaints spike
  • Adverse events possibly related to AI
  • Major EHR/equipment changes

Regulatory and Economic Considerations

FDA Oversight

Regulatory status depends on the software function, intended user, intended use, and risk. Some CDS functions are excluded from the federal device definition, while other software functions are regulated as devices (FDA, 2026). For a regulated function, verify the exact 510(k), De Novo, or PMA record, indications for use, limitations, and labeling in FDA sources.

A 510(k) clearance is based on substantial equivalence to a legally marketed predicate, and FDA notes that clinical data are not needed for most devices cleared through this pathway (FDA). Regulatory authorization therefore answers a different question from local transportability, workflow fit, or comparative clinical benefit.

Economic Evaluation

Cost considerations: - Licensing fees (annual, per-study, per-patient) - Hardware/infrastructure - Personnel (implementation, training, monitoring) - Ongoing maintenance

Benefit considerations: - Time savings (value physician time) - Improved outcomes (reduced complications, readmissions) - Quality metrics (value-based care bonuses) - Avoided costs from preventable failures, low-value alerts, and workflow disruption

Require a transparent clinical and economic case.

Practical Evaluation Checklist

Step-by-Step AI Evaluation

Step 1: Literature Review - Search for relevant peer-reviewed, regulatory, and post-deployment evidence - Assess reporting quality, risk of bias, directness, and applicability - Identify independent evidence and disclose funding and conflicts of interest

Step 2: Vendor Assessment - Request detailed validation reports - Ask the 20 questions above - Verify the applicable FDA record, labeling, and intended use - Review evidence and experience from other implementing institutions

Step 3: Institutional Review - Privacy, security, legal, regulatory, and liability review proportionate to the intended use - Review insurance implications when applicable - Clinical and informatics assessment of workflow, integration, and accountability

Step 4: Local Retrospective Testing - Test on local data - Measure performance - Identify failures

Step 5: Prospective Silent Testing - Real-time testing without clinical use (outputs invisible to clinicians) - Monitor for drift, data pipeline stability, and model downtime - Test subgroup performance (demographics, comorbidities) - Verify outputs against independent clinical ground truth

Step 6: Limited Pilot - Small group deployment - Close monitoring - User feedback

Step 7: Decision Point - Full deployment, modify, or discontinue - Document decision rationale

Step 8: Continuous Monitoring - Performance reviews at a predefined interval proportionate to risk - Periodic evaluation of performance, workflow effects, harms, and continued clinical value

Resources

FDA Device Database: 510(k) Clearances

Reporting Guidelines: - TRIPOD+AI for prediction model studies using regression or machine learning (Collins et al., 2024) - TRIPOD-LLM for studies using large language models (Gallifant et al., 2025) - STARD-AI for AI-centered diagnostic accuracy studies (Sounderajah et al., 2025) - DECIDE-AI for early-stage live clinical evaluation of AI decision support (Vasey et al., 2022) - Validation Accords call for a consensus generative-AI validation framework (Huo et al., 2026). Invitation to build standards, not a finished checklist or regulatory requirement - SPIRIT-AI for AI intervention trial protocols and CONSORT-AI for trial reports (Cruz Rivera et al., 2020; Liu et al., 2020)

Appraisal and Evidence Standards: - PROBAST+AI for quality, risk of bias, and applicability in prediction model studies (Moons et al., 2025) - QUADAS-3 for risk of bias and applicability in diagnostic accuracy studies (Whiting et al., 2026) - RoB 2 for randomized trials and ROBINS-I for nonrandomized intervention studies (Sterne et al., 2016) - GRADE for certainty across a body of evidence for specified outcomes - NICE Evidence Standards Framework for digital health evidence across design, performance, value, and deployment

Professional Organizations: - AMA augmented-intelligence policy and guidance - AMA eight-step health-system AI governance toolkit - ACR-SIIM Practice Parameter for Imaging Artificial Intelligence, approved in May 2026 and scheduled to take effect October 1, 2026

LLM-Specific Evaluation: Beyond Benchmark Accuracy

Large language models can achieve high scores on medical benchmarks such as MedQA, but benchmark accuracy does not establish reliable performance in clinical use. One useful stress test asks whether performance persists when superficial answer patterns change.

Exam-style and vignette scores still mostly test medical knowledge. Residency-level benchmarks ask whether an agent can safely act in clinical workflows (calibration, escalation, and clean handoffs) before it earns autonomy in the EHR (Lancet Digital Health, 2026). That is a design claim for evaluation, not evidence that any deployed agent meets it.

Evaluation Controls Supported by 2026 Evidence

Three recent peer-reviewed studies sharpen the design of clinical LLM evaluations:

  1. Separate process quality from patient outcomes. A pragmatic cluster-randomized trial in 16 Kenyan primary-care facilities enrolled 9,691 patients under 103 clinical officers. LLM assistance did not significantly reduce the prespecified 14-day treatment-failure outcome (2.2% vs. 2.0%; adjusted odds ratio 0.77, 95% CI 0.55–1.08), although several documentation and management-process measures differed between groups. The trial supports the process measures it actually assessed, not a claim of improved patient outcomes (Agweyu et al., 2026). A multi-country RCT (N=249 physicians in Indonesia, Kenya, and the Netherlands) found GPT-4o assistance improved vignette performance versus a no-resource control, with the largest lift in Kenya (+18 percentage points) and smaller gains in Indonesia (+10.7) and the Netherlands (+7.2) (Rounding et al., 2026). Controlled vignettes: not bedside outcomes; harms not assessed; control lacked usual guidelines/internet.

  2. Pair static benchmarks with valid stress tests. Dynamic red-teaming of 15 models found substantial failure under adaptive health-specific perturbations, including a median 94% failure rate among initially correct MedQA answers in the study’s robustness audit. Those results depend on the validity of the mutations, attack budget, and automated judging; they demonstrate brittleness under the tested conditions rather than the prevalence of error in clinical care (Pan et al., 2026).

  3. Treat evaluators as part of the measurement system. Physician rankings of model-generated clinical interpretations varied with seniority and practice setting, while automated evaluators were more homogeneous. The study did not establish which evaluator subgroup was more clinically valid. Report evaluator composition, agreement, uncertainty, and rank stability rather than one pooled expert score (Shi et al., 2026).

The evaluation protocol should define the clinical claim first, then prespecify endpoints, stress-test validity, evaluator panels, abstention and escalation rules, and post-deployment monitoring. A strong result on one component cannot substitute for a missing component.

The NOTA Test: Answer-Format Sensitivity

A 2025 study in JAMA Network Open tested whether high benchmark performance reflects genuine clinical reasoning or sophisticated pattern recognition (Bedi et al., 2025).

Methodology:

The researchers took 68 clinician-validated MedQA questions and replaced the correct answer with “None of the other answers” (NOTA). The clinical content and intended answer were preserved while the response format changed.

The test: Greater performance loss after the NOTA manipulation indicates sensitivity to answer format. It does not, by itself, determine whether a model reasons or predict how the model will perform in clinical care.

Results:

Model Original Accuracy NOTA-Modified Accuracy Accuracy Drop
DeepSeek-R1 (reasoning) 92.7% 83.8% 8.8%
o3-mini (reasoning) 95.6% 79.4% 16.2%
Claude-3.5 Sonnet 88.2% 61.8% 26.5%
Gemini-2.0-Flash 92.7% 58.8% 33.8%
GPT-4o 85.3% 48.5% 36.8%
Llama-3.3-70B 80.9% 42.7% 38.2%

Key findings:

  • All models showed statistically significant accuracy drops when NOTA replaced the correct answer
  • Standard LLMs (GPT-4o, Claude, Gemini, Llama) dropped 26-38 percentage points
  • Reasoning-focused models (DeepSeek-R1, o3-mini) showed greater resilience but still degraded (9-16 points)
  • The findings quantify benchmark sensitivity to answer format; they do not estimate clinical reliability

Why This Matters for Clinical Deployment

Novel presentations are common: Clinical medicine constantly presents unfamiliar patterns. A patient with atypical STEMI presentation, a rare medication interaction, or an unusual disease constellation requires reasoning beyond memorized patterns.

Benchmark scores do not establish performance under distribution shift: High MedQA performance may partly reflect familiarity with recurring question and answer patterns.

Model classes differed: The models labeled as reasoning-focused in the study had smaller accuracy reductions, but their performance still changed after answer-format disruption.

Clinical Implications

LLM Evaluation Checklist

  1. Benchmark accuracy answers a narrow question - High MedQA scores do not establish performance for a clinical use case

  2. Test with novel scenarios - Evaluate LLM performance on cases that differ from training patterns

  3. Compare model classes empirically - Do not infer clinical reliability from an architecture label

  4. Maintain human oversight - LLMs should support, not replace, physician clinical reasoning

  5. Require stress testing - Ask vendors how performance changes with unfamiliar presentations, altered formats, missing information, and conflicting evidence

Limitations and Context

The study had limitations: small sample size (68 questions), 0-shot evaluation only, and no comparison to human performance on NOTA questions. NOTA-style questions do not directly simulate clinical practice, where physicians generate differential diagnoses rather than select from predefined options.

The study supports a narrower conclusion: benchmark performance can be sensitive to answer format. Deployment decisions require task-specific clinical evaluation beyond this stress test.

Script Concordance Testing: A Validated Reasoning Benchmark

The Script Concordance Test (SCT) offers a more clinically grounded complement to NOTA-style disruption studies. Adapted from medical education assessment, SCT presents ambiguous clinical scenarios requiring probabilistic reasoning under uncertainty rather than retrieval of established facts. A 2025 study in NEJM AI validated SCT as a benchmark for comparing LLM clinical reasoning against reference panels of practicing physicians (McCoy et al., NEJM AI, 2025).

Why SCT adds a different test: Standard multiple-choice benchmarks emphasize well-defined problems. Clinical practice often requires reasoning under uncertainty, with incomplete information and competing hypotheses. SCT compares responses with the distribution of judgments from an expert panel rather than a single fixed answer key.

Practical implication: For clinical decision support, SCT and answer-format stress tests can complement standard benchmark accuracy. They still do not replace evaluation in the intended workflow.

Human Baselines and Real EHR Second Opinions

Brodeur et al. show what stronger LLM evaluation can look like: generated differential diagnoses, diagnostic reasoning documentation, management reasoning, probabilistic reasoning, physician-adjudicated scoring, human baselines, and a blinded emergency department second-opinion study using unstructured EHR data from 79 randomly selected patients (Brodeur et al., 2026). This is stronger than MedQA-style multiple choice testing because it compares model output against physician performance on tasks closer to clinical reasoning.

The study still does not answer the deployment question. It tested text-based second opinions, not live physician-AI collaboration, triage decisions, disposition, treatment execution, liability, cost-effectiveness, equity, or patient outcomes. A linked Science Perspective framed the next step appropriately: strong text-based reasoning performance should trigger prospective clinical trials and governance work, not autonomous deployment (Hopkins & Cornelisse, 2026).

The Interaction Gap: Benchmarks Do Not Predict Real-World Use

Pattern matching is not the only reason benchmarks overstate clinical utility. A 2026 randomized trial (n=1,298) found that LLMs correctly identified medical conditions in 94.9% of cases when tested alone, but participants using those same LLMs identified relevant conditions in fewer than 34.5% of cases, no better than a control group using internet search. MedQA scores above 80% corresponded to real-world participant scores below 20% for the same clinical topics in several cases. Simulated patient interactions (LLMs role-playing as patients) also failed to predict human performance, producing higher scores and less variability than real users (Bean et al., 2026). Benchmark accuracy, simulated testing, and real-world human interaction measure fundamentally different things.

A JAMA Perspective presses the uncomfortable design question: for some cognitive medical tasks, AI alone may outperform both physician-only care and physician-AI hybrids, which shifts the burden onto workflow, liability, regulation, reimbursement, and training rather than assuming hybrid is the safe default (Emanuel et al., 2026). Perspective: not ward outcome data; keep task-specific prospective evaluation as the gate.

A 2026 LLM-assisted systematic review of clinical LLM studies found the same evidence gap at field scale: 4,609 peer-reviewed clinical-medicine LLM studies were identified from January 2022 through September 2025, but only 19 were prospective randomized trials. Most studies evaluated simulated scenarios or exam-style tasks rather than patient-centered outcomes, reinforcing that publication volume should not be confused with deployment readiness (Chen et al., 2026).

Evaluation on authentic oncology notes provides a more direct view of reasoning failures. Across two retrospective cohorts spanning breast, pancreatic, and prostate cancer, GPT-4 produced reasoning errors in 23.1% of note interpretations; confirmation bias, anchoring bias, and omission errors were most strongly associated with potentially harmful outputs. GPT-5.1 reduced the overall error rate but retained structured failure patterns, and automated LLM evaluators detected error presence without reliably classifying subtypes (Kenaston et al., 2026). This supports clinician-reviewed error taxonomies on representative clinical notes rather than endpoint accuracy alone.

Independent Head-to-Head Testing of Clinical AI Tools

A 2026 Nature Medicine brief communication tested two specialized clinical AI tools, OpenEvidence and UpToDate Expert AI, against three general-purpose frontier LLMs, GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. The evaluation used 500 MedQA questions, 500 HealthBench items, and 100 de-identified real clinical queries reviewed in randomized, blinded fashion by 12 U.S. clinicians, producing 1,800 model-question annotations (Vishwanath et al., 2026).

The central finding is important for procurement: frontier LLMs outperformed OpenEvidence and UpToDate Expert AI across all three evaluations. On MedQA, Gemini scored 97.4%, GPT-5.2 scored 94.2%, Claude scored 90.2%, OpenEvidence scored 89.6%, and UpToDate Expert AI scored 88.4%; on HealthBench, GPT-5.2 scored 88.0 versus 62.6 for OpenEvidence and 61.3 for UpToDate Expert AI (Vishwanath et al., 2026). In the real clinical queries benchmark, frontier LLMs formed the higher-performing tier, while OpenEvidence, UpToDate Expert AI, and Google AI Overview formed a lower tier with no significant differences among those three systems (Vishwanath et al., 2026).

Practical implication: A clinical brand, retrieval layer, or subscription price should not be treated as evidence of clinical superiority. Procurement teams should demand independent head-to-head testing on clinician-relevant tasks, including blinded review of real questions from the intended workflow. The study also remains a benchmark study, not an outcomes trial: it does not show whether any system improves diagnosis, treatment selection, safety events, workflow efficiency, equity, or patient outcomes.

Mass General Brigham’s BRIDGE benchmark evaluates LLMs on 87 real-world clinical-text tasks from 59 sources across nine languages, spanning extraction, triage, diagnosis, prognosis, and billing coding rather than licensing-style exams (Wu et al., 2026). Across 95 models, overall BRIDGE performance was far below typical exam-leaderboard numbers (reported average about 44.8), with large variation by task and specialty, including especially weak billing and coding. Aggregate score across heterogeneous metrics: use BRIDGE to demand task- and document-matched evaluation, not as a single universal clinical IQ. Pair with the MedQA stress tests above: fluent, well-formed outputs can still be wrong for the patient today when the error is temporal or contextual rather than malformed.

### HealthBench: OpenAI’s Health-Specific Evaluation Framework

In May 2025, OpenAI released HealthBench, an open-source benchmark specifically designed to evaluate LLM performance on health-related tasks (OpenAI et al., 2025, preprint).

Development methodology:

HealthBench was created in collaboration with 262 physicians who have practiced in 60 countries, are proficient in 49 languages, and have training in 26 medical specialties. Unlike traditional medical benchmarks based on multiple-choice exam questions, HealthBench uses conversation-specific rubrics that reflect how clinicians actually judge response quality.

Benchmark structure:

  • 5,000 multi-turn conversations between a model and user (patient or healthcare professional)
  • 48,562 unique rubric criteria describing attributes of responses that should be rewarded or penalized
  • Criteria range from specific facts (medication dosages) to behavioral dimensions (asking clarifying questions, appropriate care escalation)
  • Conversations average 2.6 turns, with 58.3% being single-turn (emphasizing complete initial responses)

Evaluation dimensions:

HealthBench stratifies performance across seven themes:

  1. Expertise-tailored communication (adjusting complexity for lay vs. professional audiences)
  2. Response depth (appropriate level of detail)
  3. Emergency referrals (recognizing when to escalate)
  4. Health data tasks (interpreting lab values, vital signs)
  5. Global health (cross-cultural and resource-limited contexts)
  6. Responding under uncertainty (appropriate hedging and caveats)
  7. Context seeking (asking clarifying questions when needed)

Benchmark variations:

  • HealthBench Consensus (3,671 examples): Filtered subset with high physician agreement, useful for studying high-confidence failure cases
  • HealthBench Hard (1,000 examples): Challenge subset identified as especially difficult for frontier models

Model performance (2025):

Model HealthBench Score
o3 60%
GPT-4o 32%
GPT-3.5 Turbo 16%

The OpenAI-authored preprint reported that GPT-4.1 nano outperformed GPT-4o on HealthBench and was 25 times cheaper at the cited API prices; this vendor benchmark and pricing comparison does not establish clinical superiority (Arora et al., 2025, preprint).

Human baseline comparison:

Physicians were asked to produce responses both with and without model assistance. In this OpenAI-authored benchmark and rubric configuration, recent models scored above unassisted physicians, while assisted physicians scored higher than unassisted physicians. These benchmark ratings support a hypothesis about augmentation, not a clinical-outcome claim or a universal ranking of models and physicians (Arora et al., 2025, preprint).

Critical Caveats for HealthBench

Vendor-developed benchmark: HealthBench was created by OpenAI. Models may be optimized specifically for this evaluation, and the benchmark may not capture failure modes that competitors would identify.

Rubric-based evaluation uses LLM grading: Responses are evaluated by GPT-4.1 as the grader, introducing potential bias toward OpenAI model outputs.

Benchmark ≠ clinical validation: High HealthBench scores do not substitute for prospective clinical trials demonstrating patient benefit.

Not device-specific regulatory evidence: HealthBench does not replace the evidence, labeling, quality controls, and submission requirements applicable to a regulated software function.

When vendors cite HealthBench performance, reviewers should ask: (1) What is the product’s performance on independent benchmarks? (2) Has the product undergone prospective clinical evaluation for the claimed use? (3) How does performance generalize to the intended patient population and workflow?

HealthBench Professional: Clinician-Specific Evaluation

OpenAI extended the HealthBench framework with HealthBench Professional (OpenAI, HealthBench Professional, 2026), an evaluation specifically designed for clinician-facing AI tasks. Unlike the general HealthBench (5,000 synthetic health conversations), HealthBench Professional uses real conversations written by physicians testing ChatGPT for Clinicians during its development.

Benchmark structure:

  • 525 physician-authored tasks
  • Three clinician-facing use cases: care consult, writing and documentation, medical research
  • Rubric-graded by GPT-5.4 at low reasoning

Performance results:

System Overall Care Consult Writing/Documentation Medical Research
ChatGPT for Clinicians (GPT-5.4) 59.0 51.0 64.1 67.0
Base GPT-5.4 48.1 N/A 34.6 58.1
Specialty-matched physicians with web access and unlimited time 43.7 42.7 32.1 56.3

The report describes ChatGPT for Clinicians as a GPT-5.4-based system used with clinician-facing product tools. Its scores should be interpreted as results for that system and benchmark configuration, not for base GPT-5.4 or clinical care generally.

Independence Limitation

The benchmark conversations were written by physicians testing ChatGPT for Clinicians during product development. That development dependence limits benchmark independence and may align the task distribution with the product. The results require corroboration on independent data and in the intended workflow.

MedHELM: Multi-Dimensional Evaluation for Medical LLMs

MedHELM, formally named “Holistic Evaluation of Large Language Models for Medical Tasks,” evaluates LLMs across multiple clinical task categories rather than only medical exam questions (Bedi et al., 2026).

Task taxonomy:

MedHELM organizes evaluations into clinical note generation, patient communication, medical research assistance, clinical decision support, and administration and workflow. Performance varied across tasks and models, so an aggregate benchmark score should not be treated as evidence for every clinical use.

Evaluation method:

The published evaluation used an LLM panel as judges and compared those ratings with clinician ratings. Reported agreement was modest (ICC 0.47 for LLM-judge versus clinician ratings and 0.43 between clinicians), which supports scalable screening but not replacement of clinical review (Bedi et al., 2026).

Stanford Health Care has also described using MedHELM for initial model selection in its ChatEHR program (Shah et al., 2026). The Nature Medicine version of record states that benchmark-based evaluations are insufficient for monitoring clinician-driven EHR interactions; temporal confusion and numeric-value errors still appeared after benchmark selection.

Safeguards for LLM-as-a-Judge

An LLM judge may support scalable screening, but it should not be the sole release criterion for a high-stakes clinical use. In a global-health evaluation, the highest-performing LLM judge matched human evaluators on only 4 of 11 criteria (Williams et al., 2026). A clinical LLM evaluation framework used randomized, blinded, task-specific rubric review by practicing physicians and measured reviewer agreement and reproducibility after a washout period (Kocaman et al., 2025). Those controls support local calibration of an LLM judge against clinical review, rather than treating a judge score as a safety determination.

  • Pre-specify the rubric, clinically material error categories, decision threshold, and sample of outputs that will receive independent clinician ratings.
  • Compare judge and clinician ratings blindly where feasible, report agreement overall and by error category, and investigate clinically consequential disagreement.
  • Version-lock the judge model, system instructions, rubric, and threshold. Recalibrate after a material change to any of them or to the evaluated workflow.
  • Do not use a judge to assess outputs from the same exact model without documenting and testing for self-preference; record shared developer or model relationships.
  • Escalate clinically consequential outputs and unresolved judge-clinician disagreement to qualified human review. A judge score does not authorize autonomous clinical action.

A 2026 preprint, CLExEval, stress-tested that caution on rare diagnostic narratives with progressive information masking and 5,600 expert scores: GPT-4o-mini accuracy fell from 95.0% on full cases to 32.5% under the most abstracted masking level, and 68.6% of that model’s diagnostic failures showed a reasoning-to-output mismatch: the correct diagnosis appeared in the thinking trace but not the final answer (Ajmal et al., 2026, preprint). On a human-verified consensus failure set (n = 142), LLM judges still assigned passing scores to clinically incorrect outputs at a Hallucination Approval Rate of 47.9% for GPT-4o-mini and 100% for HuatuoGPT-o1 among validly scored cases, even when given the gold diagnosis (Ajmal et al., 2026, preprint). Exam-style fluency and automated judge scores are not substitutes for expert-grounded clinical validity. This is a depth-oriented rare-case preprint audit of two models, not a prospective outcomes trial or a universal leaderboard.

Using MedHELM in Vendor Evaluation

Request results for the tasks that match the intended use, then reproduce the evaluation on independent, representative data. Task-stratified benchmark performance can identify where additional testing is needed, but it does not authorize deployment or establish clinical benefit.

The framework is open source and available at crfm.stanford.edu/helm/medhelm.

MAST: Task-Specific Evaluation of Medical AI

Goh and colleagues propose a task- and context-specific framework for evaluating progress toward medical AI superintelligence. The framework compares systems with strong clinicians or clinical teams across diagnostic reasoning, management reasoning, multimodal interpretation, safety and harm avoidance, and calibration under uncertainty. It gives particular attention to longitudinal, multiturn, and agentic tasks, and identifies patient-important outcomes as the eventual standard for assessing real-world impact (Goh et al., 2026). The supplementary evidence tables place current results in context: the cited head-to-head studies largely use structured vignettes, simulated consultations, retrospective data, or sandboxed EHR tasks. These studies provide evidence about specific capabilities, while clinical safety, effectiveness, and patient outcomes require additional forms of evaluation.

The literature cited by Goh and colleagues illustrates how different study designs answer different questions. A randomized virtual OSCE tested longitudinal disease management across three visits per scenario (Liévin et al., 2026); MedAgentBench tested multi-step actions in a FHIR-compliant virtual EHR (Jiang et al., 2025); and the population-based MASAI trial evaluated AI-supported mammography using interval cancer rate, sensitivity, and specificity (Gommers et al., 2026). Each provides evidence for a defined task and setting, rather than an interchangeable measure of medical capability.

The Medical AI Superintelligence Test (MAST) puts this framework into practice through an ARISE-run benchmark suite and public leaderboard. Its current methodology combines diagnostic reasoning, management reasoning, safety, radiology, and medical-image performance into composite scores. Agentic EHR tasks are reported separately and are not yet included in those composites (ARISE MAST methodology). The suite includes First Do NOHARM for management safety (Wu et al., 2025, preprint), Script Concordance Testing for reasoning under uncertainty, and virtual-EHR benchmarks for agentic tasks. A public repository provides submission tooling and runnable subsets.

The July 2026 revision of the NOHARM preprint reports 1,100 text-based tasks derived from primary-care-to-specialist electronic consultations and physician-approved variants. Across the tested systems, rubric-scored potential severe errors occurred in 2.9%–24.6% of cases, and omissions accounted for more than 80% of severe errors. In a randomized study of 101 U.S.-licensed generalist physicians, AI assistance improved written management-plan scores compared with conventional resources, but assisted physicians frequently omitted useful AI recommendations and still scored below many AI systems tested alone (Wu et al., 2025, revised 2026 preprint). The endpoint was a rubric score on written consultation plans, not observed patient harm or clinical effectiveness. The reported performance of a computational union of physician and AI recommendations was not an observed collaborative workflow.

MAST remains in preview, and exact scores may change during validation and tuning. Composite weighting necessarily simplifies clinical performance, and human comparators are not available uniformly across tasks. Results are most informative when task-level findings, comparator conditions, judge design, data provenance, and failure modes are considered alongside the chapter’s intended-use evidence framework.

Evidence Gaps in Clinical LLM Evaluation

A systematic review of 519 health care LLM studies published from January 2022 through February 19, 2024 found that 44.5% evaluated medical knowledge through question-answering tasks, while 5.0% used real patient care data. Fairness, bias, or toxicity was evaluated in 15.8% of studies, and calibration or uncertainty in 1.2% (Bedi et al., 2025). These findings describe the studies included in that review; they should not be treated as current prevalence estimates for all clinical AI research.

Evaluation should therefore move beyond exam-style accuracy while remaining clear about what stronger simulations establish. CRAFT-MD uses simulated AI patient agents, automated scoring, and expert review to evaluate conversational diagnostic reasoning across 12 medical specialties (Johri et al., 2025). That design tests a more realistic interaction than a fixed multiple-choice item, but it still does not establish performance in live clinical workflow or effects on patient outcomes.

Clinical implication: Ask whether the evaluation matches the intended clinical task, uses representative data, measures uncertainty and subgroup performance, and tests effects in the intended workflow.

Evaluating Explainability: Beyond the Black Box

Many clinical AI systems are not directly interpretable. Explainable AI (XAI) methods aim to characterize factors associated with model outputs, but a post hoc explanation is not necessarily a faithful account of how a model produced a prediction. For background on XAI techniques, see the AI Basics chapter; this section focuses on evaluating XAI claims in vendor offerings.

Why Explainability Matters for Evaluation

Safe use requires information proportionate to the intended user, context, and risk. Joint guiding principles from the FDA, Health Canada, and the UK Medicines and Healthcare products Regulatory Agency call for transparency that is appropriate to the intended users, risks, and context of use (FDA, 2024). These principles are not binding FDA guidance and do not require the same form of explanation for every system.

Evaluation implications:

  • Trust calibration: Appropriate transparency and validated explanations may help users calibrate reliance, but explanation alone does not prevent automation bias. In a randomized clinical-vignette study, image-based explanations did not significantly improve accuracy when the AI model was systematically biased (Jabbour et al., 2023); see Human-AI Collaboration Evidence
  • Error investigation: Feature attributions and other explanations can generate hypotheses about inappropriate model behavior; confirmation requires analysis of data provenance, timing, and model performance
  • Regulatory assessment: Evaluate transparency against the intended use, users, risks, and applicable regulatory requirements rather than treating explainability as a universal checkbox

Dominant XAI Methods in Clinical AI

A systematic review of 62 studies published from 2018 through 2025 found that SHAP and LIME were the most frequently used XAI methods in clinical decision support, with SHAP prominent for tabular data and saliency-based methods for imaging (Abbas et al., 2025).

SHAP (SHapley Additive exPlanations): Estimates feature contributions to a prediction under the assumptions of the selected explainer and background data. One study used SHAP to examine a mortality-prediction model among hospitalized patients with sepsis (Zhang et al., 2024).

LIME (Local Interpretable Model-agnostic Explanations): Approximates complex models with simple, interpretable models around specific predictions. Produces intuitive visual explanations for image-based AI.

Saliency maps and attention visualization: Highlight which image regions or text segments influenced predictions. However, attention weights may not faithfully reflect model reasoning (Jain & Wallace, 2019).

Saliency maps that emphasize rulers, skin markers, borders, or acquisition artifacts should be treated as failure hypotheses. Confirm the hypothesis through controlled perturbation and independent performance testing rather than assuming the map faithfully explains the model.

The Interpretability-Performance Myth

A common vendor claim: “We sacrifice accuracy for interpretability” or “Black boxes are necessary for complex predictions.” The evidence challenges this:

For structured clinical data (labs, vitals, demographics), interpretable models often match complex black-box models (Rudin, 2019). The choice is not always black box versus interpretable; often it is black box with post-hoc explanations versus inherently interpretable design.

Ask vendors: “Why did you choose a black-box approach? Have you compared performance to interpretable alternatives?”

Evaluating Vendor XAI Claims

Questions to Ask About Explainability
  1. Which explanation method is used? (SHAP, LIME, saliency, inherently interpretable model?)
  2. Are explanations validated? Do they reflect actual model behavior, or are they post-hoc rationalizations?
  3. Can clinicians act on explanations? Does knowing a feature importance score change clinical decision-making?
  4. How stable are explanations? Same patient, same input, same explanation every time?
  5. What are explanation limitations? Honest vendors acknowledge what explanations cannot reveal.

For a structured checklist, the CLIX-M framework provides 14 clinician-informed items for evaluating XAI in clinical decision support (Brankovic et al., NPJ Digital Medicine, 2025). See the Vendor Evaluation appendix for procurement integration.

Red Flags in XAI Evaluation

Explanations highlight implausible features: Patient identifiers, time of day, bed number, or non-clinical metadata suggest spurious correlations rather than clinical reasoning.

Explanations highlight treatment variables: Treatment variables may reflect label leakage, post-outcome information, or a legitimate contemporaneous signal. Investigate feature timing, intended prediction time, and data generation before drawing a conclusion. See Common Validation Pitfalls.

Explanations are unstable: Materially different explanations for the same input should trigger investigation of the explanation method, baseline, stochasticity, and model behavior. Instability alone does not prove that the underlying prediction is unreliable.

Vendor claims “proprietary explainability”: Insufficient disclosure of the method, inputs, assumptions, or validation prevents independent assessment.

Evaluation Conclusion

Explanations are evaluation tools, not proof of model reasoning. Require risk-appropriate transparency and validate any explanation method for the intended task, user, and decision context.

Post Hoc Interpretability Audit of a Reinforcement-Learning Policy

A retrospective study of one reinforcement-learning policy for Medicaid care coordination combined attention analysis, SHAP attribution, and sparse autoencoders to examine divergent recommendations (Basu et al., 2026). Investigators adjudicated 200 divergent cases and grouped them into premise, calibration, and contextual categories. Those categories describe this study and should not be presented as a general taxonomy of clinical AI failure.

The study also reported retrospective estimates from a conformal safety layer. Such estimates depend on the evaluated data, policy, outcome definition, and exchangeability assumptions; they do not establish prospective safety or transportability to another health system. Agreement across attention, SHAP, and sparse-autoencoder analyses can strengthen a hypothesis about model behavior, but it does not prove that a post hoc explanation faithfully represents the model’s internal computation.

Evaluation implication: Use interpretability analyses to identify testable failure hypotheses. Confirm those hypotheses with temporal data review, subgroup and error analyses, prospective monitoring, and evaluation in the intended setting.

Algorithmic Auditing: A Practical Guide

Clinical teams should not rely solely on vendor-reported performance metrics. Algorithmic auditing provides systematic methods for detecting hidden failures, bias, and performance disparities before deployment.

The Medical Algorithmic Audit Framework

The Lancet Digital Health framework (Liu et al., 2022) recommends three core audit components:

  1. Exploratory error analysis: Identify patterns in model failures (are errors random or systematic?)
  2. Subgroup testing: Evaluate performance across clinically relevant patient groups
  3. Adversarial testing: Probe model behavior under edge cases and unusual inputs

Identifying Underperforming Subgroups

Average performance may mask significant disparities. The AFISP framework (Algorithmic Framework for Identifying Subgroups with Potential Performance Disparities) provides a data-driven approach to automatically detect subgroups where a model underperforms (Subbaswamy et al., 2024).

Why this matters:

  • Performance drops often cluster in intersectional groups (for example, older Black women with diabetes) rather than single demographic categories
  • Vendor-reported “subgroup analysis” may only test obvious categories (age, sex, race) while missing clinically relevant phenotypes

Practical subgroup analysis steps:

Step Action Tool/Method
1. Define subgroups Clinical relevance: age, comorbidities, disease severity, demographics Domain expertise
2. Calculate metrics per subgroup Sensitivity, specificity, PPV, NPV Standard statistics
3. Evaluate differences Compare with prespecified clinically meaningful performance floors using confidence intervals and adequate subgroup sample sizes Statistical analysis
4. Intersectional analysis Test combinations (age, sex, race together) Aequitas, Fairlearn
5. Root cause investigation Why does model underperform here? XAI methods, data review

Open-Source Auditing Toolkits

Physicians and informatics teams can use established toolkits:

  • Aequitas (University of Chicago): Audits ML models for discrimination across multiple fairness metrics; generates bias reports
  • Fairlearn (Microsoft): Python package for fairness assessment and bias mitigation; integrates with scikit-learn workflows

Bias Detection Through Comparison Testing

A five-step audit framework developed for LLMs in specific clinical applications illustrates the elements needed for a defensible bias evaluation; it should not be applied unchanged to every clinical AI system (Templin et al., 2025).

The comparison protocol:

  1. Engage affected stakeholders to define the audit purpose, questions, outcomes, methods, and risk tolerance
  2. Select the LLM and evaluation design for the intended population, expected effect size, calibration needs, and statistical power
  3. Test clinically relevant scenarios; use matched inputs only when the counterfactual comparison remains clinically coherent
  4. Interpret effect sizes and uncertainty against non-AI-assisted clinical decisions, accounting for repeated sampling or multiple comparisons where relevant
  5. Monitor the deployed system for drift and recurrence of identified disparities
Five Questions for Every AI Audit
  1. What is the worst-performing subgroup? Demand vendor disclosure of performance floor, not just average
  2. How was subgroup performance tested? Single demographics or intersectional combinations?
  3. What fairness metrics were applied? Equal accuracy, demographic parity, equal opportunity?
  4. Is there performance monitoring for drift? Thresholds and review frequency should be predefined from clinical risk and statistical uncertainty
  5. Can the institution conduct an independent audit? Access to model outputs on local data enables institutional validation

Statistical Rigor in Benchmark and Audit Comparisons

Benchmark tables throughout this chapter report accuracy as point estimates. A 2024 statistical framework for language model evaluations argues that a bare percentage hides the question a reader actually needs answered: is a reported gap a genuine capability difference, or could it plausibly be sampling noise from the specific question set used (Miller, 2024, preprint)? The framework treats evaluation questions as a sample drawn from a larger, unseen pool of possible questions, not as a fixed, complete test, and makes four practical recommendations that extend the audit protocol above.

Report a standard error alongside the point estimate. For a simple accuracy score from n independent questions, the standard error is approximately the square root of p(1-p)/n, where p is the observed accuracy. Applied to the published NOTA results (n = 68 questions), GPT-4o’s reported 85.3 percent original accuracy gives an illustrative standard error of roughly 4 percentage points, and its 48.5 percent NOTA-modified accuracy gives roughly 6 points (Bedi et al., 2025). A 68-question benchmark score reported without an error bar is missing several points of uncertainty in either direction. This is a recomputation from published aggregate percentages, not a reanalysis of the underlying study, and it assumes the 68 questions behave independently. If questions share a vignette or answer pattern, the uncertainty calculation should account for clustering.

Cluster the standard error when questions are correlated. Many benchmarks group several questions under the same passage, case, or vignette. Treating grouped questions as independent can understate uncertainty when within-cluster outcomes are positively correlated. On some public benchmarks, naive standard errors were roughly one-third of clustered standard errors (Miller, 2024, preprint). Before citing a benchmark table in a procurement decision, ask whether the question set clusters by case and whether the uncertainty calculation accounts for it.

Use paired analysis when the same questions are reused. The NOTA table above is a paired design: each model answered the identical 68 questions twice, once in the original format and once with the correct answer replaced. A model’s accuracy drop is the difference between two measurements on the same items, not two independent samples. Matched-pairs analysis, which uses the correlation between a model’s original and NOTA-modified performance on the same question, is substantially more statistically powerful than treating the “before” and “after” scores as unrelated samples, which is the standard rationale for matched-pairs tests such as McNemar’s test in this kind of same-item, before-and-after design. The same reasoning applies whenever two models are compared on one shared question set, as in the HealthBench and Vishwanath et al. head-to-head results above: a paired comparison of the same questions is the statistically stronger analysis, not two separately reported percentages.

Size the evaluation to the gap that matters. The number of questions depends on how small a capability gap the evaluation must reliably detect. In one worked example, resolving a 3-percentage-point gap with conventional error rates required roughly 1,000 questions (Miller, 2024, preprint). The 68-question NOTA study detected large accuracy drops, but its published aggregate results do not establish power for an unspecified smaller difference (Bedi et al., 2025). Institutions conducting local audits should size the question set to the smallest performance gap that must be detected, not to the sample that is most convenient to assemble.

Post-Deployment Surveillance

Algorithm performance can change as data, populations, workflows, and care patterns change. The STANDING Together recommendations primarily address transparency about health data diversity, representativeness, and subgroup evaluation; that information helps institutions judge applicability and plan monitoring (Alderman et al., 2025). Post-deployment controls should follow a risk-based monitoring plan such as the one described earlier in this chapter.

Monitoring checklist:

  • Performance and error metrics by clinically relevant subgroup
  • Distribution-shift tests selected for the data type and deployment context
  • Predefined investigation thresholds that account for uncertainty and clinical consequence
  • Clear escalation, pause, recalibration, and retirement pathways
  • Periodic independent review at an interval proportionate to risk

Operational example: A 2026 Mayo Clinic review describes 17 internally developed radiology AI algorithms in live clinical use, with monitoring at different cadences for utilization, failure modes, data drift, and end-user feedback through automated alerts and dashboards (Cook et al., Mayo Clinic Proceedings: Digital Health, 2026). This illustrates one institutional implementation, not a universal monitoring schedule. Each organization should define cadence, thresholds, ownership, and response actions from the system’s risk and local workflow.

When to Reject an Algorithm

Auditing may reveal that an algorithm should not be deployed. Red lines include:

  • Performance below a predefined safety or utility threshold in a clinically relevant subgroup
  • Evidence of harm in validation data (missed diagnoses concentrated in vulnerable populations)
  • Vendor refusal to provide subgroup performance data
  • No mechanism for local validation or ongoing monitoring
  • Validated evaluation demonstrates clinically inappropriate feature reliance (see XAI Red Flags)

The audit is not a checkbox. It is a gatekeeping function that should prevent harmful deployments. A rigorous audit that leads to rejection is more valuable than a superficial audit that enables deployment of a flawed tool.

Questions About Clinical AI Evaluation

What is AUC and how do I interpret it for medical AI?

AUC (area under the receiver operating characteristic curve) measures discrimination across thresholds. It does not establish calibration, threshold-specific clinical value, or patient benefit. Interpretation must consider the intended use, comparator, prevalence, and consequences of false-positive and false-negative results.

What is external validation and why does it matter for clinical AI?

External validation evaluates a locked model on data independent of model development. Temporal, geographic, and multi-site evaluations can test different aspects of transportability, but applicability still depends on the intended population, setting, measurements, and operating threshold.

Does FDA clearance guarantee an AI tool works clinically?

No. Regulatory status depends on the software function and intended use. A 510(k) clearance is based on substantial equivalence to a legally marketed predicate. Authorization does not by itself establish local transportability, workflow fit, or improved patient outcomes.

How do I know if AI training data was representative of my patients?

Compare the development and evaluation data with the intended population, setting, devices, data systems, prevalence, and clinical pathway. Review subgroup performance with uncertainty estimates. Material differences raise an applicability concern that should be tested rather than assumed away.

Evidence Appraisal Exercises

These exercises use published studies rather than fictional patient cases. For each study, distinguish the claim that the design can support from questions that remain unresolved.

Exercise 1: External Performance Is Not Clinical Utility

Wong and colleagues evaluated the Epic Sepsis Model in 38,455 hospitalizations among 27,697 patients at Michigan Medicine. The hospitalization-level AUROC was 0.63; at a score threshold of 6, sensitivity was 33% and PPV was 12% (Wong et al., 2021).

Appraisal questions:

  1. Which claims does this external retrospective evaluation support?
  2. How do the selected threshold, outcome definition, and single-center setting affect interpretation?
  3. What prospective evidence would be needed to estimate alert burden, clinician response, and patient benefit?
  4. What local evidence would be required before using the model in a different health system?

Interpretation: The study directly tests performance in one external setting. It does not establish that showing alerts to clinicians improves care, that the same threshold is appropriate elsewhere, or that the findings apply to later model versions.

Exercise 2: Adoption Does Not Establish Benefit

Lehman and colleagues analyzed 625,625 digital screening mammograms interpreted by 271 radiologists at 66 facilities. Use of computer-aided detection was not associated with improved sensitivity, specificity, or cancer detection (Lehman et al., 2015).

Appraisal questions:

  1. What clinical-impact claim does this observational design address?
  2. Which confounding, workflow, or selection factors could affect the comparison?
  3. Which downstream outcomes and harms should be measured in addition to diagnostic accuracy?
  4. Why should this study not be generalized to every contemporary radiology AI system?

Interpretation: Large-scale use and regulatory authorization do not substitute for evidence of benefit. The study is informative about first-generation mammography CAD in the evaluated period, not a universal estimate for modern imaging AI.

Exercise 3: A Positive Trial Still Requires Applicability Review

GOLDEN BRIDGE II was a cluster-randomized trial of an AI-based clinical decision support intervention involving 21,603 patients with acute ischemic stroke at 77 hospitals in China. New vascular events at 3 months occurred in 2.9% of the intervention group and 3.9% of the control group (adjusted hazard ratio, 0.74) (Zhang et al., 2026).

Appraisal questions:

  1. Was the effect attributable to the prediction model alone or to the complete decision-support and implementation strategy?
  2. Were allocation, analysis, missing data, and outcome ascertainment handled in ways that minimize risk of bias?
  3. How similar are the intended patients, hospitals, workflows, and usual care to the trial setting?
  4. What monitoring would be needed after adoption to confirm benefits and detect unintended effects?

Interpretation: A comparative trial can support a causal claim about the evaluated intervention in its study context. It does not validate unrelated systems or remove the need to assess transportability, implementation fidelity, harms, and post-deployment performance.

Conclusion

Clinical AI should be evaluated as an intervention embedded in a clinical system, not as an isolated accuracy score. Define the intended use and claim, select evidence that answers that claim, appraise risk of bias and applicability, and require monitoring proportionate to the consequences of error.

For protocol design, trial matching, recruitment endpoints, trial-outcome prediction, and trials of AI interventions, continue to Clinical Trials and AI.