Clinical AI Safety and Risk Management
Clinical AI safety is a property of the complete sociotechnical system, not the model alone. A model can retain discrimination performance while an interface, routing rule, data feed, threshold, staffing pattern, or software update creates a new hazard. The safety unit therefore includes the algorithm, data pipeline, interface, users, escalation pathway, monitoring, downtime process, and vendor change controls.
After completing this chapter, clinicians should be able to:
- Distinguish model performance from clinical safety and effectiveness
- Identify data, model, integration, human-factors, and organizational failure modes
- Apply risk-based failure analysis without treating illustrative scores as measured probabilities
- Interpret current FDA AI-device guidance and reporting duties accurately
- Design monitoring, escalation, stop, and decommissioning controls
- Evaluate automation bias, alert burden, cognitive anchoring, and loss of downtime capability
Introduction
In 2021, a retrospective external validation of the Epic Sepsis Model at one academic center reported 33% sensitivity and 12% positive predictive value at a threshold of 6 (Wong et al., 2021). The cohort study included 27,697 hospitalized patients and quantified the model’s predictions against sepsis outcomes. It did not activate the alert as an intervention, observe clinicians responding to it, or test whether its use changed care or patient outcomes.
The Epic sepsis study illustrates a fundamental truth about medical AI safety: retrospective performance in one setting does not establish transportability, workflow benefit, or patient safety in another. AI-enabled and conventional medical devices can both fail through hardware, software, user-interface, maintenance, environmental, or organizational pathways. AI adds particular dependence on data provenance, threshold selection, model version, and changing clinical context.
The Risk of Technological Solutionism
WHO guidance on large multimodal models names a pattern physicians should recognize: technological solutionism, the tendency to overestimate AI benefits while ignoring or downplaying challenges in safety, efficacy, and utility (WHO, 2024).
The Epic sepsis example shows why an internally reported metric cannot substitute for independent evaluation. The external study found materially different threshold performance at its site. That discrepancy supports local validation and prospective workflow evaluation, but the study does not establish that the model itself caused patient harm.
Technological solutionism manifests as:
- Uncritical adoption: Deploying AI without prospective validation because “AI is the future”
- Overlooking failure modes: Assuming vendor performance claims generalize to the deployment population
- Ignoring workflow integration: Treating AI as a technology problem rather than a human factors problem
- Dismissing skepticism: Labeling concerns about AI safety as “resistance to innovation”
The antidote is not rejecting AI, but demanding evidence proportionate to the claim and risk. A health system should know which evidence supports the exact product, version, intended use, population, workflow, comparator, and endpoint before relying on it.
AI Safety Within the Patient Safety Tradition
AI safety is not a new problem requiring new frameworks. It is the latest chapter in a patient safety tradition crystallized by the Institute of Medicine’s landmark 1999 report, To Err is Human: Building a Safer Health System (Kohn et al., 2000).
That report made the historically influential estimate that 44,000–98,000 deaths occurred annually in U.S. hospitals because of medical errors, using studies and extrapolations available at the time. The number should not be read as a current annual estimate. Its enduring contribution was the systems insight: errors often reflect interacting process, design, communication, and organizational failures rather than a single careless person. The safety response is to redesign the system so errors are less likely, more detectable, and less harmful when they occur.
This framing applies directly to AI safety:
AI safety events require systems analysis. A clinician’s action remains relevant, but review should also examine output presentation, uncertainty, training, workload, escalation, policies, data, and workflow. Assigning every event either to the model or to the user misses interactions that created the hazard.
Blame-free reporting enables learning. The IOM advocated for non-punitive error reporting so organizations could learn from failures. AI safety requires the same culture: physicians must feel safe reporting when AI recommendations were wrong or when they overrode AI incorrectly.
Design for safety, not just performance. The IOM emphasized building safety into systems through redundancy, forcing functions, and fail-safes. AI systems need similar design: not just high accuracy, but graceful degradation, uncertainty quantification, and human oversight at critical decision points.
The physicians who successfully navigate clinical AI will be those who understand that AI safety is patient safety, requiring the same systems thinking, just culture, and continuous improvement that the IOM articulated 25 years ago.
Why AI Safety is Different
Traditional Medical Device Safety
Medical devices have well-established safety paradigms: - Established failure analysis: Pacemakers have battery depletion modes, monitors have sensor failures, and software-controlled devices can have complex failure pathways - Testable before deployment: Devices can be bench-tested, stress-tested, validated in controlled conditions - Controlled configuration: Manufacturers and health systems manage versions, maintenance, calibration, cybersecurity, and labeling throughout the lifecycle - Detection controls: Some failures are obvious, while latent software, calibration, use, or maintenance failures may not be
AI System Safety Challenges
Medical AI introduces fundamentally different risks:
1. Silent Failures: - AI can produce plausible-looking but incorrect outputs - Errors may not be immediately apparent to clinicians - Example: AI misses subtle fracture on X-ray, radiologist trusts AI and also misses it
2. Context-Dependent Performance: - AI performs differently across populations, hospitals, workflows - What works at academic center may fail at community hospital - Performance varies with disease prevalence, patient demographics, image acquisition protocols
3. Performance Drift Over Time: - Clinical practice evolves (new treatments, changing patient populations) - AI trained on historical data becomes outdated - Performance degrades silently unless monitored (Finlayson et al., 2021)
4. Unpredictable Edge Cases: - AI may fail severely on inputs unlike training data - Impossible to test all possible scenarios - Example: a chest-radiograph classifier can rely on acquisition artifacts that do not transport across datasets - In one benchmark study, changing familiar answer patterns reduced model accuracy by 26–38 percentage points for the evaluated tasks and models; this does not establish a universal LLM failure rate (Bedi et al., 2025)
5. Inscrutability: - Some model outputs are difficult to interpret or audit at the feature and reasoning level - Root-cause analysis must therefore inspect data, configuration, model behavior, interface, and workflow rather than rely on a generated explanation - An explanation can affect trust without proving that the model used valid clinical reasoning
6. Cascading Failures: - AI errors propagate through clinical workflows - Wrong output can influence a clinical decision and contribute to harm when safeguards fail - Multiple systems may compound errors
These differences demand new approaches to safety assessment and monitoring (Kelly et al., 2019).
A 2026 Nature review maps LLM safety and security onto five development stages (design, data, model, inference, and environment) and names prompt injection, training-data poisoning, sycophancy, and informal shadow use of consumer models inside hospitals as hazards the six challenges above do not isolate (Clusmann et al., 2026). The authors assign mitigations to specific stakeholders, including local clinical-AI oversight and centralized AI Security Operations Centers. The paper is a literature synthesis, not an outcomes trial, and several authors report industry consulting relationships.
Sheng et al. argue that current medical-AI frameworks miss patient harm that is delayed, hard to attribute, and irreversible, and they propose four clinician-facing safety tiers for the deployment, not the model: S1 narrow and immediately testable, S2 decision support with clinician review, S3 harm that appears only after follow-up, and S4 autonomous high-risk action that should be restricted (Sheng et al., 2026). The same diabetic-retinopathy tool is S2 when a clinician reviews the read, S3 when safety depends on whether the patient reaches ophthalmology, and approaches S4 if the model is used to skip judgment. That is a Perspective classification of workflow risk, not an FDA device class and not a trial of delayed harm.
The “Jagged Frontier”: Superhuman and Brittle
The State of Clinical AI Report 2026 identifies a critical safety concept: AI systems exist on a “jagged frontier” where models demonstrate superhuman capabilities on some controlled tasks while remaining brittle when confronted with uncertainty.
The pattern:
- Several 2025 studies showed LLMs matching or outperforming physicians on diagnostic reasoning and treatment planning when tested on fixed clinical cases
- Some described this performance as “superhuman”
The brittleness:
- On tests designed to measure reasoning under uncertainty, AI systems performed closer to medical students than experienced physicians (McCoy et al., NEJM AI, 2025)
- Models tended to commit strongly to answers even when clinical ambiguity was high
- When models had to ask follow-up questions, manage incomplete information, or revise decisions as new details emerged, performance dropped significantly (Johri et al., Nature Medicine, 2025)
- In the evaluated tasks and models, accuracy dropped 26–38 percentage points when familiar answer patterns were disrupted (Bedi et al., 2025)
Why this matters for safety:
The gap between benchmark confidence and real-world uncertainty creates a dangerous pattern: AI systems may appear highly capable while lacking the appropriate humility about their limitations. A model that “commits strongly to an answer even when ambiguity was high” is precisely the type of system that will miss atypical presentations or fail to escalate appropriately.
Clinical implication: Demand evidence that AI systems can recognize and communicate uncertainty, not just achieve high accuracy on unambiguous cases.
Deferral as a Safety Mechanism
Perfect accuracy is unachievable in clinical AI, but fortunately, it is also not required. Safety emerges not from flawless performance but from knowing when not to act. A well-designed AI system should recognize its limitations and explicitly defer to human judgment when confidence is low or the clinical scenario falls outside its validated scope (Azad et al., Nature Medicine, 2026).
The deferral principle:
Each clinical AI task requires a prespecified benefit-harm framework. No single maximum error rate establishes net clinical benefit. The acceptable operating point depends on the clinical consequence, comparator, prevalence, action triggered, available safeguards, and uncertainty. Critical metrics should align with real-world consequences:
- Under-triage rates for emergency systems that might misclassify urgent cases as low-acuity
- Omission rates for documentation tools that could miss medications and allergies
- Unsafe non-deferral rates for patient messaging systems that provide confident responses when escalation was warranted
The goal is not perfection, but appropriate caution paired with measurable benefit. For illustration, a system that defers on a prespecified subset of out-of-scope or uncertain cases may be safer than one that always answers. The safe deferral rate cannot be selected in the abstract; it must be tested against errors, delays, workload, access, and outcomes in the intended workflow.
Evaluating deferral capability:
When assessing AI systems, ask: Can this system recognize its limitations and appropriately request human intervention? Systems that fail this test lack frontline readiness. Deferral awareness, the ability to know what the system does not know, is a first-class safety requirement, not a limitation to hide.
For selective autonomy gated by decision-time behavioral consistency under on-premise governance, see Zhang et al. (doi:10.1038/s41591-026-04609-x); detail lives on the evaluation chapter.
Contrast: When prediction works
Not all deterioration-prediction research produces the same result. A prospective observational study enrolled 888 non-ICU adult visits and evaluated a continuous wearable-monitoring algorithm against adverse outcomes. There were 135 visits with the composite outcome, while the detailed lead-time analysis for specific adverse events involved 11 events. The study supports prospective performance evaluation of continuous monitoring, not proof that the system improved care, prevented deterioration, or saved costs. Those claims require a comparative intervention study (Scheid et al., 2025).
FDA Regulatory Framework for Medical AI
FDA regulates device software functions under the Federal Food, Drug, and Cosmetic Act and implementing regulations. Whether software is a device, its classification, and its marketing pathway depend on intended use and risk. The term Software as a Medical Device is useful internationally, but it does not replace the U.S. product-specific classification record.
SaMD Classification
The following classes describe the general U.S. device framework. They are not a product catalog, and the examples cannot be assigned to a class without checking the exact FDA record:
Class I: - Generally subject to general controls - Some devices are exempt from premarket notification, but exemption is classification-specific - A clinical function should not be called Class I based only on a broad use description
Class II: - Generally subject to general controls and applicable special controls - Many Class II devices reach market through 510(k) clearance; novel devices can also be classified through De Novo authorization - Substantial equivalence and De Novo classification are regulatory determinations, not proof of outcome benefit in every deployment setting
Class III: - Generally subject to premarket approval unless another statutory route applies - PMA evidence and controls are product-specific - Autonomy does not by itself determine device class or pathway
FDA’s public list of AI-enabled medical devices is informative but not a complete denominator for every marketed AI function. Verify the product name, model or version, intended use, pathway, decision date, classification, and labeling in the primary FDA record.
Current FDA AI-Device Guidance
FDA’s 2021 AI/ML action plan is historical context. Current implementation should use the controlling guidance and the product’s authorization record.
1. Predetermined Change Control Plans (PCCP): - A marketing submission can include a plan for specified future modifications and the protocol used to implement and validate them - Must specify: - Types of changes anticipated (new training data, architecture modifications) - Methodology for updates (retraining protocols, validation procedures) - Impact assessment (when changes require new submission) - Only modifications within the authorized plan are covered; other changes may require a new marketing submission (FDA, 2025)
2. Good Machine Learning Practice (GMLP): - Ten nonbinding guiding principles published by FDA, Health Canada, and the United Kingdom’s MHRA - Covers: - Data quality and representativeness - Feature engineering and selection - Model training and testing - Performance monitoring - Documentation and transparency
3. Modification protocol: - A PCCP describes the method for developing, validating, and implementing each specified modification - Acceptance criteria, impact assessment, labeling, and verification should be traceable to the authorized plan
4. Postmarket controls: - Applicable quality-system, complaint, correction and removal, and Medical Device Reporting duties depend on the manufacturer, device, and event - Performance monitoring should be defined for the specific product and risk; a general AI label does not create one universal monitoring mandate
5. Transparency: - FDA materials emphasize information appropriate to the intended user, such as: - How algorithm works - Training data characteristics - Intended use and limitations - Known failure modes - Explainability is not one universal statutory test and should not be confused with validated clinical usefulness
Implications for Healthcare Organizations: - FDA authorization does not establish clinical benefit in every local workflow - 510(k) clearance is based on a substantial-equivalence determination for the proposed intended use, not a finding that every deployment improves patient outcomes. - PMA, De Novo, and 510(k) pathways apply different statutory standards and product-specific evidence; none should be reduced to a universal evidence hierarchy - Organizations should determine whether additional local verification or evaluation is needed for the intended workflow and risk - Authorization does not end lifecycle oversight. A 2025 cross-sectional analysis identified 182 recall events involving 60 of 950 FDA-authorized AI-enabled devices in its dataset. Recall counts were associated with reported validation categories, but the observational study cannot establish that a validation design caused or prevented a recall (Lee et al., 2025)
FDA 2025 Draft Guidance: Total Product Lifecycle Framework
In January 2025, FDA published draft guidance titled “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations” (docket FDA-2024-D-4488). Because it is draft guidance, its recommendations should not be represented as final binding requirements.
Total Product Lifecycle (TPLC) Approach: - The draft organizes recommendations across the device lifecycle: design and development, marketing submission, and postmarket monitoring - Applies to all AI-enabled device software functions, whether seeking 510(k), De Novo, or PMA authorization - It recommends documentation addressing data representativeness and performance across relevant subgroups - It recommends describing the intended use setting, user interface, and role of the output in the clinical workflow
Key Submission Content Areas: - Device description: Architecture, intended use, user interface design - Risk assessment: Systematic risk analysis across all lifecycle phases - Data management: Data governance, preprocessing, and representativeness documentation - Model development and validation: Training/tuning/test set specifications, performance benchmarks, subgroup analysis - Device performance monitoring: Post-market surveillance plans, drift detection, adverse event reporting - Cybersecurity: Threat modeling for AI-specific attack vectors
Why this matters for procurement and governance: Healthcare organizations can use the draft’s lifecycle domains as procurement questions, while recognizing that a vendor’s marketing packet is not the FDA submission and the draft is not a final requirement. The controlling source should be checked for status changes before relying on it (FDA, January 2025).
Failure Mode and Effects Analysis (FMEA) for AI
FMEA is a structured method for identifying potential failures before they cause harm. Applied to AI, it helps a multidisciplinary team map hazards and controls across the clinical workflow. FMEA scores are local prioritization judgments, not measured probabilities, universal cutoffs, or substitutes for observed safety data.
FMEA Process
Step 1: Map Clinical Workflow - Document end-to-end process where AI will be used - Identify inputs, outputs, decision points, handoffs
Example: AI for Pulmonary Embolism (PE) Detection on CT - Input: CT pulmonary angiography scan - AI processing: Algorithm analyzes images, outputs PE probability - Notification: Alerts radiologist if high probability - Review: Radiologist reviews images and AI output - Reporting: Radiologist issues final report - Action: Clinical team acts on report
Step 2: Identify Potential Failure Modes
For each step, brainstorm what could go wrong:
| Workflow Step | Potential Failure Modes |
|---|---|
| Image acquisition | Poor image quality (motion, contrast timing), incompatible scanner |
| AI processing | Software crash, wrong patient, dataset shift, spurious correlation |
| Notification | Alert does not fire, alert sent to wrong person, alert buried in inbox |
| Radiologist review | Automation bias (misses error), cognitive anchoring (AI biases judgment), alert fatigue (ignores AI), misinterprets AI output, loss of independent skills |
| Reporting | Report unclear, does not reach ordering provider |
| Clinical action | Provider does not see report, misinterprets recommendation, delays treatment |
Step 3: Assess Severity, Likelihood, and Detectability
For each failure mode: - Severity: How bad if it happens? (1=negligible, 10=catastrophic) - Likelihood: How often will it happen? (1=rare, 10=frequent) - Detectability: Will failure be caught before harm? (1=always detected, 10=never detected) - Risk Priority Number (RPN) = Severity × Likelihood × Detectability
Illustrative local example:
The scores below are hypothetical. A real team should define its scale, document the rationale and evidence for each rating, and revisit the ratings after testing and incident review.
| Failure Mode | Severity | Likelihood | Detectability | RPN | Priority |
|---|---|---|---|---|---|
| AI misses PE | 10 | 3 | 7 | 210 | HIGH |
| Cognitive anchoring to incorrect AI | 8 | 7 | 6 | 336 | CRITICAL |
| False positive PE | 4 | 6 | 3 | 72 | MEDIUM |
| Alert not delivered | 9 | 2 | 8 | 144 | HIGH |
| Radiologist ignores alert | 8 | 4 | 6 | 192 | HIGH |
Step 4: Implement Risk Mitigations
For high-priority failure modes, design safeguards:
Cognitive Anchoring to Incorrect AI (illustrative RPN=336): - Mitigation 1: Require radiologist to document preliminary impression before viewing AI output (for complex cases) - Mitigation 2: Training on anchoring bias recognition with case examples - Mitigation 3: Audit AI-concordant errors (cases where both AI and radiologist were wrong) - Mitigation 4: Monitor override rates (low override rate may indicate over-reliance) - Hypothesized effect: the team could provisionally rescore likelihood and detectability after implementing and testing the controls. The example change to an RPN of 128 is illustrative, not a measured effect.
AI Misses PE (RPN=210): - Mitigation 1: Radiologist reviews all cases (not just AI-flagged ones) - Mitigation 2: Quality assurance sampling (re-review AI-negative cases) - Mitigation 3: Performance monitoring (track missed PE rate) - Hypothesized effect: the example rescoring reduces detectability from 7 to 3 and the RPN to 90. Actual rescoring requires local evidence.
Alert Not Delivered (RPN=144): - Mitigation 1: Redundant notification (EHR inbox + page for critical findings) - Mitigation 2: Require acknowledgment within 1 hour - Mitigation 3: Escalation if not acknowledged - Hypothesized effect: the example rescoring reduces likelihood from 2 to 1 and the RPN to 72. Acknowledgment and escalation performance must be tested.
Step 5: Document and Monitor - Document FMEA findings and mitigations - Revisit FMEA periodically (workflows and AI change) - Track actual failures and update risk assessments
AI-Specific FMEA Considerations
1. Data Quality Failures: - Incorrect patient matched to AI input - Missing or corrupted data elements - Data format incompatible with AI expectations
2. Model Performance Failures: - Dataset shift (population differs from training) - Adversarial inputs (deliberately fooling AI) - Edge cases not in training data
3. Integration Failures: - AI output misinterpreted by clinicians - Timing issues (AI result arrives too late) - AI recommendations conflict with other clinical data
4. Human Factors Failures: - Automation bias (over-reliance on AI) - Cognitive anchoring (AI biases clinician judgment) - Alert fatigue (too many false positives) - Loss of clinical skills from AI dependence (de-skilling) - Inability to diagnose independently when AI unavailable or unreliable
5. LLM-Specific Failure Modes (Conversational AI): - Sycophantic validation: This represents a safety failure mode, not merely a stylistic issue. A 2026 Science study across 11 leading AI models found they affirmed users’ actions 49% more often than humans, including in scenarios involving deception, illegality, or other harms, increasing users’ confidence that they were right while reducing willingness to take responsibility or repair conflicts (Cheng et al., 2026). Patient-facing and clinician-facing conversational systems should be tested for inappropriate affirmation, not only factual accuracy. - Delusion reinforcement and crisis escalation: Modeling work suggests that systematic validation can increase confidence in false beliefs under the study’s assumptions (Chandra et al., 2026, preprint). A published case report described new-onset psychosis after intensive chatbot interaction, but a case report cannot establish incidence or causation (Pierre et al., 2025). Company estimates of conversations containing possible indicators are not clinical diagnoses and should not be presented as a population incidence rate (OpenAI, 2025). - Parasocial attachment and emotional dependency: Patients may substitute AI relationships for human therapeutic relationships - ECRI named “Misuse of AI Chatbots in Healthcare” the #1 health technology hazard of 2026, noting that tools such as ChatGPT, Claude, Copilot, Gemini, and Grok produce “human-like and expert-sounding responses” but are not regulated as medical devices and have not been validated for clinical use. ECRI’s annual hazard list is a leading indicator for accreditation scrutiny; the designation signals that governance bodies will increasingly expect healthcare organizations to have formal policies governing patient and clinician use of general-purpose LLMs (ECRI, January 2026). - For a Perspective on patient-facing RAG chatbot deployment leaks (system prompt, knowledge base, and stored conversations retrieved without authentication via ordinary browser tools), see Madrid-García et al. (2026) in NEJM AI (doi:10.1056/AIp2600583). - See Psychiatry and Behavioral Health for clinical implications
Common AI Failure Patterns
Understanding how AI systems fail helps prevent and detect errors.
1. Dataset Shift and Generalization Failure
What It Is: AI trained on one population/setting performs poorly when deployed in different context.
Why It Happens: - Training data not representative of deployment population - Clinical workflows differ between development and deployment sites - Disease prevalence, patient demographics, or comorbidities differ
Examples:
COVID-19 Chest X-ray Classifier Audit (DeGrave et al., 2021): - Researchers retrospectively trained and audited COVID-19 chest-radiograph classifiers across several datasets - Saliency and performance analyses showed that models could exploit source, laterality, acquisition, and other nonpathologic signals - The study demonstrates shortcut-learning risk in curated datasets - It did not evaluate a pre-pandemic pneumonia product deployed during the pandemic, a clinical intervention, or patient outcomes
Pneumonia Detection Dataset Shift (Zech et al., 2018): - Pneumonia detection reached AUC 0.931 on internal test data from the hospitals that supplied the training set - The same model dropped to AUC 0.815 at an external hospital system - Reason: AI learned hospital-specific artifacts (patient positioning, X-ray machine markers) instead of pneumonia
Mitigation: - Train on diverse data from multiple institutions - External validation before deployment - Monitor real-world performance continuously - Retrain when performance drifts
2. Spurious Correlations (Clever Hans Effect)
What It Is: AI learns irrelevant patterns that happen to correlate with outcome in training data but do not reflect true causal relationships.
Why It Happens: - Training data contains confounding variables - AI optimizes for accuracy, not clinical reasoning - Limited data causes AI to latch onto any predictive signal
Examples:
Skin Cancer Detection and Rulers (documented in dermatology AI literature): - Dermatology AI appeared highly accurate - Post-hoc analysis revealed the model partially relied on rulers and color calibration markers present in images - Malignant lesions more likely to be photographed with rulers (clinical documentation practice) - AI learned “ruler = cancer” instead of the visual features of cancer itself (Winkler et al., JAMA Dermatology, 2019)
Hypothetical ICU mortality example: - A model could learn an admission-time association that reflects staffing, referral, transport, or case-mix patterns rather than physiology - A temporal feature can still carry legitimate predictive information, so its use should be examined through transport testing, clinical review, and sensitivity analysis rather than declared spurious automatically
Mitigation: - Careful feature engineering (include only clinically relevant variables) - Interpretability analysis (understand what AI is using) - Adversarial testing (remove expected signals, see if performance drops) - Clinical review of AI features/logic
3. Automation Bias and Over-Reliance
What It Is: Clinicians uncritically accept AI recommendations, even when wrong or when contradicted by other clinical information.
Why It Happens: - Cognitive bias toward trusting automated systems - AI presented as authoritative (“algorithm says…”) - Time pressure and cognitive load - Deskilling from prolonged AI use (loss of independent judgment)
Evidence:
Controlled clinician studies: - In a study involving 140 radiologists across 15 tasks, incorrect AI advice reduced aggregate performance; effects varied by task and clinician (Yu et al., 2024) - In a randomized experiment involving 220 physicians, local feature explanations increased reliance on both correct and incorrect AI advice. An explanation therefore changed trust behavior without guaranteeing a correct recommendation (Prinster et al., 2024) - These controlled tasks demonstrate risk mechanisms, not a universal clinical error rate or evidence that every interface produces the same effect
Cognitive Anchoring as a Critical Safety Failure Mode
Cognitive anchoring represents one of the most insidious AI safety risks: clinicians systematically bias their assessment toward AI recommendations, losing the capacity for independent clinical judgment that serves as the essential safety net for catching AI errors.
The Anchoring Effect:
When clinicians see an AI recommendation before forming their own clinical impression, the AI output can become a cognitive anchor that influences their final judgment. This differs from simple automation bias (accepting AI uncritically) in that clinicians may believe they are exercising independent judgment while their reasoning has been channeled by the AI’s suggestion. A physician account illustrates this concern but does not estimate its frequency or causal effect (Khullar, 2025).
Why Anchoring is Particularly Dangerous:
- Difficult-to-observe influence: A clinician may not recognize how the initial output affected information search or weighting
- Not limited to novices: Experience does not eliminate susceptibility, although effects vary by person, task, and interface
- Undermines error detection: The AI’s primary safety check (physician oversight) becomes compromised
- Potential cumulative effect: Repeated exposure may change search patterns or independent performance, which should be measured longitudinally rather than assumed
Illustrative FMEA Risk Assessment for Cognitive Anchoring:
| Parameter | Rating | Rationale |
|---|---|---|
| Severity | High (8/10) | Missed diagnoses, inappropriate treatments, patient harm |
| Likelihood | High (7/10) | Hypothetical local rating for prioritization |
| Detectability | Medium (6/10) | Difficult to distinguish from appropriate AI-concordant decisions |
| Risk Priority Number | 336 | Illustrative priority, not a measured risk |
Evidence of Anchoring in Clinical AI:
Observed mechanisms in controlled studies: - Incorrect AI advice can reduce clinician performance, and explanations can alter reliance on both correct and incorrect advice (Yu et al., 2024; Prinster et al., 2024) - The effect is not adequately summarized by a single override percentage. Relevant measures include diagnostic accuracy, error concordance, attention, decision time, confidence, escalation, and outcomes - Interface design should be tested empirically because apparently helpful explanation or localization can create misplaced confidence
Mitigation Strategies for Anchoring:
1. Workflow Design: - Independent assessment first: Require clinicians to document preliminary impression before viewing AI output (for high-stakes diagnoses) - Delayed AI display: AI recommendations appear only after clinician forms initial judgment - Blind review sampling: Periodic audits where clinicians interpret cases without AI access - Two-stage review: Independent clinician review, then AI-assisted review, then reconciliation
2. Training and Awareness: - Education on anchoring bias and its mechanisms - Case-based training showing examples of AI-induced anchoring - Regular feedback on cases where clinician anchored to incorrect AI - Competency assessment: Can physicians diagnose accurately without AI?
3. Performance Monitoring: - Audit AI-concordant errors: Review cases where clinician agreed with incorrect AI - Interpret override rates with context: A low or high rate can reflect case mix, model quality, threshold, interface, or workflow. No universal override range distinguishes healthy use, anchoring, or a useless system. - Review disagreements and agreements: Sample appropriate overrides, inappropriate overrides, correct concordance, and AI-concordant errors - Track diagnostic accuracy with vs. without AI: Compare performance when AI available vs. unavailable
4. AI Interface Design: - Present AI as “additional data point,” not “recommendation” - Require clinicians to justify agreement or disagreement with AI - Show confidence intervals/uncertainty (discourage anchoring to low-confidence outputs) - Avoid authoritative framing (“AI diagnosis:”) in favor of neutral language (“AI analysis suggests consideration of:”)
5. Organizational Safeguards: - Regular case conferences reviewing AI-discordant cases (AI wrong, clinician caught it) and AI-concordant errors (AI wrong, clinician missed it) - Celebrate appropriate AI overrides (reinforce that disagreeing with AI is professionally acceptable) - Continuous learning culture: every anchoring-related near-miss triggers workflow review
De-skilling as Long-Term Safety Risk
Repeated reliance on AI diagnostic support may create a secondary safety risk if independent performance or downtime readiness declines. The size, direction, and persistence of that effect are not established across specialties. The appropriate response is to measure competency and system resilience, not to assume that every user will lose skill.
The De-skilling Phenomenon:
What Happens: - Users may change how they search, reason, or allocate attention when assistance is consistently available - Rare independent tasks may receive less practice - Training can become organized around AI prompts rather than an independently generated assessment - Downtime testing can reveal whether a service remains safe when the AI, interface, network, or upstream data feed is unavailable.
Why It Matters for Safety: - AI safety model assumes physician can catch AI errors through independent judgment - If independent performance declines, a safety layer may weaken - Operational dependency can also arise even without cognitive de-skilling, for example when staffing or routing assumes the tool is always available - Particularly concerning for rare diseases or atypical presentations (where AI may be unreliable and physician pattern recognition critical)
Evidence boundary:
Controlled studies demonstrate that incorrect advice and interface explanations can alter clinician decisions. They do not establish that all prolonged clinical use causes permanent de-skilling. Longitudinal studies should separately measure assisted performance, unassisted performance, retention, calibration, and downtime readiness. Until those data exist for the specific task, de-skilling should be treated as a plausible hazard to test rather than a quantified inevitability.
Certification and Competency Questions:
The rise of AI assistance forces difficult questions about physician competency:
- Can physicians diagnose without AI? If not, what happens during system downtime or novel scenarios where AI is unreliable?
- Should board certification exams include AI-free assessments? Ensuring baseline independent diagnostic capability
- How do we maintain skills in AI era? Deliberate practice without AI assistance to preserve independent judgment
- What is minimum acceptable independent performance? Standards for physicians working with AI
Mitigation Strategies for De-skilling:
1. Training Programs: - Independent competency assessment: Training programs can verify baseline performance before relying on assistance for assessed tasks - Purposeful unassisted sampling: Programs can evaluate selected cases without assistance when educationally justified - Competency assessments: Periodic testing of independent diagnostic capability (without AI) - Cross-training: Rotate between AI-assisted and AI-free workflows
2. Workflow Integration: - Independent-first workflow where appropriate: For selected high-stakes tasks, test whether a preliminary impression before AI consultation improves performance or creates delay - Downtime exercises: Rehearse care when the tool or data feed is unavailable - Case variety: Ensure physicians see sufficient case volume and diversity to maintain pattern recognition
3. Institutional Policies: - Competency standards: Define minimum independent diagnostic capability regardless of AI availability - Skills maintenance plans: Continuing education focused on the independent capabilities required for the role - Backup protocols: Procedures for AI system downtime that do not compromise patient safety - Hiring and credentialing: Assess independent diagnostic capability, not just AI-assisted performance
4. System Design: - Degradable AI: Systems designed to provide varying levels of assistance, allowing skills practice - Deliberate assessment: Periodically evaluate performance without assistance under a defined educational or safety protocol - Educational mode: AI provides feedback after independent assessment rather than during
Cross-Reference: See Integration into Clinical Workflow for workflow designs that preserve independent judgment while leveraging AI capabilities. The goal is not to avoid AI assistance, but to design collaboration patterns that maintain rather than erode physician diagnostic expertise.
Mitigation (General): - Present AI as “second opinion,” not ground truth - Require independent clinical assessment before viewing AI output (for high-stakes decisions) - Training on automation bias and anchoring recognition - Audit cases where clinician agreed with incorrect AI - Calibrate trust: highlight when AI is uncertain or in novel scenario - Monitor override rates to detect over-reliance - Ensure physicians maintain diagnostic competency independent of AI
4. Alert Fatigue and Integration Failures
What It Is: AI produces too many alerts (often false positives), causing clinicians to ignore all alerts, including true positives.
Why It Happens: - AI optimized for high sensitivity, accepting low specificity - Poor integration with clinical workflow (alerts at wrong time, wrong place) - No prioritization (all alerts treated equally)
Examples:
Epic Sepsis Model external validation (Wong et al., 2021): - At a threshold of 6, the model had 33% sensitivity and 12% positive predictive value in the study cohort - Those results indicate missed cases and substantial potential alert burden - The retrospective study did not activate the alert or measure clinician desensitization, ignored alerts, treatment changes, or patient outcomes
General EHR Alert Fatigue: - Drug-interaction alert override rates vary widely by system, alert severity, setting, and study design; a broad range should not be transferred to a specific AI alert without direct evidence - An alert should be evaluated as a routed clinical intervention, including who receives it, when it appears, what action it requests, its burden, and what happens after acknowledgment.
Mitigation: - Tune AI threshold based on acceptable false positive rate (not just maximizing sensitivity) - Smart alerting: right information, right person, right time, right format - Tiered alerts: critical vs. informational - Require acknowledgment for critical alerts with escalation - Monitor alert override rates and reasons
5. Performance Drift Over Time
What It Is: AI performance degrades after deployment as clinical practice, patient populations, or data characteristics evolve.
Why It Happens: - Clinical practice changes (new treatments, diagnostic criteria, guidelines) - Patient demographics shift - Changes in data collection or EHR systems - AI becomes outdated but continues to be used
Example: Cardiovascular Risk Prediction (Finlayson et al., 2021): - Risk models trained on historical data - Performance degrades over time as treatment improves (statins, blood pressure management) - Historical risk factors less predictive in modern era
Mitigation: - Continuous performance monitoring - Set performance thresholds triggering retraining - Regular scheduled revalidation (e.g., annually) - Version control and change management - Willingness to decommission outdated AI
Safety Monitoring and Adverse Event Reporting
Ongoing monitoring is essential for catching AI failures before widespread harm.
Real-World Performance Monitoring
What to Monitor:
1. Discrimination Metrics: - Sensitivity, specificity, AUC-ROC - Track overall and by patient subgroups - Set threshold for acceptable performance
2. Calibration: - Do predicted probabilities match observed outcomes? - Example: Of patients AI predicts 30% mortality risk, do ~30% actually die? - Miscalibration can reflect model error, prevalence change, data change, thresholding, or sampling and warrants investigation
3. Alert Metrics: - Alert rate (alerts per day) - Override rate (% of alerts ignored) - False positive and false negative rates - Positive predictive value in clinical practice
4. Clinical Outcomes: - Patient outcomes under AI-supported versus relevant comparator workflows, using a design capable of supporting the intended inference - Time to treatment, missed diagnoses, unnecessary testing - Ideally compare against pre-AI baseline
5. Subgroup Performance: - Performance across race, ethnicity, age, sex, insurance status - Detect disparate impact or bias - Ensure equity
6. User Metrics: - Physician trust and satisfaction - Workflow disruption reports - Time spent reviewing AI outputs
How to Monitor:
Automated Dashboards: - Real-time or daily updates on key metrics - Alert when metrics fall below thresholds - Drill-down capability for root cause analysis
Periodic Audits: - Sample cases for detailed review - Compare AI output to ground truth - Identify systematic errors
Prospective Studies: - Prospective cohort studies can assess workflow, feasibility, and associations under use - Randomized or otherwise credible comparative studies are needed for causal claims about benefit or harm; the appropriate design depends on the intervention and endpoint
Adverse Event Reporting
What Counts as AI Adverse Event: - Incorrect AI output leading to patient harm (delayed diagnosis, wrong treatment) - AI system failure preventing timely care - Alert fatigue causing true positive to be ignored - Workflow disruption from AI integration - Privacy breach from AI system
Reporting Mechanisms:
Internal Reporting: - Easy-to-use reporting system for clinicians - Non-punitive culture (just culture, not blame culture) - Rapid response to reports - Feedback to reporters on outcomes
FDA Reporting (Medical Device Reporting, MDR): - Duties differ for manufacturers, importers, and device user facilities. A physician office is not automatically a device user facility under the MDR definition - Manufacturers generally submit reports within 30 calendar days after becoming aware of reportable deaths, serious injuries, or malfunctions, with specified events requiring 5-day reports - Device user facilities generally report a device-related death to FDA and the manufacturer within 10 workdays. They report a device-related serious injury to the manufacturer, or to FDA when the manufacturer is unknown, within 10 workdays - Institutions should determine the applicable reporter, event category, timeline, and parallel internal duties for the exact event (FDA, Mandatory Reporting Requirements)
Institutional Quality/Safety Reporting: - Incorporate AI into existing safety event reporting - Root cause analysis (RCA) for serious AI-related events - Failure mode analysis to prevent recurrence
Learning from Events: - Share lessons across institutions (de-identified case reports) - National registries for AI adverse events (emerging) - Vendor accountability (require vendors to address identified failures)
Building a Safety Culture for AI
Technology alone does not ensure safety. Organizational culture matters.
Core Principles
1. Physician Oversight is Non-Negotiable: - AI assists, humans decide (especially for high-stakes decisions) - Physicians retain ultimate authority and accountability - Cannot delegate responsibility to algorithms
2. Transparency About Limitations: - Honest communication about what AI can and cannot do - Do not oversell AI capabilities to staff or patients - Acknowledge uncertainty
3. Maintain Downtime Readiness and Appropriate Competency: - Define the independent capabilities required for the task and the service’s downtime plan - Evaluate unassisted performance where loss of assistance could create a material hazard - Use training and competency assessment matched to role, intended use, and risk - Do not assume a specific AI-free schedule prevents de-skilling without measuring the relevant outcome
4. Just Culture: - Encourage error reporting without blame - Focus on system improvement, not individual fault - Psychological safety for raising concerns about AI - Applies the IOM principle: most errors result from faulty systems, not faulty people (Kohn et al., 2000)
5. Continuous Learning: - Every failure is learning opportunity - Regular review of AI performance and incidents - Update protocols based on lessons learned
6. Patient-Centered: - Safety trumps efficiency or cost - Patient welfare always first priority - Equitable AI performance across patient populations
National Safety Framework: Joint Commission and CHAI
In September 2025, The Joint Commission and the Coalition for Health AI (CHAI) released voluntary guidance on responsible AI use in health care. It is an organizational governance framework, not an accreditation requirement, federal regulation, or evidence that a particular AI system is safe. The framework identifies seven areas:
- AI policies and governance structures: Formal policies with multidisciplinary governance teams (clinical programs, IT, compliance, data privacy, operations)
- Patient privacy and data transparency: Policies on data access and use; informing patients how their data is used in AI systems
- Data security and HIPAA compliance: Encryption, access controls, security risk assessments, and incident response planning for AI systems
- Ongoing quality monitoring: Regular testing against known standards; tracking for changes in clinical outcomes
- Risk and bias assessment: Documented processes for categorizing AI bias; verification that tools are tuned to the deployment population
- Education and training: Role-specific training on AI capabilities, limitations, and risks; access controls limiting AI tool use to appropriate staff
- Voluntary blinded safety event reporting: Confidential reporting of AI-related adverse events to independent entities such as Patient Safety Organizations
The guidance is designed to be flexible across institutions at different stages of adoption. It recommends integrating validation, monitoring, policy, education, and reporting into existing quality processes. The Joint Commission described a future voluntary certification pathway; readers should verify its current status before treating the announcement as an available credential (Joint Commission and CHAI, September 2025).
Organizational Safeguards
AI Governance Committee: - Multidisciplinary: clinicians, informatics, quality/safety, ethics, legal - Reviews AI before deployment (safety assessment, FMEA) - Monitors AI performance and adverse events - Authority to pause or decommission AI if safety concerns - Clusmann and colleagues name centralized AI Security Operations Centers as an option for cross-institution incident detection, in addition to local clinical-AI oversight; that is a recommended model, not a Joint Commission requirement (Clusmann et al., 2026)
Training and Education: - Educate clinicians about AI capabilities and limitations - Training on automation bias, anchoring effects, and appropriate AI use - Competency assessment before independent use (including AI-free diagnostic capability) - Skills and downtime exercises matched to the task and local risk assessment - Case-based training showing examples of AI-induced anchoring and appropriate AI overrides
Standard Operating Procedures: - Document clinical protocols for AI use - Escalation procedures for AI failures or uncertain cases - Criteria for overriding AI recommendations
Audit and Feedback: - Regular audits of AI-assisted cases - Feedback to clinicians on performance - Identify and address misuse or over-reliance
Case Studies: Learning from AI Safety Evidence
Case Study 1: Epic Sepsis Model
Background: - Sepsis prediction model widely deployed across U.S. hospitals - Promised early sepsis detection to improve outcomes - Performance claims and thresholds required independent evaluation
What the external validation found (Wong et al., 2021): - External validation study (University of Michigan, 27,697 patients) found: - Sensitivity only 33% (missed 67% of sepsis cases) - Positive predictive value 12% (88% false positives) - Threshold performance differed materially from expectations based on prior reporting
Evidence boundary: - This was a retrospective cohort evaluation at one academic center - It evaluated predictions against outcomes, not an activated alert intervention - It did not measure clinician response, treatment change, alert fatigue, causal harm, or outcome benefit - Dataset and implementation differences are plausible explanations for transportability problems, but the study did not isolate one cause
Lessons: - External validation essential (do not trust vendor claims alone) - Retrospective accuracy does not equal prospective clinical utility - Evaluate the exact version and threshold in the intended population and workflow before relying on it - Monitor real-world performance continuously
What can be concluded: - The study strengthened the case for transparent performance reporting and independent evaluation - It does not support an unsourced claim that most hospitals paused the model, that Epic made a particular change because of this study, or that the evaluated predictions harmed patients
Case Study 2: IBM Watson for Oncology
Background: - IBM marketed Watson as AI for personalized cancer treatment - Promised evidence-based treatment recommendations - Adopted by hospitals worldwide
What the evidence showed: - Investigative reporting described internal examples of unsafe or incorrect recommendations and concerns about the development process (Ross and Swetlitz, 2018) - Peer-reviewed evaluations primarily measured concordance with multidisciplinary recommendations rather than patient outcomes - Concordance varied by cancer, stage, treatment option, and local practice, limiting transferability of any single percentage (Jie et al., 2021)
Examples reported in investigative journalism: - Recommended chemotherapy for patient with severe bleeding (contraindication) - Suggested drugs in combinations not proven safe - Treatment plans contradicting evidence-based guidelines
Evidence boundary: - A reported internal example is a safety signal, not a measured population error rate - Concordance with an expert panel is not the same endpoint as patient benefit, treatment safety, or survival - Product version, local guidelines, patient representation, and workflow affect transferability
Lessons: - Demand evidence matched to the claim, not demonstrations alone - Marketing claims do not equal clinical validation - AI for high-stakes decisions (cancer treatment) requires highest evidence standard - Physician expertise cannot be replaced by insufficiently validated AI
Historical significance: - IBM later divested Watson Health assets. The product history remains a useful example of why commercial prominence and technical ambition do not substitute for transparent, endpoint-matched clinical evidence.
Case Study 3: Chest X-ray AI and COVID-19
Background: - Researchers assembled multiple public and institutional chest-radiograph datasets to train COVID-19 classifiers - High test performance prompted concern that dataset construction rather than pathology could drive predictions
What the retrospective audit found (DeGrave et al., 2021): - Models could learn shortcuts tied to dataset source and acquisition rather than generalizable pathology - Cross-dataset performance and saliency analyses exposed nonmedical signals that could inflate apparent performance - The study audited research models; it did not evaluate a deployed clinical intervention
Documented Issues: - AI detected “pneumonia” based on portable vs. fixed X-ray equipment - Picked up hospital-specific artifacts, text overlays, positioning - Models did not consistently localize pathologic lung regions expected to support a generalizable clinical interpretation
Root Causes: - Training data biases (sicker patients → portable X-rays) - Lack of causal reasoning (correlations mistaken for disease features) - Insufficient stress testing on out-of-distribution cases - Dataset assembly allowed source and outcome to become entangled
Lessons: - AI does not truly “understand” disease, it learns statistical patterns - Training data biases lead to spurious correlations - Test transportability across relevant sites, equipment, acquisition patterns, and patient groups before deployment - Pandemic highlighted need for robust, generalizable AI
Recommendations for Safe AI Implementation
Pre-Deployment
1. Evidence Matched to the Claim: - External validation for transportability and local verification for data and workflow compatibility - Prospective workflow evaluation when the claim concerns use in practice - Comparative evaluation with prespecified endpoints when the claim concerns patient or system benefit - Subgroup analysis relevant to the intended population and plausible harms
2. Failure Mode Analysis: - Conduct FMEA before deployment - Identify high-risk failure modes - Design mitigations and safeguards
3. Human Factors Evaluation: - Test AI in realistic clinical workflow - Assess usability, alert design, integration - Identify automation bias risks
4. Transparent Communication: - Educate clinicians about AI capabilities and limitations - Set realistic expectations - Training on appropriate use
5. Safety Protocols: - Standard operating procedures for AI use - Escalation procedures for failures or uncertain cases - Oversight and accountability structure
During Use
6. Real-World Performance Monitoring: - Continuous tracking of key metrics - Dashboards with automated alerts for performance drops - Regular reporting to governance committee
7. Adverse Event Reporting: - Easy, non-punitive reporting system - Rapid investigation and response - Sharing lessons learned
8. Physician Oversight: - AI recommendations reviewed by qualified clinicians - Physicians retain final decision authority - Cannot delegate responsibility to algorithms
9. Patient Communication: - Inform patients about AI use (tiered consent approach) - Transparency about limitations - Respect patient preferences
Ongoing
10. Regular Safety Audits: - Periodic review of AI performance and incidents - Update risk assessments and mitigations - Assess for performance drift
11. Revalidation: - Scheduled revalidation (e.g., annually) - After major clinical practice changes - When patient population characteristics shift
12. Continuous Improvement: - Learn from failures and near-misses - Update AI, protocols, or training based on lessons - Stay current with evolving best practices
13. Decommissioning: - Willingness to pause or stop AI if safety concerns - Clear criteria for decommissioning - Patient safety > sunk costs
How is AI safety different from traditional medical device safety?
AI-enabled devices share conventional device risks and add failure modes tied to data, model behavior, software integration, human factors, and changing clinical context. Some errors are plausible and difficult to recognize, so safety assessment must cover the complete sociotechnical system rather than assuming either AI or conventional devices fail in one predictable way.
What is automation bias in clinical AI?
Automation bias is inappropriate reliance on an automated output despite contrary or missing evidence. Incorrect AI advice can reduce clinician performance in controlled studies, but no universal override-rate threshold diagnoses automation bias. Evaluation must examine task, output quality, interface, user behavior, errors, and downstream outcomes.
Why did the Epic sepsis model fail?
A retrospective external validation of the Epic Sepsis Model at one academic center found 33% sensitivity and 12% positive predictive value at a threshold of 6. The study quantified performance and potential alert burden; it did not observe a live alert intervention, prove that alerts caused patient harm, or show that clinicians ignored true alerts.
What is cognitive de-skilling from AI use?
Cognitive de-skilling is the possibility that repeated reliance on assistance weakens independent performance or downtime readiness. The magnitude and conditions are not established across clinical specialties. Organizations can assess independent competency, design appropriate training, and test whether assistance changes performance rather than assume that any particular schedule prevents de-skilling.
What safety monitoring is required for clinical AI?
Safety monitoring should match intended use and harm. It can include data quality, threshold-specific performance, calibration, subgroup uncertainty, alert routing, action, burden, outcomes, incidents, downtime, and version changes. Each measure needs an owner, investigation pathway, action threshold, and stop authority.
Conclusion
Medical AI safety is not an afterthought. It’s a fundamental requirement. The promise of AI to improve diagnosis, personalize treatment, and reduce errors can only be realized if AI systems are rigorously validated, thoughtfully integrated, continuously monitored, and honestly communicated (Kelly et al., 2019; Topol, 2019).
The history of medical AI includes both successes (IDx-DR improving diabetic retinopathy screening access) and failures (Epic sepsis model, IBM Watson). The difference lies not in the sophistication of the algorithms, but in the rigor of validation, honesty about limitations, and commitment to ongoing safety monitoring.
Core Safety Principles:
- Retrospective accuracy does not establish real-world safety: match evidence design to the clinical claim
- External validation addresses transportability: it does not by itself establish clinical utility or patient benefit
- Monitor continuously: performance drifts over time
- Report failures transparently: learning requires honesty
- Human responsibility and escalation must be explicit: WHO guidance states that humans should remain in control of health-care systems and medical decisions (WHO, 2024)
- Build a safety culture: just culture, transparency, continuous improvement
- Put patients first: safety trumps efficiency or profit
AI has the potential to improve patient care dramatically. But that potential can only be realized if safety is treated as seriously as innovation. First, do no harm, for algorithms as for all medical interventions.
- Physician AI Liability and Regulatory Compliance: How safety failures translate into legal exposure, malpractice risk, and evolving standard-of-care requirements
- Integration into Clinical Workflow: Practical alert design and implementation patterns that prevent the safety failures described in this chapter
- AI-Assisted Clinical Documentation: Safety considerations specific to ambient AI scribes and AI-generated clinical notes