Medical Ethics, Bias, and Health Equity

A widely used population-health algorithm assigned lower risk scores to Black patients than to equally sick White patients because it predicted health care spending rather than clinical need (Obermeyer et al., 2019). The failure was not simply an unrepresentative dataset. It was a clinically invalid proxy embedded in a resource-allocation workflow.

Learning Objectives

After reading this chapter, clinicians should be able to:

• Apply bioethical principles (autonomy, beneficence, non-maleficence, justice) to medical AI
• Recognize and mitigate algorithmic bias and health disparities
• Navigate informed consent challenges with AI-assisted care
• Assess fairness and equity implications of AI systems
• Understand professional responsibilities when using AI
• Apply ethical frameworks for AI development and deployment

Chapter Summary (TL;DR)

Core Ethical Principles for Medical AI:

1. Autonomy and Informed Consent

• Disclosure and consent obligations depend on the system’s role, impact, research status, law, policy, and what a reasonable patient would consider material (Char et al., 2018)
• Informed consent challenges: complexity, black-box algorithms, unclear risks
• A non-AI pathway, alternatives, and opt-out consequences should be addressed when relevant and available
• Transparency about AI limitations essential for truly informed consent

2. Beneficence and Non-Maleficence

• Patient-benefit claims require patient-relevant evidence; efficiency claims require defined workflow endpoints (Topol, 2019)
• Rigorous validation required before deployment (Nagendran et al., 2020)
• Ongoing monitoring for harms essential (performance drift, unexpected failures)
• First, do no harm. This applies to algorithms as much as treatments

3. Justice and Equity

• Targets, thresholds, access, workflow, and effects should be evaluated for disparate harm (Obermeyer et al., 2019)
• Development and evaluation data should be fit for the intended population, acquisition process, and task (Daneshjou et al., 2022)
• Access to beneficial AI should be equitable, not just for well-resourced institutions
• Algorithmic fairness requires intentional design, not just technical accuracy

4. Algorithmic Bias Has Multiple Pathways

• Documented across radiology, dermatology, risk prediction, and resource allocation
• Often reflects biases in training data and healthcare systems (Gichoya et al., 2022)
• Can amplify, reproduce, reduce, or redirect disparities depending on the target, data, threshold, workflow, access, and intervention
• Requires diverse development teams and inclusive validation studies

Landmark Case Studies:

The Obermeyer Study (Science, 2019): - Commercial algorithm used healthcare costs as proxy for health needs - Black patients systematically under-prioritized despite being sicker - Algorithm affected millions of patients across the U.S. - Demonstrated how seemingly neutral technical choices encode racial bias (Obermeyer et al., 2019)

Dermatology AI Bias: - Skin cancer detection AI trained predominantly on light skin tones - Performance degradation on dark skin (higher false negatives) (Daneshjou et al., 2022) - Risk: Delayed cancer diagnosis in underrepresented populations

IDx-DR Prospective Diagnostic Evaluation: - The autonomous diabetic-retinopathy system was evaluated prospectively in primary care offices among people with diabetes without known diabetic retinopathy - The study assessed diagnostic performance against an image-based reference standard (Abràmoff et al., 2018) - It did not test whether deployment increased screening completion, prevented vision loss, or improved long-term outcomes

AI and Knee-Pain Disparity Measurement: - AI trained on patient-reported pain (not radiologist grades) explained 4.7× more racial disparities in knee pain than standard methods - Showed that the outcome target changes what a model can explain; it did not identify a causal anatomical mechanism or demonstrate reduced disparities in care - Supports patient-reported outcomes as a potentially informative target, not a proven equity intervention (Pierson et al., 2021)

Professional Responsibilities: - Physicians retain professional duties when using AI, while legal responsibility remains fact- and jurisdiction-specific (Char et al., 2018) - An AI output does not independently supply clinical reasoning, accountability, or a universal legal defense - Must understand AI limitations and failure modes before clinical use - Advocacy for patients over algorithmic recommendations when appropriate - Duty to report and address bias when observed

Clinical Bottom Line: Medical AI is not ethically neutral technology. Every AI system embeds values, makes trade-offs, and can perpetuate or mitigate inequities. Physicians have professional obligation to ensure AI serves all patients equitably, respects autonomy, and improves (not just automates) care.

Introduction

Medical AI is often framed as a purely technical challenge: train algorithms on medical data, validate performance, deploy in clinical settings. But this framing obscures profound ethical questions:

• Who benefits from medical AI? Patients at well-resourced academic centers, or also those in rural clinics and safety-net hospitals?
• Who is harmed when AI fails? Disproportionately, underrepresented populations excluded from training data.
• Who decides when AI is “good enough”? Developers optimizing for overall accuracy, or patients for whom the algorithm performs poorly?
• How does AI change the physician-patient relationship? Does it enhance clinical judgment or erode physician autonomy and accountability?

These are not hypothetical concerns. Medical AI has already perpetuated racial bias (Obermeyer et al., 2019), performed inequitably across skin tones (Daneshjou et al., 2022), and raised fundamental questions about informed consent and physician responsibility (Char et al., 2018).

The ethical deployment of medical AI requires more than technical validation. It demands intentional attention to fairness, equity, transparency, and accountability, principles central to medical professionalism but often absent from algorithmic development.

Bioethical Principles Applied to Medical AI

The four principles of biomedical ethics (autonomy, beneficence, non-maleficence, and justice) provide a framework for evaluating medical AI (Char et al., 2018).

Autonomy and Informed Consent

The Principle: Patients should receive information and choices appropriate to the decision and context. Patient disclosure, clinical informed consent, authorization for data use, and research consent are distinct obligations whose legal requirements vary by jurisdiction and use case.

Challenges in Medical AI:

1. Complexity and Opacity: Many modern models do not provide faithful, clinically meaningful explanations of how a particular output was produced. Informed consent does not require a patient to understand model mathematics, but it does require understandable information about the system’s role, material limitations, alternatives, and decision consequences when those facts are relevant.

2. Unclear Risks: Traditional medical interventions have well-characterized risk profiles (e.g., “1% risk of infection”). AI risks are less clear: What is the risk of a false negative? How does the algorithm perform for patients like me (my race, age, comorbidities)?

3. Lack of Alternatives: If AI is integrated into standard workflows (e.g., radiology interpretation, sepsis screening), can patients meaningfully opt out? What alternatives exist?

4. Hidden AI: Many AI systems operate silently in the background (EHR-based risk scores, automated lab flagging). Patients may be unaware AI influenced their care.

Ethical Framework for Informed Consent (Char et al., 2018):

Patient Disclosure for AI Use

Disclosure is more likely to be material when: - AI communicates directly with a patient or materially affects diagnosis, treatment, access, or the medical record - The use is experimental, part of research, or has known limitations relevant to the decision - A reasonable patient would consider the system’s role or an available non-AI alternative important - Law, professional guidance, research requirements, or institutional policy requires notice, consent, authorization, or an opt-out

Separate questions remain for lower-impact uses: - Administrative functions can still affect access, privacy, or discrimination even when they do not make a diagnosis - A clinician’s final review does not automatically eliminate a disclosure obligation - Validation and subgroup performance do not create a universal exemption from transparency

Key principle: Disclosure should improve understanding of a real decision, not become a generic liability ritual.

Practical Approaches:

For High-Stakes AI (e.g., cancer detection, surgical risk prediction): > “In evaluating your imaging, I use an AI system that assists in detecting abnormalities. This system has been validated in large studies but is not perfect. I review all AI findings independently and make final diagnostic and treatment recommendations based on my clinical judgment. Do you have questions about how this system works or how it might affect your care?”

For a Verified Autonomous Screening System: > “This retinal camera uses an autonomous AI system to assess whether the images meet quality requirements and whether the result indicates referral. The exact device, version, intended use, and FDA record have been verified for this setting. The system can still produce false-positive, false-negative, or insufficient-quality results. A referral result is not a complete ophthalmic diagnosis.”

Documentation: When clinically material and consistent with institutional policy, document the exact system and version, the output, how it affected care, relevant limitations, and the responsible clinician’s assessment. Do not document “independent physician review” when the product is autonomous or when no independent review occurred.

Beneficence: AI Must Benefit Patients

The Principle: Medical interventions should improve patient outcomes. “Do good.”

Challenges in Medical AI:

1. Efficiency ≠ Benefit: Many AI systems improve workflow efficiency (faster reads, reduced documentation burden) but do not clearly improve patient outcomes. Efficiency benefits institutions and physicians; do they benefit patients?

2. Surrogate Outcomes: AI often validated on surrogate outcomes (detection accuracy, sensitivity, specificity) rather than clinical outcomes (mortality, morbidity, quality of life). High accuracy does not guarantee patient benefit (Nagendran et al., 2020).

3. Unintended Consequences: AI can have hidden downsides: alert fatigue, deskilling of physicians, over-diagnosis from ultra-sensitive algorithms.

Evidence Standard for Beneficence:

Medical AI should face evidence proportional to its claim and risk. Drug-trial phases are not a direct template for software, but the underlying principle is useful: analytical or retrospective performance cannot substitute for evaluation of the configured intervention in practice.

• Technical and retrospective evaluation: Establishes performance on defined data and endpoints
• External and prospective workflow evaluation: Tests transportability and use under intended conditions
• Comparative clinical evaluation: Tests claimed effects on prespecified workflow or patient outcomes relative to a relevant comparator
• Post-deployment surveillance: Monitors harms, burden, access, version changes, and performance over time

Critical Question: Has this AI been shown to improve patient outcomes in prospective studies, or only to achieve high accuracy in retrospective datasets?

Retrospective performance is insufficient for claiming that use of a system improves patient outcomes (Topol, 2019).

Example: IDx-DR (Diabetic Retinopathy Screening)

What the pivotal study demonstrated: - Prospective diagnostic evaluation in primary care offices - Autonomous point-of-care referral output evaluated against an image-based reference standard - Diagnostic-performance evidence in the enrolled population and setting (Abràmoff et al., 2018) - No randomized comparison of screening completion, treatment, vision loss, or long-term patient outcomes

Example: Epic Sepsis Model

What an external validation demonstrated: - The evaluated Epic Sepsis Model implementation had 33% sensitivity and 12% positive predictive value at the studied threshold (Wong et al., 2021) - The study documented substantial false-alert burden but did not test whether deploying the model caused or prevented alert fatigue - It did not establish a patient-outcome benefit from model use

Non-Maleficence: First, Do No Harm

The Principle: Medical interventions should not harm patients. When harm is unavoidable (e.g., chemotherapy side effects), benefits must outweigh harms.

Harms from Medical AI:

1. Direct Harms: - False negatives: Missed diagnoses (cancer, fractures, strokes) leading to delayed treatment - False positives: Unnecessary testing, procedures, anxiety, overtreatment - Incorrect treatment recommendations: AI suggesting wrong medication, wrong dose, contraindicated therapy

2. Indirect Harms: - Alert fatigue: Too many AI warnings causing physicians to ignore all alerts, including true positives - Deskilling: Over-reliance on AI eroding clinical skills (radiologists losing ability to detect findings without AI) - Delayed care: AI-driven workflows introducing bottlenecks (e.g., waiting for AI report before human review)

3. Equity Harms: - Algorithmic bias: AI performing worse for underrepresented groups, causing disparate harm (Obermeyer et al., 2019; Daneshjou et al., 2022) - Access disparities: Beneficial AI only available to wealthy institutions, widening quality gaps

4. Psychological Harms: - Loss of trust: Patients losing confidence in physicians who defer to algorithms - Dehumanization: Care feeling automated and impersonal

Risk Mitigation Strategies:

Preventing AI Harm in Clinical Practice

Before Deployment: 1. Match evidence to the claim: External validation for transportability, prospective workflow studies for use in practice, and comparative outcome studies for benefit claims (Nagendran et al., 2020) 2. Assess subgroup performance: How precise is performance for relevant populations, thresholds, and acquisition conditions? 3. Understand failure modes: How and why does the AI fail? 4. Design effective oversight: Define who reviews which outputs, when review occurs, and what authority, time, and information make intervention possible

During Use: 5. Monitor real-world performance: Does AI perform as expected in your setting? 6. Track harms: Capture false negatives, alert fatigue, workflow disruptions 7. Maintain fallback capability: Preserve safe care during downtime, abstention, or unreliable output 8. Assign decision authority: Some systems are assistive and others autonomous within a labeled task; responsibility must match the actual workflow

After Adverse Events: 9. Root cause analysis: Was AI contributory? How can recurrence be prevented? 10. Reporting mechanisms: Report AI failures to vendors, FDA (if applicable), and institutional quality/safety teams

Precautionary Principle: When evidence is uncertain, the required assurance should increase with the consequence, irreversibility, exposure, and difficulty of detecting failure. The organization proposing deployment should document why the remaining uncertainty is acceptable and how harm will be detected and contained (Vayena et al., 2018).

Justice and Health Equity

The Principle: Medical resources should be distributed fairly. Benefits and burdens should not fall disproportionately on particular groups.

Justice Challenges in Medical AI:

1. Training Data Bias: Development data can encode institutional, acquisition, and demographic patterns. Gichoya and colleagues showed that deep learning models could predict self-reported race from multiple medical-imaging modalities even though expert readers could not identify the image features supporting that prediction (Gichoya et al., 2022). This finding shows that omitting race from an input list does not make an imaging model race-blind; it does not prove a universal direction of subgroup error.

2. Access Disparities: - Advanced AI often deployed first at wealthy institutions - Rural, safety-net, and under-resourced hospitals lack infrastructure for AI - Widens existing quality gaps between haves and have-nots

3. Algorithmic Amplification of Bias: AI can amplify existing healthcare disparities: - If training data reflects biased medical practice (e.g., Black patients receiving less pain medication), AI learns and perpetuates that bias - If outcome proxies are biased (e.g., healthcare costs correlating with access, not need), AI recommendations will be biased (Obermeyer et al., 2019)

4. Representation in AI Development: - Lack of diversity in AI development teams can lead to blind spots about bias and harm - Lack of diversity in leadership means equity considerations may be deprioritized

Algorithmic Bias: Documented Cases and Lessons

Algorithmic bias in medicine is not theoretical. It’s documented, measurable, and consequential.

Case 1: Racial Bias in Healthcare Resource Allocation

The Obermeyer Study (Science, 2019) (Obermeyer et al., 2019):

Background: - Commercial algorithm used by healthcare systems to allocate care management resources - Predicted which patients would benefit from extra support (care coordination, disease management) - Used healthcare costs as proxy for healthcare needs

The Bias: - At any given risk score, Black patients were significantly sicker than white patients - Algorithm systematically under-predicted risk for Black patients - Result: Black patients needed to be much sicker than white patients to receive same level of care

Why It Happened: - Healthcare costs reflect access to care, not just medical need - Black patients face barriers to accessing care (insurance, transportation, discrimination), leading to lower healthcare spending despite higher illness burden - Algorithm learned that “lower spending = lower need,” perpetuating inequity

Impact: - The evaluated algorithm was used for population-health management at scale - Replacing predicted cost with a measure of health need would have increased the proportion of Black patients selected for additional care from 17.7% to 46.5% in the study population (Obermeyer et al., 2019) - The study supports auditing proxy targets; it does not establish the same disparity for every cost model or institution

Lessons: - Choice of outcome variable matters: “Cost” and “need” are not the same - Historical biases propagate: If training data reflects biased systems, AI learns bias - Disparities can be subtle: Algorithm did not explicitly use race, but outcomes were racially biased - External auditing essential: Bias was discovered by independent researchers, not algorithm developers

Case 2: Dermatology AI Bias Across Skin Tones

External Evaluation (Daneshjou et al., 2022):

Background: - Widely used dermatology datasets and models were evaluated on diverse external images - Representation, disease mix, and image source differed across available datasets

The Bias: - The evaluated models showed performance limitations and disparities across skin tones and datasets - The study does not establish one universal melanoma false-negative rate or the same direction of disparity for every dermatology system - Clinical consequences require evaluation of the intended lesion types, threshold, referral pathway, and population

Why It Happened: - Training data reflect existing disparities: dermatology textbooks and databases predominantly feature light skin - Developers did not intentionally exclude dark skin. They used available data - Performance testing often does not stratify by skin tone, hiding the problem

Lessons: - Representation in training data is critical - Performance must be evaluated across relevant subgroups (not just overall accuracy) - “Color-blind” algorithms are not fair algorithms: ignoring race/ethnicity can perpetuate disparities - Field-specific challenges: Dermatology must actively recruit diverse image datasets

Case 3: Pulse Oximetry Bias

Recent Findings (Sjoding et al., 2020):

Background: - Pulse oximeters measure oxygen saturation non-invasively - Critical for managing COVID-19, sepsis, respiratory failure - Generally considered accurate and unbiased

The Bias: - Pulse oximeters overestimate oxygen saturation in Black patients (compared to arterial blood gas) - Black patients more likely to have hidden hypoxemia (low oxygen despite “normal” pulse ox) - May delay recognition of deterioration and treatment escalation

Why It Happened: - Medical devices calibrated and validated primarily on white participants - Skin pigmentation affects light absorption, but devices not adjusted for this - Decades of use before bias recognized

Lessons: - Bias exists even in established, widely-used technologies - Real-world performance monitoring essential (not just initial validation) - Equity requires intentionality: assuming fairness is insufficient - Simple technologies can have complex bias (not just AI-specific problem)

Regulatory response: The FDA’s January 2025 draft guidance recommends controlled studies with at least 150 diversely pigmented participants, at least 25% in each of three Monk Skin Tone groups, and at least 3,000 paired observations. As of August 15, 2026, FDA still labels the document “Draft, not for implementation”; it has not superseded the 2013 final guidance.

Case 4: AI Reducing Bias in Knee Pain Assessment

The Pierson Study (Nature Medicine, 2021) (Pierson et al., 2021):

The previous cases show how AI can perpetuate or amplify bias. This case demonstrates a different possibility: a patient-reported target can reveal patterns omitted by a conventional radiographic severity score.

Background: - Black patients with osteoarthritis report higher levels of knee pain than white patients with similar radiographic findings - Physicians have historically attributed this to psychological or social factors rather than undetected pathology - Standard radiographic grading (Kellgren-Lawrence) was developed in a 1957 study of a homogeneous English population

The Study: - Researchers trained a deep learning model on knee X-rays to predict patient-reported pain scores directly - Critically, the AI was trained to match patient pain reports, not radiologist grades - This approach starts with what patients experience, then works backward to find the anatomical explanation

Key Findings: - Standard radiologist grading explained only 9% of racial disparities in knee pain - AI predictions explained 43% of racial disparities (4.7× more) - The model captured radiographic information associated with reported pain beyond the standard grade; the analysis did not identify a causal anatomical mechanism - Similar results for disparities by income and education level

Why It Worked: - Training target matters: AI trained to match patient-reported outcomes revealed what radiologists were missing - Standard grading was never designed to capture all sources of knee pain - AI found patterns in X-rays that correlate with pain but are not part of the traditional Kellgren-Lawrence criteria - Black patients’ pain was real and had anatomical correlates; previous methods simply could not detect them

Lessons: - AI can expose limitations in an existing measurement framework, but deployment is needed to test whether it reduces inequity in care - Training on patient outcomes instead of expert labels can reveal blind spots in clinical practice - Historical standards may embed bias: Medical criteria developed in homogeneous populations may not generalize - Patient-reported outcomes have diagnostic value: What patients tell us contains information that objective measures miss

Contrast with Case 1: Case 1 shows a model producing biased allocation when trained on a misaligned proxy (healthcare costs). Case 4 shows that a patient-centered target can explain more of an observed pain disparity. The training target shapes what the model learns, but equitable clinical impact must be measured after the output enters care.

Case 5: Chest X-Ray Underdiagnosis in Under-Served Groups

The Seyyed-Kalantari Study (Nature Medicine, 2021) (Seyyed-Kalantari et al., 2021):

Cases 1 and 4 show how a proxy or a target can hide or reveal inequity. This case documents a different failure: chest X-ray classifiers can systematically label disease as “no finding” more often in under-served groups, an underdiagnosis error that can delay care and worsen unequal treatment if deployed (Seyyed-Kalantari et al., 2021).

Background: - Automatic chest X-ray pathology classifiers are widely studied as candidates for screening and triage - Underdiagnosis, in this study, is the false-positive rate of the no finding label: disease is present, but the model calls the examination healthy - In a triage workflow, patients labeled healthy are deprioritized for clinician review

The Study: - Researchers trained multi-label chest X-ray classifiers on MIMIC-CXR, CheXpert, ChestX-ray14, and a combined multi-source dataset - Models used published deep learning pipelines for chest radiographs, not a single-hospital sample - Underdiagnosis rates were compared across sex, age, race/ethnicity, and insurance type (Medicaid as a socioeconomic proxy), including intersectional subgroups - The evaluation was retrospective. It is not a randomized trial, an FDA clearance, or a study of live triage outcomes

Key Findings: - Classifiers consistently and selectively underdiagnosed under-served groups in several settings, including female, Black, Hispanic, Medicaid-insured, and younger patients (Seyyed-Kalantari et al., 2021) - Intersectional subgroups had higher underdiagnosis rates (for example, Hispanic female patients) - Labels came from clinical records and natural-language labelers and may themselves encode existing care bias; the observed gaps can therefore include bias amplification - The finding is multi-dataset, not a single-hospital artifact; ChestX-ray14, drawn from a research hospital, can differ from the clinical-hospital sets

Lessons: - Underdiagnosis is a selective miss, not generic noise: a higher no finding false-positive rate means disease is called healthy - Audit the no finding operating point by subgroup, not only overall discrimination - Label provenance matters: if reports already under-document disease in some groups, models can amplify that pattern - Unexamined deployment can worsen unequal treatment if underdiagnosed patients are deprioritized

Contrast with Cases 1 and 4: Case 1 is a misaligned proxy (cost for need). Case 4 is a patient-centered target that explained more of an observed pain disparity. Case 5 is a thresholded no finding error that can withhold attention from patients who already have disease.

Mitigating Bias and Promoting Equity

Addressing algorithmic bias requires action across the AI lifecycle: development, validation, deployment, and monitoring.

During AI Development

1. Data Fit for Intended Use: - Evaluate whether development data cover the intended population, disease spectrum, sites, devices, languages, and acquisition conditions - Stratify by race, ethnicity, sex, age, geography, socioeconomic status - Partner with diverse healthcare institutions (not just academic centers)

2. Relevant Development Expertise and Participation: - Include clinicians who care for underserved populations - Involve ethicists, health equity experts, community representatives - Record whose perspectives shaped the target, labels, workflow, and acceptable trade-offs; team composition is not a substitute for empirical evaluation

3. Equity as a Design Goal: - Define fairness explicitly: Equal performance across groups? Equal access? Equal benefit? - Different fairness definitions involve trade-offs. Be transparent about choices (Parikh et al., 2019) - Test for bias proactively (do not assume fairness)

4. Choice of Outcome Variables: - Scrutinize proxies for actual outcomes (cost ≠ need, admissions ≠ severity) - Consider how historical disparities may bias outcome definitions - Validate that proxy measures do not encode existing inequities

STANDING Together: International Consensus for Dataset Transparency

An international initiative involving more than 350 experts from 58 countries established 29 consensus recommendations for tackling algorithmic bias and promoting transparency in health datasets, published simultaneously in Lancet Digital Health and NEJM AI (Alderman et al., Lancet Digital Health, 2025). The framework establishes 18 documentation requirements for dataset creators and 11 use requirements for data users. The core principle: no dataset is free of limitations, and transparent communication of those limitations is professional obligation rather than acknowledged failure.

STANDING Together targets regulatory agencies, funding bodies, ethical review committees, and developers, addressing the upstream source of algorithmic bias at the data curation stage rather than the deployment stage. The framework operationalizes what the chapter principles below describe: mandatory subgroup reporting, documentation of missing data by demographic, and disclosure of known performance disparities before deployment.

Training-set sex balance in chest radiography

Public chest X-ray corpora often skew male. In a large-scale study across three deep neural network architectures and two public chest radiograph datasets with thoracic disease labels, Larrazabal et al. (2020) varied sex imbalance in the training set and found a consistent decline in classifier performance for the underrepresented sex when representation fell below adequate balance. Check sex representation in development data before trusting aggregate computer-aided diagnosis metrics. This estimand differs from Case 5: underdiagnosis there is the false-positive rate of a no-finding label at a deployment threshold among groups who already have disease (Seyyed-Kalantari et al., 2021), whereas training-set sex skew shifts overall thoracic classification performance by sex. Both patterns support prespecified subgroup reporting during validation, not headline accuracy alone.

During Validation

5. Prespecified Subgroup Analysis: - Report performance in clinically and ethically relevant groups with sample sizes and uncertainty - Do not hide subgroup disparities in overall accuracy metrics - Define acceptance criteria from the intended use and consequences of error, not a universal parity threshold (Nagendran et al., 2020)

6. External Validation in Diverse Populations: - Validate in institutions serving underrepresented populations - Test across geographic, socioeconomic, and practice setting diversity - Do not infer transportability from a random split; test distinctions relevant to intended use (Vabalas et al., 2019)

7. Assess Calibration, Not Just Discrimination: - Discrimination (AUC-ROC) measures ability to rank risk - Calibration measures whether predicted probabilities match actual outcomes - Calibration often differs across subgroups even when discrimination appears similar - Miscalibration can lead to disparate treatment

8. Consider Intersectionality: - Bias often compounds across multiple identities (e.g., Black + female, or rural + elderly) - Test performance in intersectional subgroups, not just single demographic categories

Equity Dimensions Beyond Race

Algorithmic bias extends far beyond racial categories. A 2025 systematic review found that “diverse and marginalized populations are often incorrectly or insufficiently represented” in AI health systems, potentially exacerbating inequalities across multiple dimensions (Marko et al., 2025).

Geographic disparities (rural vs. urban):

• Training datasets systematically exclude rural patients due to concentration in academic medical centers
• AI optimized for urban workflows may fail in resource-limited settings
• Connectivity and infrastructure requirements may make AI inaccessible in rural areas
• Clinical thresholds developed in well-resourced settings may not apply to under-resourced contexts

Socioeconomic status:

• The Obermeyer study demonstrated that cost-based proxies disadvantage low-income patients
• Insurance status affects data availability (uninsured patients have sparser records)
• Social determinants of health are poorly captured in EHR training data
• AI deployment costs may create two-tiered care: AI-augmented for wealthy institutions, standard care elsewhere

Language and health literacy:

• NLP models trained primarily on English text perform poorly for non-English speakers
• Medical terminology in training data may not reflect how linguistically diverse patients describe symptoms
• AI-generated patient materials may not account for health literacy levels
• Translation AI can miss cultural context in medical conversations

Disability:

• AI often fails to account for baseline functional status differences
• Mobility, sensory, or cognitive disabilities may not be appropriately represented in training data
• Evaluate whether summaries, recommendations, or interfaces introduce disability-related stereotypes or omit accommodations
• Accessibility of AI interfaces themselves may exclude patients with disabilities

LGBTQ+ populations:

• Sexual orientation and gender identity are largely missing from EHR training data
• Binary or incomplete sex and gender fields can misrepresent transgender, intersex, and nonbinary patients; the clinically relevant variable should be defined rather than inferred from a proxy
• Inference of sensitive attributes from medical data raises privacy concerns
• Health disparities unique to LGBTQ+ populations may not be captured in AI risk models

Age-related bias:

• Training datasets often underrepresent elderly and pediatric populations
• Geriatric patients with atypical presentations may be misclassified
• Polypharmacy and multimorbidity in elderly patients may confound AI predictions
• Adult evidence should not be transferred to pediatric use without age- and task-relevant validation

Implications for evaluation:

Subgroup evaluation should follow the populations, measurements, and access pathways relevant to the intended use, rather than applying one fixed demographic checklist to every task. When relevant subgroup data are unavailable or imprecise, disclose the evidence gap and manage it as a deployment risk.

During Deployment

9. Transparent Communication: - Disclose known limitations and subgroup performance differences - Do not hide uncertainties or deployment risks (Char et al., 2018) - If AI performs worse for certain groups, inform clinicians and patients

10. Human Oversight and Physician Autonomy: - AI should inform, not dictate, decisions - Physicians must have ability to override algorithms when clinical judgment differs - Preserve physician accountability (cannot blame algorithm for bad decisions)

11. Equitable Access: - Ensure beneficial AI available across practice settings (not just wealthy institutions) - Consider cost, infrastructure requirements, and workflow fit - Avoid creating two-tiered healthcare: AI-augmented for the wealthy, standard care for everyone else

During Ongoing Monitoring

12. Continuous Performance Monitoring: - Track real-world performance across demographic subgroups - Monitor for changes over time; discrimination, calibration, burden, or subgroup performance may improve, remain stable, or deteriorate as practice evolves - Establish use-case-specific investigation and stop criteria rather than a universal drift percentage

13. Adverse Event Reporting: - Create mechanisms for clinicians and patients to report suspected AI harms - Investigate patterns suggesting bias (e.g., more false negatives in particular group) - Share lessons learned across institutions

14. Regular Bias Audits: - Independent audits of AI fairness (not just self-reporting by vendors) - Involve community stakeholders and equity experts - Share methods, limitations, and findings with affected users and patients to the extent permitted by privacy, security, contractual, and legal obligations

15. Willingness to Decommission: - If AI is found to perpetuate bias and cannot be corrected, stop using it - Do not continue harmful AI just because it’s already deployed - Patient welfare > sunk costs

Informed Consent in the Age of Medical AI

Traditional informed consent assumes patients can understand their condition, treatment options, risks, and benefits. AI challenges these assumptions.

Challenges to Informed Consent

1. Complexity: Patients do not need to understand every parameter to make an informed decision. They need a clear explanation of the system’s role, material uncertainty and limitations, alternatives, human review, data use, and possible consequences at a level appropriate to the decision.

2. Uncertainty: AI risks are often unknown: “We don’t know how often this algorithm fails in patients like you because it hasn’t been studied in your demographic group.” Can consent be truly informed when key information is missing?

3. Voluntariness: If AI is integrated into standard workflows, can patients decline? If opting out means foregoing evidence-based care, is consent truly voluntary?

4. Hidden AI: Many AI systems operate invisibly: risk scores auto-calculated in EHRs, imaging findings flagged by algorithms, lab values interpreted by AI. Patients often unaware AI influenced their care.

Professional Society Guidance on AI Disclosure

The AMA’s H-480.931 policy uses a risk- and impact-based approach to documentation and disclosure, including disclosure when AI affects access, point-of-care decision-making, patient care, or the medical record. Its 2025 explainability policy calls for clinically useful explanations and independent assessment while stating that explainability cannot substitute for evidence of safety and effectiveness. These are professional policies, not universal statements of law, and disclosure should not be conflated with informed consent or research consent.

Practical Approaches to AI Consent

Institutional discussion framework informed by ethical analysis (Vayena et al., 2018):

The following tiers are a practical discussion aid, not a framework stated by Vayena and colleagues, a universal legal rule, or a substitute for local legal, IRB, and policy review.

Consent and Disclosure Questions for Medical AI

Tier 1: Consider explicit informed consent or research consent

When: - Experimental or investigational AI - AI with known subgroup performance disparities - AI that directly determines treatment (not just advises) - High-stakes decisions (cancer diagnosis, surgical risk prediction)

How: - Document AI use in consent forms - Explain in lay terms what AI does - Disclose known limitations and failure modes - Offer alternatives if available - Allow opt-out

Tier 2: Consider transparent notification when separate consent is not required

When: - AI is well-validated and performs equitably - AI assists physician but does not replace judgment - Physician retains final authority over decisions - Reasonable alternatives exist

How: - Inform patients AI is used - Explain role in clinical workflow - Reassure physician oversight - Answer questions if asked

Tier 3: Consider general disclosure for low-impact background functions

When: - AI provides administrative functions (scheduling, billing) - AI does not influence clinical decisions - AI performs purely background tasks (imaging enhancement, data standardization)

How: - General institutional disclosure (e.g., on website, in patient handbook) - Whether encounter-specific notice is needed depends on the function, impact, law, and policy

Practical Example (Radiology AI):

Poor Consent: > “We use advanced technology to interpret your scans. Sign here.”

Better Consent: > “Your chest X-ray will be analyzed by both a radiologist and an AI system that detects abnormalities. The AI has been trained on thousands of X-rays and helps ensure nothing is missed. However, the radiologist is responsible for the final interpretation and will review all AI findings. The AI works well but isn’t perfect. It sometimes flags normal findings or misses subtle abnormalities. Do you have questions about this process?”

Best Consent (High-Stakes, Experimental): > “We’re offering participation in a study using AI to predict surgical complications. The AI analyzes your medical record and provides a risk estimate. This is experimental. The AI hasn’t been widely validated, and we don’t know how accurate it is for patients with your specific characteristics. If you participate, the AI’s predictions will be shared with you and your surgical team to inform decision-making, but you and your surgeon will make the final decision about surgery. You can decline to participate and still receive standard preoperative evaluation. Would you like to participate?”

Special Considerations for Vulnerable Populations

Pediatrics: - Parental consent + child assent (age-appropriate) - Consider developmental capacity to understand AI - Do not transfer adult evidence to pediatric use without age- and task-relevant validation

Cognitive Impairment: - Surrogate decision-makers may not understand AI - Simplify explanations without oversimplifying risks - Document consent process carefully

Language Barriers: - Provide consent information in patient’s preferred language - Follow applicable language-access requirements and use qualified interpreters for complex discussions; do not assume an AI translation tool is an adequate substitute - Ensure cultural appropriateness of consent process

Low Health Literacy: - Use plain language, avoid jargon - Visual aids (diagrams, infographics) can help - Teach-back method: “Can you tell me in your own words how this AI will be used?”

Professional Responsibilities in the AI Era

Medical AI does not eliminate professional responsibility. It redistributes tasks and creates additional duties for clinicians, organizations, developers, and other actors according to the actual workflow.

Physician as Steward of AI

1. Understand Before Using: - Do not use AI as a black box: “I don’t know how it works, but it’s accurate” - Understand (in general terms): What data does AI use? How was it trained? What are known limitations? - Know how AI was validated and in which populations

2. Maintain Independent Judgment: - AI recommendations are inputs to decision-making, not final decisions - Physician must independently assess patient, formulate differential, consider AI output in clinical context - Avoid automation bias (uncritically accepting AI recommendations)

The Dual Duty Tension

Physicians face an emerging ethical tension: the duty to use beneficial technology versus the duty to maintain independent clinical judgment. As AI tools demonstrate value, failing to use them may harm patients. Yet over-reliance on AI erodes the independent judgment that allows physicians to catch AI errors (Mello & Guha, 2024).

Navigating the tension:

• Neither rejection nor uncritical adoption is ethically defensible. Physicians must thoughtfully integrate AI while maintaining diagnostic skill.
• Structured collaboration patterns (see Integration into Clinical Workflow) help balance efficiency with skill preservation.
• This tension creates legal exposure from both directions. See Physician AI Liability and Regulatory Compliance for detailed analysis of dual liability risk.

The ethical path requires ongoing calibration: adopt AI where evidence supports benefit, verify outputs through independent reasoning, and maintain the foundational skills to practice when AI is unavailable or wrong.

Anthropic reported a large, company-run qualitative survey of AI users across many countries that grouped perceived benefits and harms into tensions involving learning, reliability, dependency, productivity, autonomy, and work (Anthropic, 2025). This is vendor research, not a peer-reviewed clinical study, and it does not establish rates or causal effects in physicians. Its value here is question generation: a clinical implementation should measure whether assistance changes skill, reliance, workload, or professional autonomy rather than assuming the direction of effect.

3. Recognize and Override When Appropriate: - If AI recommendation conflicts with clinical judgment, investigate - Do not defer to algorithm when patient-specific factors (not captured by AI) are relevant - Document reasoning when overriding AI

4. Protect Patient Interests: - Advocate for patients over algorithmic efficiency - If AI-driven workflow harms patient (delays, errors, loss of personalized care), escalate concerns - Professional obligation to prioritize patient welfare over institutional AI investments

Physician as Advocate for Equity

5. Demand Evidence of Fairness: - Ask: “How does this AI perform for my patient population?” - Refuse to use AI with known bias unless no alternative exists and bias is disclosed to patients

6. Monitor for Disparate Impact: - If you suspect AI performs worse for certain patients, document and report - Advocate for inclusive validation and bias mitigation

7. Ensure Equitable Access: - Support policies ensuring beneficial AI available to all patients (not just those at elite institutions) - Treat language access as an AI equity requirement. A 2026 npj Digital Medicine research agenda for AI interpreter services warned that translation accuracy alone is insufficient; evaluation should include patient understanding, trust, performance across languages, and escalation to human interpreters for high-risk encounters (Lynch et al., 2026).

Physician as Learner

8. Stay Current: - Medical AI evolves rapidly. Yesterday’s evidence may be outdated - Engage with medical AI literature (not just vendor claims) - Participate in institutional AI governance and education

9. Teach Others: - Educate trainees, colleagues, and patients about AI capabilities and limitations - Model appropriate AI use (thoughtful integration, not blind adherence)

Ethical Frameworks for Institutional AI Governance

Healthcare institutions deploying AI need structured governance to ensure ethical use (Char et al., 2018).

Gu and Topol argue that health AI should be regulated and evaluated in relation to the degree to which it can influence and shape an individual’s health trajectory, including when it is used outside the clinic (Gu and Topol, 2026). Comment in Nature Health; accessible abstract/metadata only, no multi-level decision-authority scale is claimed here.

Core Components of AI Governance

1. AI Ethics Committee: - Multidisciplinary: clinicians, ethicists, informaticists, legal, community representatives - Reviews proposed AI deployments for ethical concerns - Authority to approve, modify, or reject AI adoption

2. Equity Impact Assessment: - Required before deploying AI - Analyzes potential disparate impact on vulnerable populations - Identifies mitigation strategies - Includes language-access impact when AI replaces, supplements, or routes interpreter services

3. Transparency Requirements: - AI systems must be documented: purpose, training data, validation studies, known limitations - Performance data (including subgroup performance) made available to clinicians - Patients informed about AI use (tiered consent approach)

4. Ongoing Monitoring: - Real-world performance tracking (overall and subgroup) - Adverse event reporting mechanisms - Regular bias audits

5. Accountability: - Clear designation of responsibility (who is accountable when AI fails?) - Decision authority and accountability match the actual system, intended use, institutional policy, and workflow; an autonomous device and an assistive tool are not governed by the same review pattern - Vendors held accountable for undisclosed risks or misrepresented performance

6. Sunset Provisions: - AI deployment is not permanent. Reevaluate periodically - Decommission AI that performs poorly, perpetuates bias, or becomes outdated

FUTURE-AI: International Consensus Guideline

The FUTURE-AI framework is an international consensus guideline for trustworthy and deployable clinical AI, developed over two years with 117 interdisciplinary experts from 50 countries and 30 lifecycle-spanning best practices (Lekadir et al., 2025).

The FUTURE Acronym:

The framework defines six guiding principles for AI across its entire lifecycle:

Principle	Meaning	Key Requirements
Fairness	Equitable performance across patient populations	Performance stratified by demographics; bias audits; diverse training data
Universality	Applicable across healthcare contexts and settings	Generalizability testing; documentation of intended use populations
Traceability	Transparent development and decision processes	Data provenance; model versioning; audit trails
Usability	Practical integration into clinical workflows	Clinician-centered design; training requirements; workflow fit assessment
Robustness	Reliable performance under real-world conditions	Stress testing; edge case handling; performance under distribution shift
Explainability	Interpretable outputs for clinical decision-making	Output explanations appropriate to clinical context; uncertainty quantification

30 Best Practices Across the AI Lifecycle:

The framework provides 30 actionable recommendations spanning:

• Design and development: Data governance, bias mitigation, model documentation
• Validation: Multi-site testing, subgroup analysis, prospective evaluation
• Regulation: Regulatory pathway navigation, post-market surveillance planning
• Deployment: Implementation readiness, training protocols, integration testing
• Monitoring: Performance tracking, adverse event reporting, model updating

Why This Framework Matters:

Unlike reporting guidance such as TRIPOD+AI and CONSORT-AI, FUTURE-AI addresses development, validation, regulation, deployment, and monitoring across the AI lifecycle. Its recommendations provide a common vocabulary for examining barriers between reported model performance and dependable clinical use.

For Institutional Governance:

When evaluating AI systems, FUTURE-AI principles translate to specific institutional questions:

• Fairness: “Show me performance metrics stratified by race, age, sex, and socioeconomic status.”
• Universality: “Where was this validated? How does performance change across settings?”
• Traceability: “Provide complete documentation of training data sources and model versions.”
• Usability: “What clinician training is required? How does this integrate with our EHR?”
• Robustness: “How does performance degrade with data quality issues or edge cases?”
• Explainability: “What explanations does the system provide for its recommendations?”

FUTURE-AI provides a structured framework for the evaluation questions every institution should ask before deploying clinical AI.

Value Alignment and Transparency Frameworks

Beyond fairness metrics and bias audits, a deeper question remains: what values does an AI system embed, and whose priorities does it serve?

Consider a hypothetical 60-year-old patient with lower back pain. An AI system optimized for greater diagnostic sensitivity could favor MRI, while a system optimized to reduce low-value imaging could favor watchful waiting and physical therapy. Neither recommendation can be judged from the optimization target alone. The clinically important question is which values, evidence, constraints, and patient preferences produced the recommendation.

The Values In the Model (VIM) Framework

The RAISE (Responsible AI for Social and Ethical Healthcare) consortium, convening clinicians, ethicists, legal scholars, and technologists, has proposed a “Values In the Model” (VIM) framework for clinical AI transparency (Goldberg et al., NEJM AI, 2026).

The VIM framework would extend existing model cards (Mitchell et al., 2019) with explicit disclosure of how AI systems navigate value-laden clinical trade-offs:

Value Dimension	What VIM Would Disclose
Resource allocation	Does the system lean toward intervention or conservative management? Was it trained to maximize revenue, minimize costs, or optimize clinical outcomes?
Autonomy vs. paternalism	When patient preferences conflict with clinical benefit, how does the AI weigh these?
Individual vs. population	Does the system consider public health or resource constraints when making individual recommendations?
Certainty thresholds	How does the AI handle diagnostic uncertainty? Does it recommend further testing or watchful waiting?

Why Values Matter for Clinical AI

The VIM consensus paper describes a clinical scenario presented at the 2025 RAISE symposium: a young woman hospitalized with cardiac complications of anorexia refuses oral supplements while her weight remains stable. Human respondents split 47% toward autonomy and 53% toward preventing imminent harm. Fourteen large language models also produced divergent recommendations. The example illustrates value-sensitive disagreement, not a clinical validation study of the models (Goldberg et al., 2026).

AI systems embed values whether their developers intend it or not. Training data reflects institutional incentives and historical practice patterns. Optimization targets (accuracy, cost, utilization) embed priorities. Reinforcement learning from human feedback encodes the values of those providing feedback. Without transparency, these hidden commitments shape clinical decisions at scale.

The RAISE consortium also proposed MedLog, a protocol for event-level logging of clinical AI that creates structured records whenever AI models interact with humans, other algorithms, or act independently (Noori et al., 2025, preprint). This logging system would enable continuous surveillance and auditing, complementing the VIM framework’s upfront disclosure with ongoing monitoring of how AI behaves in practice.

Practical Implications

The VIM paper presents the labeling approach as a proposal, and implementation evidence is not yet established. Its proposed next steps include parallel tracks:

1. Public debate about how values should be addressed in medical AI
2. Pilot projects in leading health systems testing VIM-style disclosures for AI systems already in clinical use
3. Benchmark development using clinical vignettes that probe value trade-offs, tested against diverse stakeholder perspectives (clinicians, patients, insurers, ethicists)

For Institutional Governance Committees:

When evaluating AI systems, institutional governance committees can ask vendors:

• What optimization target was this system trained on? (Accuracy? Cost reduction? Utilization?)
• How does the system handle clinical scenarios where reasonable experts disagree?
• Was the system aligned through human or AI feedback? If so, whose feedback and evaluation criteria shaped it?
• Does the system’s behavior differ when configured for different healthcare contexts (fee-for-service vs. capitated)?

If a vendor cannot answer these questions, the institution lacks evidence needed to determine whether the system’s behavior aligns with its clinical mission and patient interests.

As Persad and colleagues argued regarding allocation of scarce medical interventions, “no single principle is sufficient to incorporate all morally relevant considerations and therefore individual principles must be combined into multiprinciple allocation systems” (Persad et al., Lancet, 2009). The same applies to clinical AI: its future depends not just on technical sophistication but on transparent alignment with values that patients and communities deem acceptable.

Questions About Medical AI Ethics

Frequently Asked Questions

What is algorithmic bias in healthcare?

Algorithmic bias refers to systematic differences in performance, allocation, access, or effects that disadvantage patients or groups. It can arise from the care system, sampling, measurement, labels, proxies, model design, thresholds, workflow, or the intervention triggered by an output.

How do I know if a medical AI system is biased?

Review the intended population, target, labels, subgroup sample sizes, uncertainty, threshold-specific performance, calibration, missingness, and downstream actions. A subgroup gap requires investigation, but no single fairness metric or demographic-parity rule establishes equitable clinical value.

Can AI improve health equity?

AI may support equity by changing access, measurement, or resource allocation, but technical performance alone does not establish equitable benefit. Evaluation must include who receives the intervention, who can act on the output, subgroup effects, workflow burden, and patient outcomes.

Who is responsible when AI causes harm?

Ethical and legal responsibility depends on the facts, jurisdiction, professional role, product design and warnings, institutional governance, workflow, and applicable law. FDA authorization does not determine civil liability, and responsibility should not be reduced to a universal rule that the physician or vendor is always liable.

Conclusion: Toward Ethical Medical AI

Medical AI can improve specific diagnostic, prognostic, and workflow tasks, but technical performance alone does not establish clinical benefit, equitable access, or ethical acceptability. Those claims require evidence matched to the intended use and deployment context.

The history of medical AI includes prospective diagnostic evaluation of autonomous diabetic-retinopathy testing and documented racial bias in a population-health allocation algorithm. These examples differ in task, evidence, and context, but both show that an AI system must be judged by its target, intended use, measured endpoints, affected population, and operational pathway (Abràmoff et al., 2018; Obermeyer et al., 2019).

Ethical medical AI requires:

1. Centering equity: Fit-for-purpose data, subgroup evaluation with uncertainty, bias mitigation, and attention to access
2. Respecting autonomy: Disclosure and consent practices proportionate to the system’s role, risk, and effect on patient choice
3. Demonstrating benefit: Prospective validation of clinical outcomes, not just retrospective accuracy
4. Preventing harm: Rigorous testing, ongoing monitoring, effective human oversight, fallback pathways, and accountability mechanisms
5. Professional stewardship: Understanding system boundaries, maintaining independent judgment, and protecting patient interests

The goal is neither categorical rejection nor uncritical adoption. It is clinical AI governed by medicine’s commitments to patient welfare, human dignity, equity, and protection of people at greatest risk of harm.

Clinicians, patients, health systems, developers, regulators, and other affected communities each hold part of this responsibility. Decision authority, evidence obligations, and accountability should be explicit rather than assumed.

---

Check Your Understanding

The following scenarios are fictional composites for decision practice. Names, systems, performance metrics, institutional actions, legal conclusions, outcomes, and economics are illustrative unless a source is cited. They do not describe actual events, establish a standard of care, or predict liability in any jurisdiction. Each preserves realistic ethical tensions while separating those tensions from factual case reporting.

Hypothetical Composite 1: Risk Stratification Algorithm Perpetuates Racial Bias

You’re a family medicine physician and medical director at a large, urban safety-net hospital serving predominantly Black and Latino patients (72% of patient population). Your health system recently implemented a commercial “high-risk patient identification algorithm” integrated into the EHR to identify patients who would benefit from intensive care management programs.

Algorithm background: - Vendor: Major health IT company with widely-deployed algorithms - Purpose: Predict which patients will have high healthcare utilization in next 12 months - Intended use: Identify patients for enrollment in care management programs (nurse case managers, care coordinators, close follow-up) - Training data: 3 million patients from 50+ health systems nationwide - Reported performance: AUC 0.86 for predicting healthcare utilization - FDA status: Not regulated (administrative use, not diagnostic)

Implementation at your hospital: - Care management program capacity: 500 patients (limited by staffing) - Patient selection: Top 500 highest-risk scores enrolled automatically - Algorithm generates risk scores for all 15,000 patients in practice - Score range: 0-100 (higher = higher predicted utilization, higher priority for care management)

Month 3 - You notice troubling patterns:

Case 1 - Your patient excluded from care management: Patient: 68-year-old Black woman with uncontrolled diabetes (HbA1c 10.2%), CKD stage 4, heart failure, recent hospitalization - Algorithm risk score: 58/100 - Not enrolled in care management (below threshold) - Your clinical assessment: High-risk patient, would greatly benefit from care management

Case 2 - Lower-acuity white patient enrolled: Patient: 64-year-old white man with well-controlled diabetes (HbA1c 6.8%), hypertension, hyperlipidemia, no recent hospitalizations - Algorithm risk score: 73/100 - Enrolled in care management program - Your clinical assessment: Moderate-risk patient, already well-managed in primary care

You raise concerns to administration: “I’m seeing sicker Black patients excluded from care management while healthier white patients are enrolled. Something seems wrong with this algorithm.”

Administration response: “The algorithm doesn’t use race as an input variable. It’s race-neutral. We’re just following the scores.”

You conduct informal analysis: - Review your patient panel (n=347 patients) - Compare algorithm risk scores to your clinical assessment of patient complexity - Finding: For the same level of clinical complexity (# of chronic conditions, hospitalizations, ED visits), Black patients consistently score 10-15 points lower than white patients

You request formal investigation:

Hospital data analytics team retrospective analysis (6 months of data):

Overall findings: - 15,000 patients with algorithm risk scores - Top 500 (enrolled in care management): 31% Black/Latino, 69% white - Your patient population: 72% Black/Latino, 28% white - Dramatic underrepresentation of Black/Latino patients in care management program

Subgroup analysis by race/ethnicity:

For patients with identical clinical profiles (same # of chronic conditions, same prior-year hospitalizations, same ED visits):

Race/Ethnicity	Average Risk Score	% Enrolled in Care Mgmt (out of top 500)
White	67.2	69% (345/500)
Black	53.8	23% (115/500)
Latino	55.1	8% (40/500)

Finding: Black and Latino patients score 13-14 points lower than white patients with identical clinical complexity

Investigation into algorithm design:

Data analytics team reviews vendor documentation:

Algorithm’s prediction target: “Total healthcare costs in next 12 months”

Why costs, not clinical need? - Vendor’s rationale: “Costs are objective, consistently measured across all health systems, and available in billing data” - Used as proxy for healthcare needs

The problem - Cost ≠ Need:

Further analysis at your hospital reveals:

Race/Ethnicity	Avg Healthcare Costs	Avg # Chronic Conditions	Avg # Hospitalizations
White	$18,450	3.2	0.8
Black	$12,100	4.1	1.2
Latino	$10,800	3.9	1.1

Critical finding: Black and Latino patients have higher clinical complexity (more chronic conditions, more hospitalizations) but lower healthcare costs than white patients

Why costs are lower despite higher clinical needs: - Structural barriers to care access: - Lack of insurance (higher uninsured rate in Black/Latino patients: 18% vs. 6% white) - Transportation barriers (fewer Black/Latino patients have cars: 42% vs. 81%) - Work schedule inflexibility (inability to take time off for appointments) - Healthcare system discrimination: - Longer wait times for specialist appointments - Less likely to receive expensive treatments (cardiac cath, joint replacement, advanced imaging) - Less likely to be admitted (higher threshold for hospitalization) - Patient factors: - Medical mistrust (historical and ongoing discrimination) - Cost concerns (avoiding care due to inability to pay)

Result: Black and Latino patients receive less care (lower costs) despite being sicker → Algorithm learns “lower costs = lower need” → Systematically under-prioritizes Black/Latino patients for care management

The ethical violation: This is exactly the scenario described in the Obermeyer Science paper (Obermeyer et al., 2019) occurring at your institution

Questions for Analysis:

1. What ethical principles were violated by this algorithm’s deployment?

Principle #1: Justice and Health Equity (MAJOR VIOLATION)

Definition: Healthcare resources should be distributed fairly based on medical need, not on race or socioeconomic status

Violation: - Care management resources (500 slots) distributed based on algorithm that systematically under-prioritizes Black and Latino patients - Sicker minority patients excluded while healthier white patients enrolled - Algorithm perpetuates and amplifies existing healthcare disparities - 72% Black/Latino patient population → only 31% representation in care management program

Harm: - Black/Latino patients denied care management despite higher clinical need - Existing health disparities widened (patients who need help most receive least) - The pattern warrants immediate civil-rights and regulatory review; whether it constitutes a legal violation depends on the governing program, current law, evidence, and jurisdiction

Principle #2: Beneficence (Doing Good)

Violation: - Algorithm supposed to identify patients who would most benefit from care management - Instead, identifies patients with highest historical costs (not highest needs) - Result: Care management resources allocated suboptimally - Wrong patients enrolled (lower-need white patients) while high-need Black/Latino patients excluded

Harm: - Care management program fails to achieve intended benefit (reducing hospitalizations, improving outcomes for high-risk patients) - Resources wasted on lower-need patients - Higher-need patients experience preventable complications, hospitalizations

Principle #3: Non-Maleficence (First, Do No Harm)

Violation: - Algorithm causes direct harm to Black/Latino patients by excluding them from beneficial intervention - Creates systematic discrimination in resource allocation - Harm is predictable and preventable (vendor and hospital should have detected bias before deployment)

Harm: - Black/Latino patients with uncontrolled chronic conditions, recent hospitalizations, poor medication adherence excluded from care management - Predictable consequences: preventable readmissions, disease progression, complications, reduced quality of life - Harm disproportionately affects vulnerable populations already facing health disparities

Principle #4: Transparency and Informed Consent (Partial Violation)

Violation: - Patients unaware algorithm determines their access to care management program - No disclosure that algorithm uses cost (not clinical need) as proxy for risk - No informed consent for use of algorithmic resource allocation - Patients have no opportunity to appeal algorithmic exclusion

2. Who is responsible and liable for the algorithmic bias?

This case involves DISTRIBUTED ETHICAL AND LEGAL RESPONSIBILITY:

Hospital/Health System (Primary Responsibility):

Ethical failures: - Failed to evaluate algorithm for bias before deployment - Did not request subgroup performance data from vendor - Did not perform local retrospective validation stratified by race/ethnicity - Did not question use of “cost” as proxy for “need” - Deployed algorithm without equity impact assessment - No analysis of potential disparate impact on vulnerable populations - No involvement of ethics committee, community representatives, health equity experts - Failed to monitor for bias post-deployment - Took 3 months and physician complaint to investigate racial disparities - No systematic tracking of care management enrollment by race/ethnicity - Abdicated responsibility to algorithm - Administration response: “We’re just following the scores” (as if algorithm decisions are neutral and physicians have no responsibility to question them)

Legal questions requiring counsel: - Title VI and Section 1557: Federal civil-rights requirements can apply to covered recipients and health programs, but the applicable legal theory, proof, remedies, and private enforcement route are fact- and jurisdiction-specific. The enrollment disparity is a serious trigger for investigation, not a self-executing legal verdict. - Institutional negligence: A claimant could examine whether the hospital used reasonable selection, validation, monitoring, warning, and appeal safeguards. The fictional facts alone do not establish breach or causation. - Disability law: ADA or Rehabilitation Act questions would require analysis of coverage, disability status, the challenged exclusion, reasonable modifications, and causation. Chronic illness does not make every algorithmic disparity an automatic ADA violation.

Possible institutional and regulatory responses: - Preserve the data, suspend automatic allocation, and conduct privileged legal and independent equity review - Notify the appropriate compliance, civil-rights, research, or regulatory offices when required - Redesign eligibility and appeals, provide remediation where warranted, and monitor the revised pathway - Do not predict an investigation, class certification, verdict, remedy, or settlement from a hypothetical fact pattern

AI Vendor (Secondary Responsibility):

Ethical failures: - Poor choice of outcome variable - Used “cost” as proxy for “need” without adequate consideration of how costs reflect access (not just illness severity) - Failed to recognize that historical healthcare costs encode racial disparities - Inadequate bias testing - Did not report subgroup performance by race/ethnicity in validation studies - May have tested for bias but not disclosed (publication bias toward positive results) - Misrepresented algorithm as “race-neutral” - Marketing materials likely claimed algorithm does not use race → therefore fair - Ignoring race does not make algorithm fair when outcome variable (cost) is itself racialized

Vendor-related legal questions: - Product, warranty, contract, consumer-protection, or misrepresentation theories could be examined if the design, documentation, warnings, or sales claims were inaccurate - The authorization and intended-use status of the software, contract allocation, applicable state law, and causal evidence would matter - Hospital governance duties and vendor duties can coexist; neither party’s involvement automatically eliminates the other’s obligations

Defensible conclusion: - Liability and remedies cannot be predicted from this exercise - The operational priority is to stop inequitable allocation, preserve evidence, determine the proxy’s failure mechanism, and redesign or replace the system - The Obermeyer study supplies a documented mechanism for the exercise, but the fictional hospital and vendor facts should not be presented as the history of that study

Physician/Medical Director (YOU) - Ethical Responsibility:

Ethical duties: - Recognize and report bias - YOU fulfilled this duty by noticing pattern and raising concerns - Many physicians would not notice (too busy, trust algorithm, do not track outcomes by race) - Advocate for patients - YOU appropriately advocated for excluded Black/Latino patients - Pushed for formal investigation despite administration resistance - Protect vulnerable populations - Duty to ensure healthcare resources distributed equitably - Responsibility to question algorithms that produce unjust outcomes

The physician’s recognition, documentation, escalation, and patient advocacy satisfy important ethical duties in the exercise. Legal exposure cannot be determined categorically and may depend on what the physician knew, could control, documented, and did after discovery.

3. How should the hospital have evaluated this algorithm BEFORE deployment?

Pre-Deployment Equity Impact Assessment:

Step 1: Critical evaluation of algorithm design

Questions to ask vendor: - “What is your algorithm predicting?” (Answer: Healthcare costs) - “Why use costs as proxy for healthcare needs?” (Probe assumptions) - “Have you validated that costs correlate with clinical need across all racial/ethnic groups?” - “Could structural barriers to care (insurance, transportation, discrimination) affect costs independent of medical need?”

Red flags: - Vendor uses cost, utilization, or access-based metrics as proxies for clinical need - These metrics encode existing disparities - If the vendor cannot provide evidence connecting the proxy to the intended clinical need across relevant groups, pause deployment until the evidence gap is resolved

Step 2: Request subgroup performance data from vendor

Demand: - Algorithm performance stratified by race, ethnicity, age, sex, insurance status, ZIP code - For this algorithm: Correlation between predicted risk scores and actual clinical complexity (# chronic conditions, disease severity) BY RACE - If the vendor refuses or lacks relevant subgroup evidence, treat that gap as a material governance issue and pause deployment or select an alternative

Step 3: Local retrospective validation

Test on YOUR patient population BEFORE deployment: - Select 1,000-2,000 patients from your health system - Run algorithm to generate risk scores - Have clinical team independently assess patient complexity (# chronic conditions, hospitalizations, disease severity) - Critical analysis: Do risk scores correlate with clinical complexity EQUALLY across racial/ethnic groups?

For this algorithm, local validation would reveal: - Black/Latino patients score 13-14 points lower than white patients with identical clinical profiles - Algorithm systematically under-predicts risk for minority patients - This finding should STOP deployment

Step 4: Equity impact assessment

Required before deploying any algorithm that allocates scarce resources:

Questions to answer: 1. Who benefits? Which patients will be enrolled in care management based on algorithm scores? 2. Who is harmed? Which patients will be excluded? 3. Disparate impact analysis: Does enrollment by race/ethnicity match your patient population demographics? 4. Access barriers: Could the algorithm reflect barriers to care rather than medical need? 5. Fairness definition: What does “fair” allocation mean? (Equal access? Proportional to need? Maximize overall benefit?) 6. Alternatives: Are there less biased ways to identify high-risk patients? (Clinical complexity scores, physician referral, patient self-referral)

Require multidisciplinary review: - Clinicians who care for vulnerable populations - Ethicists - Health equity experts - Community representatives (Black/Latino patient advocates) - Legal counsel (Title VI compliance)

Decision criteria: - If equity impact assessment reveals disparate impact → Do not deploy until bias mitigated - If bias cannot be mitigated → Do not deploy, use alternative method

Step 5: Pilot testing with close monitoring

If algorithm passes equity impact assessment: - Deploy to limited pilot (e.g., 100 patients) - Track enrollment by race/ethnicity weekly - Compare to patient population demographics - Investigate any disparities immediately - Expand only if pilot demonstrates equitable allocation

For this algorithm: - Pilot would have revealed 69% white enrollment despite 28% white patient population - Pilot should have stopped deployment before full rollout

4. How should the algorithm be redesigned to eliminate bias?

Option 1: Change outcome variable from “cost” to “clinical complexity”

Instead of predicting: “Healthcare costs in next 12 months”

Predict: “Clinical complexity score” or “Risk of poor outcomes”

Clinical complexity score inputs: - Number of chronic conditions - Disease severity (HbA1c for diabetes, ejection fraction for heart failure, GFR for CKD) - Prior hospitalizations (count, not just cost) - ED visits (count) - Medication non-adherence (fills vs. prescribed) - Social determinants of health (housing instability, food insecurity, transportation barriers)

Key difference: Clinical complexity and social needs are DIRECT measures of who would benefit from care management, not proxies like cost

Validation: Test that clinical complexity scores correlate with care management benefit EQUALLY across racial/ethnic groups

Option 2: Explicitly adjust for known disparities

Acknowledge that Black/Latino patients face barriers to care access: - Add adjustment factor to scores for patients from underserved populations - Example: If Black patient scores 54 but clinical complexity suggests score should be 67, apply +13 point adjustment - Controversial but may be necessary to achieve equitable allocation

Option 3: Abandon algorithmic allocation entirely

Use physician referral or patient self-referral: - Physicians identify patients who would benefit from care management based on clinical judgment - Patients can self-refer if they feel overwhelmed managing their conditions - Advantages: Incorporates clinical nuance, patient autonomy, social factors that algorithms miss - Disadvantages: Physician bias possible, may miss patients who do not seek help

Hybrid approach: - Algorithm provides preliminary risk scores (redesigned to use clinical complexity, not cost) - Physicians review scores and can nominate additional patients based on clinical judgment - Ensures algorithm supports (not replaces) clinical decision-making

Option 4: Use algorithm but monitor for bias rigorously

If algorithm is used: - Monthly monitoring: Track care management enrollment by race/ethnicity, insurance status, ZIP code - Trigger for investigation: Predefine locally justified action thresholds using the expected eligible population, clinical need, uncertainty, and consequences of inequitable exclusion; no universal percentage applies - Transparency: Inform patients how care management enrollment decisions are made - Appeal process: Patients and physicians can appeal if excluded from care management

5. Key lessons for physicians on algorithmic bias and health equity:

Lesson #1: “Race-Neutral” Algorithms Can Be Racially Biased - Algorithm does not need to use race as input variable to produce racially biased outputs - If algorithm uses variables that correlate with race (cost, ZIP code, insurance), it can perpetuate disparities - Always ask: “Does this algorithm perform equally across racial/ethnic groups?”

Lesson #2: Proxies for Outcomes Often Encode Bias - Cost ≠ need (reflects access, not just severity) - Utilization ≠ need (reflects access) - Even seemingly objective measures (lab values, vital signs) can be biased if collection rates differ by race

Lesson #3: Validation Must Include Subgroup Analysis - Overall accuracy metrics hide disparities - Demand race-stratified performance data BEFORE deployment - If vendor lacks or refuses to provide → RED FLAG

Lesson #4: Equity Requires Intentional Design - Algorithms optimized for overall accuracy will often perform worse for minority groups - Equity must be explicit design goal (not assumed) - Diverse development teams, inclusive validation, ongoing bias monitoring required

Lesson #5: Physicians Have Duty to Identify and Report Bias - Do not assume algorithms are neutral or fair - Track outcomes by race/ethnicity in your practice - Speak up when you notice disparities - Professional obligation to protect vulnerable patients

Lesson #6: “Just Following the Algorithm” Is Not an Ethical Defense - Physicians retain professional responsibility for patient care - Cannot delegate ethical judgment to algorithms - Duty to question and override algorithms that produce unjust outcomes

Lesson #7: Health Equity Impact Assessment Should Be Mandatory - Before deploying any algorithm that allocates resources or affects clinical decisions - Multidisciplinary review including community representatives - Ongoing monitoring, not just pre-deployment assessment

Lesson #8: Algorithmic Bias Is a Civil Rights Issue - Title VI prohibits discrimination in federally-funded programs - Facial neutrality does not eliminate civil-rights risk when an allocation pathway produces inequitable exclusion - The governing legal standard, enforcement route, and remedies require current, jurisdiction-specific review

Hypothetical Composite 2: Informed Consent Failure with Experimental AI

You’re a surgical oncologist at an academic medical center. Your hospital recently partnered with a medical AI startup to pilot an experimental algorithm that predicts post-operative complications for cancer surgery patients. The algorithm uses machine learning to analyze preoperative data (demographics, labs, imaging, tumor characteristics) and generates a personalized risk score for complications (surgical site infection, pneumonia, cardiac events, mortality).

Study details: - Goal: Validate algorithm in real-world clinical setting, collect data to improve model - Design: Prospective observational study; all colorectal cancer surgery patients offered enrollment - Intervention: Patients in study receive personalized risk report pre-surgery, shared with surgical team - Control: No control group; all enrolled patients receive AI risk prediction - IRB status: Approved as minimal risk study (observational, no change to standard surgical care) - Informed consent: Written consent obtained by research coordinator

Your involvement: - Principal investigator for surgical oncology patients - Responsible for explaining study to patients and obtaining informed consent when research coordinator unavailable

Month 2 - Complex patient enrolled in study:

Patient: 71-year-old Black man with stage III colon cancer - Medical history: Hypertension, type 2 diabetes, obesity (BMI 34), former smoker - Social history: Retired postal worker, married, supportive family, lives in rural area 90 miles from hospital - Baseline functional status: Independent, active, good quality of life - Cancer: Locally advanced but resectable; standard treatment is surgical resection followed by adjuvant chemotherapy - Prognosis with surgery: Good chance of cure (60-70% 5-year survival) - Prognosis without surgery: Progression, metastasis, death within 2-3 years

Study enrollment conversation (conducted by research coordinator, you present but silent):

Research coordinator: “Dr. Johnson’s team is conducting a research study using artificial intelligence to predict surgical complications. We’d like to invite you to participate. If you enroll, a computer algorithm will analyze your medical information and provide a personalized risk estimate for complications after surgery. This information will be shared with you and your surgical team. Participation is voluntary and won’t change your surgical care. Do you have any questions?”

Patient: “Will this computer thing help me avoid complications?”

Research coordinator: “It provides information about your risk, which may help the surgical team plan your care.”

Patient: “Okay, sounds good. Where do I sign?”

[Patient signs consent form, study enrollment complete]

Pre-operative AI risk report generated:

Algorithm output: - Predicted 30-day post-op complication risk: 68% (HIGH RISK) - Predicted 30-day mortality risk: 12% (HIGH RISK) - Factors driving high risk: Age 71, BMI 34, diabetes, Black race, ASA class III

Breakdown compared to average patient: - Average patient undergoing similar surgery: 24% complication risk, 2% mortality risk - This patient: 2.8× higher complication risk, 6× higher mortality risk

The report flags: “HIGH RISK PATIENT - Consider enhanced perioperative monitoring, ICU admission post-op, cardiology consultation”

You review the report and meet with patient to discuss:

You: “The AI system we’re using for the study has generated your personalized risk report. It estimates you have a 68% chance of complications and 12% chance of death within 30 days after surgery. These numbers are higher than average.”

Patient’s reaction: Visible distress, tearful, hands shaking

Patient: “12% chance of death? I didn’t know surgery was that dangerous. My wife and I were planning a trip after I recover. Should I even do this surgery?”

You (attempting to provide reassurance): “These are just predictions from a computer model. I’ve done hundreds of these surgeries. Your risk is higher than some patients because of your age and diabetes, but I think surgery is still your best option. Without surgery, the cancer will spread.”

Patient: “But the computer says 68% complications. That’s more than half. What kind of complications?”

You: “Infection, pneumonia, heart problems. We’ll monitor you closely. The AI is supposed to help us prepare.”

Patient (hesitant, uncertain): “I need to think about this. Can I talk to my wife?”

[Patient leaves, clearly shaken by risk numbers]

Next day - Patient returns with wife:

Patient: “We’ve decided not to do surgery. The risks are too high. 12% mortality is too scary. We’ll try other treatments.”

You: “Other treatments won’t cure your cancer. Surgery gives you the best chance.”

Wife: “But the computer says he has a 68% chance of complications. That’s not acceptable. We’re not doing it.”

You spend 45 minutes explaining: Cancer prognosis without surgery, benefits of surgery, how to interpret risk percentages, limitations of AI predictions

Patient remains undecided, leaves to “think more”

2 weeks later - Patient misses surgery date:

You call patient at home:

Patient: “We decided not to do the surgery. We’re going to try alternative medicine and diet changes. We researched online and found stories about people beating cancer naturally. The surgery risks are too high.”

You (frustrated, concerned): “I strongly advise surgery. This is a curable cancer if we operate now. Delaying surgery means the cancer will grow and spread. Alternative medicine won’t cure stage III colon cancer.”

Patient: “The computer said 68% complications. I’ve made my decision.”

[Patient declines surgery, opts out of further oncology care at your hospital]

6 months later - Patient presents to emergency department at different hospital:

Status: Cancer progression, now stage IV with liver metastases, bowel obstruction requiring emergency surgery - Emergency surgery performed: Palliative colostomy, no curative resection possible (metastatic disease) - Prognosis: Incurable, median survival 12-18 months with chemotherapy

You learn about outcome when reviewing study data: - Patient was enrolled in AI study - Declined curative surgery after seeing AI risk prediction - Now has incurable metastatic disease

You feel ethical distress: Did the AI risk prediction and inadequate informed consent contribute to this patient’s refusal of life-saving surgery?

Questions for Analysis:

1. What informed consent failures occurred in this case?

Failure #1: Inadequate Explanation of AI Uncertainty and Limitations

What patient should have been told: - “This AI has been trained on thousands of patients, but it’s NOT been specifically validated for patients like you” - “The algorithm’s predictions have uncertainty. Actual risk could be higher or lower” - “We don’t know how accurate this AI is for Black patients because the training dataset had few Black patients” - “Your individual circumstances may differ from algorithm’s predictions”

What patient was actually told: - Research coordinator: Vague description of “computer algorithm analyzes medical information” - You: Presented AI risk numbers (68%, 12%) without adequate context about uncertainty - Patient left with impression AI predictions were precise and definitive

Harm: Patient made life-altering decision (refuse surgery) based on misunderstood AI predictions

Failure #2: No Discussion of AI Training Data Representativeness

Critical missing information: - The study team has not documented the training population, representation of relevant subgroups, calibration, or external validation in the scenario - Before discussing subgroup performance, the team should verify whether the model has been evaluated in patients who resemble the participant and report the uncertainty honestly - Because the model includes race as a predictor, the team should explain the rationale, test whether that use is clinically and ethically justified, and avoid attributing a risk difference to race without evidence

Why this matters: - The patient’s group representation and model performance are unknown from the facts provided - The estimate could be too high, too low, or well calibrated; subgroup uncertainty does not justify assuming its direction - The ethical failure is presenting a precise estimate without establishing its validity for the intended population and decision

What should have been disclosed: - Uncertainty about algorithm accuracy for this patient’s demographic profile - The verified composition and limitations of the development and validation data, including what remains unknown - Strong recommendation: Do not invent a direction or magnitude of bias when the supporting analysis has not been performed

Harm: Patient made decision based on potentially biased, inaccurate risk estimates

Failure #3: Inadequate Explanation of How AI Risk Compares to Physician Judgment

What patient should have been told: - “As your surgeon, based on 20 years of experience doing this operation, I believe your risk is elevated but manageable” - “The AI reports a 68% complication risk, but the model’s calibration, endpoint definition, uncertainty, and relevance to your circumstances must be checked before that number guides care” - “The model is one input. My recommendation also uses the clinical record, validated surgical-risk tools where appropriate, current evidence, and your goals” - “I’ve successfully operated on many patients with your risk profile”

What patient was told: - You presented AI numbers without clear comparison to your own clinical assessment - Patient left thinking: “Computer says 68%, doctor says surgery best option” → Conflicting messages without clarity about which to trust

Harm: Patient unclear about whose assessment (AI vs. physician) was more trustworthy; defaulted to trusting “objective” computer over physician’s clinical judgment

Failure #4: No Discussion of Alternative Interpretation Frameworks

AI presented risk as: “68% complication risk, 12% mortality risk”

Alternative framing: - “88% chance you’ll survive surgery” (focusing on survival probability) - “The composite endpoint must be separated into complications that differ in severity, reversibility, and importance to the patient” - “The risks of surgery and nonsurgical management should come from current, diagnosis-specific evidence, not an invented certainty” - “Your baseline health is good despite elevated risk” (emphasizing patient strengths)

Patient received: Worst-case framing without context

Failure #5: Inadequate Assessment of Patient’s Understanding

Research coordinator obtained consent, but did not assess: - Does patient understand AI is experimental and uncertain? - Does patient understand AI provides probabilities, not certainties? - Does patient understand how to weigh AI risk estimates against alternative risks (death from cancer without surgery)? - Does patient have health literacy to interpret percentages and risk?

You later attempted to explain, but patient had already formed strong negative impression from initial AI risk report

Failure #6: No Opportunity for Patient to Opt Out of Receiving AI Risk Prediction

Consent process: - Patient asked: “Do you consent to participate in AI study?” - Patient consented (thought it would “help”) - No option to participate in study data collection but NOT receive AI risk report

Ethical problem: - AI risk report was potentially harmful (induced fear, led to refusal of life-saving surgery) - Patient had no opportunity to decline receiving risk report - Consent to “participate in research” should have included separate consent to “receive AI prediction results”

Analogy: Genetic testing studies allow patients to opt out of receiving results; this study should have too

2. What ethical principles were violated?

Autonomy Violation (Inadequate Informed Consent): - Patient’s decision was NOT truly informed (lacked critical information about AI uncertainty, limitations, bias) - Consent process was pro forma (check-box) rather than substantive - Patient could not meaningfully weigh AI risks against cancer risks without adequate explanation - Result: Patient’s autonomous choice was based on inadequate, misleading information

Non-Maleficence Violation (First, Do No Harm): - The communication of the AI risk report plausibly contributed to a harmful decision pathway; the fictional sequence does not isolate a single cause - Foreseeable harm: Risk predictions can induce fear and lead to refusal of beneficial treatments - Duty to minimize harm: Should have recognized risk of harm from AI predictions, mitigated through better consent and framing - Result in the exercise: The patient later presented with metastatic disease after declining recommended surgery, but the contribution of the AI report must be assessed rather than assumed

Beneficence Violation (Duty to Benefit Patient): - Study’s primary goal: Validate AI algorithm, collect data to improve model (research goal) - Secondary goal: Help patients make informed decisions (clinical goal) - Hierarchy failure: Research goal prioritized over patient’s clinical benefit - Patient received a prediction without adequate interpretive safeguards; whether study participation caused the refusal would require comparison and causal evidence - Result: The sequence is a safety signal that the study should investigate, not proof of a model-specific causal effect

Justice Violation (Equitable Treatment): - The study team did not establish representation or subgroup performance before presenting an estimate for a consequential decision - This patient: The estimate’s validity and direction of error are unknown; race should not be invoked as the explanation without model documentation and subgroup analysis - Experimental AI with uncertain applicability was used without sufficient communication and monitoring safeguards

3. Who is liable for the patient’s progression to incurable cancer?

This is a complex case of distributed ethical responsibility. Liability cannot be assigned from the exercise alone:

Institutional Research Review Board (IRB) - Significant Responsibility:

Failures: - Approved study as “minimal risk” without adequate consideration of harms from AI predictions - Risk assessment focused on physical harm from surgery (no different than standard care) - Failed to consider psychological and decision-making harms from AI risk predictions - Risk predictions can induce fear, anxiety, refusal of beneficial treatments - NOT minimal risk for patients receiving potentially inaccurate, fear-inducing predictions - Inadequate informed consent process - Consent form likely did not adequately explain AI uncertainty, limitations, training data bias - No assessment of whether patients understood how to interpret probabilistic risk estimates - No provision for patients to opt out of receiving AI results while still participating in data collection - No safeguards for vulnerable populations - The protocol did not document representation, subgroup performance, or how uncertainty would be communicated when applicability was unclear - Safeguards should follow the participant’s decision risk, evidence gap, and vulnerability to harm, without presuming incapacity or treating race alone as vulnerability

Corrective actions: - IRB should have required enhanced informed consent process - Should have required researcher to assess patient understanding before presenting AI results - Should have included option to opt out of receiving AI predictions - Should have required closer monitoring of study harms (patient refusal rates, psychological distress)

Principal Investigator (YOU) - Significant Responsibility:

Failures: - Inadequate informed consent when presenting AI risk results - Presented numbers (68%, 12%) without sufficient context about uncertainty - Did not explain AI training data limitations or potential bias for Black patients - Did not clearly state your own clinical judgment conflicted with AI prediction - Failed to recognize patient distress and intervene appropriately - Patient was “visibly distressed, tearful, hands shaking” after seeing AI report - Should have immediately provided counseling, additional explanation, possibly referral to psychology/social work - Instead, patient left shaken and made hasty decision to decline surgery - Inadequate follow-up after patient declined surgery - Called patient once, but did not offer prompt, respectful follow-up proportionate to the time-sensitive decision - Could have offered another meeting, an independent second opinion, multidisciplinary counseling, or coordination with the patient’s primary clinician while preserving decisional autonomy - Protocol design flaw - Study designed to give AI predictions to all patients without control group receiving standard counseling - No way to assess if AI predictions helped or harmed decision-making

Legal liability: - Informed consent case: Patient could argue decision to decline surgery was based on inadequate informed consent about AI limitations - Standard: Would reasonable patient have declined surgery if adequately informed about AI uncertainty and training data bias? - Outcome boundary: A claim would require jurisdiction-specific analysis of duty, disclosure standards, causation, damages, research obligations, and comparative evidence. No verdict or settlement range can be predicted from this fictional account.

Research Coordinator - Some Responsibility:

Failures: - Consent conversation was cursory (patient asked one question, signed immediately) - No assessment of patient’s understanding - Presented study in overly positive light (“may help surgical team plan your care”) without balancing with potential harms

But: Research coordinator likely followed IRB-approved consent script; primary responsibility lies with PI and IRB

AI Startup Vendor - Some Responsibility:

Failures: - The scenario does not establish what validation the vendor performed, which is itself a due-diligence gap - Any accuracy or applicability claim should match the documented evidence and intended population - Warnings should identify verified limitations and uncertainty for populations with sparse or inconclusive evidence

But: Vendor provided algorithm for research study; hospital/PI responsible for appropriate use and informed consent

Patient autonomy and outcome: - Patient made autonomous decision to decline surgery based on AI risk estimates - Patient did not fully understand AI limitations (but that’s informed consent failure, not patient’s fault) - Respect for the patient’s decision does not excuse inadequate communication, and an adverse outcome does not make the patient legally or ethically blameworthy

4. How should informed consent have been conducted?

Enhanced Informed Consent Process for Experimental AI:

Step 1: Pre-consent educational session (before presenting AI results)

Research coordinator and surgeon together explain:

“We’re conducting research on artificial intelligence to predict surgical complications. If you participate, a computer algorithm will analyze your medical information and generate a risk estimate. Before you decide whether to participate, I want to explain what AI predictions mean and don’t mean:

What AI does well: - Analyzes large amounts of data quickly - Identifies patterns humans might miss - Provides statistical estimates based on thousands of similar patients

What AI does not do well: - Does not know YOU specifically (does not account for your unique circumstances, motivation, family support) - Makes predictions with uncertainty (actual risk could be higher or lower) - May be less accurate for patients from groups underrepresented in training data

Limitations of THIS specific AI: - This algorithm is experimental. We’re testing how well it works - It was trained mostly on patients at academic medical centers, with fewer Black, Latino, and rural patients - We do not know for certain how accurate it is for patients like you - It provides probabilities, not certainties

How you should use AI predictions: - AI provides one piece of information among many - Dr. Johnson’s clinical judgment based on 20 years of experience is equally (or more) important - We’ll discuss AI results together and help you interpret them - You can ask questions, challenge the AI, and make your own decision

Participation is voluntary: - You can decline to participate - You can participate in the study (allowing us to collect data) but opt out of receiving AI risk predictions - You can receive AI predictions but also get a second opinion from another surgeon - You can withdraw from the study at any time

Do you have questions before deciding whether to participate?“

Step 2: Assess patient’s understanding before enrolling

Ask patient: - “Can you explain in your own words what the AI does?” - “What does it mean if the AI says you have a 60% risk of complications?” - “If the AI prediction differs from Dr. Johnson’s judgment, whose assessment would you trust more?” - “What will you do if the AI prediction makes you worried about surgery?”

If patient does not understand → More education before enrollment

Step 3: Present AI results with context and framing

If patient enrolls and receives AI risk prediction:

You (surgeon) meet with patient BEFORE showing numbers:

“The AI has generated your personalized risk estimate. Before I show you the numbers, I want to give you my clinical assessment. I’ve reviewed your medical history, your cancer, and your overall health. Based on my experience, I believe you’re a good candidate for surgery. Your risks are elevated compared to younger, healthier patients, but I’ve successfully operated on many patients with similar or higher risk. I believe surgery gives you the best chance for cure.”

Then show AI report:

“The AI predicts you have a 68% risk of complications and 12% risk of death within 30 days. These numbers are higher than average, and I want to explain what they mean and don’t mean:

What ‘complication’ means: - Includes minor complications (wound infection, treatable with antibiotics) - Includes moderate complications (pneumonia, requiring antibiotics and extra hospital days) - Includes serious complications (heart attack, requiring ICU care) - Most complications are treatable and do not affect long-term outcomes

What ‘12% mortality risk’ means: - 88% chance you’ll survive surgery - This is a statistical average from thousands of patients - Your individual risk depends on factors AI cannot fully capture (your determination, family support, our surgical team’s experience)

Why the estimate remains uncertain: - The scenario does not establish the model’s development population, subgroup calibration, confidence interval, or external validation - The team should retrieve and explain those facts rather than assume that subgroup error is an overestimate - The use of race as a predictor requires a documented clinical rationale and evaluation of whether it improves or distorts the intended estimate

My clinical judgment: - I will not substitute an unsupported personal percentage for an unsupported model percentage - I will compare the model with validated risk tools, multidisciplinary assessment, current disease-specific evidence, and the operative team’s documented outcomes when available - I will explain the expected benefits, burdens, alternatives, uncertainty, and time sensitivity in terms appropriate to this patient’s cancer and goals - I will distinguish population estimates from individual certainty

Comparing risks: - Surgery and nonsurgical management each require evidence-based estimates with time horizons, endpoints, and uncertainty - Avoid presenting either pathway as certain unless the evidence truly supports certainty

From my perspective, surgery is clearly your best option despite elevated risks. But this is your decision. What questions do you have?“

Step 4: Shared decision-making with time to process

Do not rush decision: - “This is a lot of information. Take time to process it.” - “Talk to your wife, your family, your primary care doctor.” - “We can schedule another appointment to discuss further before surgery.”

Offer second opinion: - “If you’d like another surgeon’s perspective, I can arrange that.” - “Would you like to speak with patients who’ve had similar surgeries?”

Assess ongoing understanding: - At follow-up appointment: “Can you tell me your understanding of the risks and benefits of surgery?” - “What factors are most important to you in making this decision?” - “Do you have concerns we haven’t addressed?”

Step 5: Document informed consent process

In medical record: - “Patient enrolled in AI surgical risk prediction study. Extensive counseling provided about AI limitations, uncertainty, potential bias for underrepresented populations.” - “AI predicted 68% complication risk and 12% mortality risk. I documented the endpoint definitions, available validation, uncertainty, and any known subgroup limitations. I did not assign a different numerical risk without a validated basis.” - “Patient expressed understanding of risks and benefits. Patient chose to proceed with surgery [or declined surgery]. Patient’s decision was informed, voluntary, and made after careful consideration.”

5. Key lessons for physicians on informed consent with AI:

Lesson #1: Experimental AI Requires Enhanced Informed Consent - Standard consent insufficient when AI is unvalidated, uncertain, or potentially biased - Patients must understand AI limitations, not just capabilities - IRBs should require enhanced consent for AI studies with decision-making implications

Lesson #2: AI Predictions Are Not Certainties - Probabilistic estimates have uncertainty - Patients often misinterpret percentages (68% sounds like “will definitely happen”) - Frame AI predictions clearly: “statistical average” not “your certain outcome”

Lesson #3: Disclose Training Data Limitations and Bias - If AI trained predominantly on non-representative populations, inform patients - “This AI may be less accurate for patients like you” is critical disclosure - Especially important for minority patients given pervasive AI training data bias

Lesson #4: Physician Judgment Must Contextualize AI Predictions - Do not present AI numbers in isolation - Clearly state your clinical assessment and how it compares to AI - If you disagree with AI → explain why, help patient weigh both perspectives

Lesson #5: Assess Patient Understanding, Do Not Assume It - Use teach-back method: “Can you explain in your own words?” - Patients with limited health literacy may need more support interpreting AI predictions - Do not rush consent process

Lesson #6: Consider Psychological Harms from AI Predictions - Risk predictions can induce fear, anxiety, refusal of beneficial treatments - IRBs and researchers must consider these harms in risk assessment - “Minimal risk” determination incorrect if AI predictions likely to alter patient decisions

Lesson #7: Offer Option to Opt Out of Receiving AI Predictions - Some patients may want AI to inform their care; others may find predictions unhelpful or distressing - Ethical to participate in research data collection without receiving AI results - Respect patient autonomy in how they use (or do not use) AI information

Lesson #8: Follow Up Aggressively When Patients Refuse Beneficial Treatment After AI Predictions - If AI prediction leads to refusal of life-saving treatment → intervene immediately - Offer additional counseling, second opinions, time to reconsider - Do not accept patient’s refusal as final after single conversation

Hypothetical Composite 3: Automation Bias Leads to Missed Diagnosis

You’re an emergency medicine physician at a busy urban Level I trauma center. Your ED recently implemented a fictional AI-powered chest X-ray interpretation system, described in the exercise as FDA-authorized for specified findings, that analyzes chest radiographs and flags abnormalities. The product, metrics, workflow, and marketing language below are illustrative and do not describe an actual device.

AI system details: - Vendor: Major medical imaging AI company - Regulatory assumption for the exercise: FDA 510(k) clearance for specified findings - Illustrative vendor-reported sensitivity: 94% for pneumonia, 92% for pneumothorax, 91% for pulmonary edema - Deployment: Integrated with PACS (Picture Archiving and Communication System) - Workflow: Radiologist and ED physician both see AI outputs overlaid on images - Marketing claim: “Reduces missed diagnoses by 30%”

Implementation at your ED: - Go-live 6 months ago - All chest X-rays automatically analyzed by AI - AI generates color-coded overlays on images (red boxes around abnormalities) - ED physicians see AI analysis before or concurrently with radiologist preliminary read - Training: 2-hour online module + 1 in-person session on AI interface

Night shift - High-volume night (typical): - Volume: 47 patients in ED, 12 waiting to be seen - Staffing: You + 2 other ED physicians, 1 resident - Acuity: 3 critical (trauma, MI, stroke), 8 urgent, 36 non-urgent

11:30 PM - New patient arrival:

Patient: 34-year-old woman, presents with chest pain and shortness of breath - Chief complaint: “Chest pain and hard to breathe for 2 days” - History: Started 2 days ago, gradual onset, worse with deep breath, no radiation, no associated symptoms - Past medical history: No significant medical history, takes oral contraceptives - Social history: Non-smoker, works as teacher, no recent travel - Vital signs: Temp 98.4°F, HR 102, BP 118/74, RR 22, O2 sat 94% on room air (mildly decreased)

Your assessment (busy, high patient volume, trying to move quickly): - Differential diagnosis: Pneumonia, pleurisy, musculoskeletal pain, anxiety - Plan: Chest X-ray, consider CBC, D-dimer if concerned for pulmonary embolism

Chest X-ray ordered, completed at 11:45 PM

AI analysis (appears on PACS within 2 minutes): - AI output: “No acute abnormalities detected” - No red boxes or highlighted regions on X-ray - Confidence score: 87% (high confidence)

You review chest X-ray on PACS: - Your viewing time: 18 seconds (time-stamped in EHR) - Glance at image on workstation while simultaneously checking lab results for another patient - See AI analysis first: “No acute abnormalities detected” - Your interpretation: Quickly scan image, no obvious infiltrates, no large pneumothorax, heart size normal - Your conclusion: “Agree with AI, chest X-ray looks okay”

Physical exam: - Lungs: Clear to auscultation bilaterally (patient has some difficulty taking deep breaths due to pain, limiting exam) - Heart: RRR, no murmurs - Chest wall: No tenderness to palpation

Your diagnosis: Viral pleurisy vs. musculoskeletal chest pain

Plan: - D-dimer ordered “just to be safe” given oral contraceptive use - Reassure patient, NSAIDs for pain, return if symptoms worsen

D-dimer result: 1,247 ng/mL (normal <500, elevated)

Your interpretation: - D-dimer elevated but non-specific (can be elevated in many conditions) - Chest X-ray negative, patient clinically stable - Decision: Low suspicion for PE, do not pursue CT angiography (CTA chest) - Reasoning: “Chest X-ray negative, probably viral pleurisy with some inflammation causing elevated D-dimer”

Discharge plan: - NSAIDs (ibuprofen 600mg Q6H) - Return precautions: If shortness of breath worsens, chest pain worsens, or if you develop fever - Primary care follow-up in 1 week - Patient discharged at 1:15 AM

Next day, 3 PM - Patient returns to ED:

Patient presentation: - Chest pain much worse, severe shortness of breath, light-headed - Vital signs: Temp 98.9°F, HR 128, BP 96/62, RR 32, O2 sat 88% on room air (significantly decreased) - Exam: Tachycardic, tachypneic, diaphoretic, in moderate distress

Day shift ED physician (different physician) management: - High suspicion for pulmonary embolism given tachycardia, hypoxia, pleuritic chest pain, oral contraceptive use, elevated D-dimer from prior visit - CTA chest ordered STAT

CTA chest result: - Bilateral pulmonary emboli (moderate-large clot burden in right and left main pulmonary arteries) - Right heart strain on imaging (RV dilation)

Patient management: - Admitted to ICU - Anticoagulation (heparin drip) - Supplemental oxygen - Hemodynamically stable after treatment, improves over 48 hours - Outcome: Survives, discharged on anticoagulation, no long-term sequelae

Root cause analysis triggered:

Quality and safety team reviews case:

Key question: Why was PE not diagnosed on initial ED visit despite pleuritic chest pain, tachycardia, mild hypoxia, oral contraceptive use, and elevated D-dimer?

Chest X-ray re-review by attending radiologist (not on call during initial visit):

Retrospective interpretation: - The radiologist notes a small peripheral opacity and vascular asymmetry that are nonspecific and were not diagnostic of pulmonary embolism - The radiograph may help evaluate alternative diagnoses, but a negative or nonspecific chest radiograph does not rule out pulmonary embolism

Radiologist conclusion: “The retrospective findings are subtle and nonspecific. The central error was allowing a negative chest-radiograph result to lower concern raised by the clinical presentation and positive D-dimer.”

AI algorithm re-review:

Quality team runs original chest X-ray through AI system again: - AI output (consistent with original): “No acute abnormalities detected” - AI confidence: 87% - AI MISSED the subtle PE findings (Hampton’s hump, Westermark sign, enlarged PA)

Investigation: Why did AI miss PE findings?

AI vendor response: - “Our algorithm is trained to detect pneumonia, pneumothorax, pulmonary edema, masses, and fractures” - “Pulmonary embolism findings were not included in primary training objectives” - “Subtle PE findings (Hampton’s hump, Westermark sign) are rare in training dataset” - AI was not designed or validated to detect PE, despite broad fictional marketing language that users interpreted as comprehensive

Your cognitive process reconstructed (through interview during RCA):

Cognitive bias identified: Automation bias

You explain: - “I was very busy that night, 47 patients in ED, multiple critical cases” - “I looked at the chest X-ray, saw the AI said ‘no acute abnormalities,’ and agreed” - “I trusted the AI as a second reader. Thought if AI didn’t flag anything, I could move on quickly” - “In retrospect, I should have looked more carefully at the right lung base and pulmonary arteries” - “The AI output influenced my interpretation. I was looking to confirm ‘no abnormality’ rather than actively searching for findings”

Automation bias: Tendency to over-rely on automated systems and under-value contradictory information from other sources (clinical presentation, elevated D-dimer)

RCA findings:

Contributing factors to missed PE diagnosis: 1. Automation bias: Your over-reliance on AI “no abnormalities” output reduced your scrutiny of X-ray 2. High patient volume: Busy ED, limited time to carefully review each X-ray 3. AI system limitations: AI not trained to detect PE findings, but this was NOT disclosed to ED physicians 4. Inadequate training: Training module did not explain AI limitations, failure modes, or situations where AI might miss findings 5. Subtle findings: Hampton’s hump and Westermark sign are subtle, easily missed even without AI

RCA conclusion: “The missed diagnosis was multifactorial. The ED physician exhibited automation bias, over-relying on AI output. However, the AI system’s failure to detect PE findings (and lack of transparency about this limitation) contributed to the diagnostic error. The hospital bears some responsibility for deploying AI without adequately informing physicians of its limitations.”

Questions for Analysis:

1. What is automation bias and how did it contribute to the missed diagnosis?

Automation Bias Definition:

Automation bias is the propensity for humans to favor suggestions from automated systems and to ignore contradictory information from other sources, even when the automated system is incorrect (Goddard et al., 2012).

Two types of automation bias: 1. Errors of commission: Acting on incorrect automated advice (false positive) 2. Errors of omission: Failing to detect problems because automated system did not alert (false negative) ← This case

How automation bias occurred in this case:

Normal cognitive process WITHOUT AI: - ED physician orders chest X-ray - Physician carefully reviews X-ray, actively searching for abnormalities - Considers differential diagnosis: pneumonia, pneumothorax, PE, pleurisy - Weighs X-ray findings against clinical presentation (pleuritic pain, tachycardia, hypoxia, OCP use, elevated D-dimer) - Conclusion: Elevated D-dimer + pleuritic pain + hypoxia → high suspicion for PE → order CTA

Actual cognitive process WITH AI: - ED physician orders chest X-ray - AI analysis appears first: “No acute abnormalities detected” - Physician glances at X-ray (18 seconds viewing time) - Anchoring on AI output: Physician looks to CONFIRM AI assessment rather than independently search for findings - Satisficing behavior: “AI says normal, looks okay to me, I can move on” - Reduced scrutiny: Subtle PE findings (Hampton’s hump, Westermark sign) overlooked - Dismissal of contradictory information: Elevated D-dimer not pursued because “chest X-ray negative” - Conclusion: Viral pleurisy, discharge with NSAIDs

Psychological mechanisms underlying automation bias:

1. Trust in technology: - Belief that AI is more accurate/reliable than human judgment - In the exercise, FDA-cleared for specified findings, accompanied by broad sensitivity and missed-diagnosis marketing claims - Physician reasoning: “If AI doesn’t see it, it’s probably not there”

2. Cognitive offloading: - High workload environment (47 patients, busy night shift) - AI provides mental shortcut: “AI analyzed image thoroughly, I can focus on other tasks” - Trade-off: Efficiency (faster X-ray review) vs. accuracy (missed subtle findings)

3. Confirmation bias: - Once AI says “no abnormalities,” physician looks to confirm (not challenge) that assessment - Subtle findings dismissed as artifacts or normal variants - Contradictory data (elevated D-dimer) re-interpreted to fit “normal X-ray” narrative

4. Diffusion of responsibility: - AI as “second reader” creates perception of shared responsibility - Implicit reasoning: “Two readers (me + AI) both say normal → must be normal” - Reality: AI missed findings, physician also missed → both wrong, no safety net

Key lesson: Automation bias is NOT physician laziness or incompetence. It’s predictable cognitive phenomenon that occurs even in well-trained, experienced clinicians when using automated decision support systems.

2. What evaluation and implementation failures allowed this case to occur?

Failure #1: Inadequate Transparency About AI Limitations

What ED physicians should have been told BEFORE deployment: - “This AI is trained to detect pneumonia, pneumothorax, pulmonary edema, masses, and fractures” - “AI is NOT trained to detect pulmonary embolism findings (Hampton’s hump, Westermark sign, enlarged pulmonary arteries)” - “If clinical suspicion for PE, don’t rely on AI. Independently search for PE findings and order CTA if indicated” - “AI sensitivity for subtle findings is lower than for obvious abnormalities”

What ED physicians were actually told: - “AI detects abnormalities on chest X-ray with 92-94% sensitivity” - General statement, no mention of specific conditions AI does/does not detect - Physicians assumed AI was comprehensive (detects all pathology)

Harm: Physician trusted AI “no abnormalities” output without knowing AI was not designed to detect PE

Failure #2: Inadequate Training on Automation Bias

What training should have included: - “Automation bias is real. Clinicians tend to over-rely on AI, especially when busy” - “To avoid automation bias: Review image BEFORE looking at AI output, form your own impression, then use AI as check” - “Don’t use AI as shortcut in high-stakes cases (chest pain, dyspnea, trauma)” - “If clinical presentation doesn’t match AI output → trust your clinical judgment, not AI”

What training actually included: - How to use AI interface (technical training) - Examples of AI detecting pneumonia, pneumothorax (success cases) - No discussion of cognitive biases, failure modes, or when NOT to trust AI

Failure #3: Poor Workflow Design

Current workflow: AI output appears on PACS immediately, often BEFORE physician reviews image

Problem: Physician sees AI assessment first → anchoring bias

Better workflow: - Physician reviews image FIRST, documents preliminary interpretation - THEN views AI output as “second reader” - Compare physician interpretation to AI - Resolve discrepancies (Why did I see finding and AI did not? Why did AI flag finding I did not see?)

This workflow reduces automation bias by preserving physician’s independent assessment

Failure #4: No Clinical Decision Support for PE Risk Stratification

AI provided: Image interpretation only (“no abnormalities”)

AI did not provide: Clinical risk stratification integrating imaging + clinical data

What would have helped: - A validated local pathway that prompts documentation of pretest probability and interprets D-dimer in context, without assigning the subjective Wells item automatically - For low or intermediate pretest probability with a positive D-dimer, the ACR Appropriateness Criteria for suspected pulmonary embolism rate CTPA or ventilation-perfusion imaging as usually appropriate initial imaging - This clinical decision support might have prevented a negative chest-radiograph interpretation from becoming a false rule-out

Failure #5: No Real-Time Monitoring for Automation Bias

Hospital should have tracked: - Physician viewing times for X-rays before vs. after AI deployment - Diagnostic error rates (missed PE, missed pneumothorax, etc.) before vs. after AI - Correlation between AI false negatives and physician misses (Are physicians missing cases AI misses?)

This monitoring would have identified automation bias pattern BEFORE adverse events occurred

Failure #6: Vendor Misrepresentation

Vendor marketing: “Comprehensive chest X-ray analysis” with “92-94% sensitivity”

Reality: AI trained on specific findings (pneumonia, pneumothorax, edema), NOT comprehensive

Misrepresentation: Led physicians to believe AI detected all chest X-ray pathology

Vendor should have: - Clearly disclosed training objectives (what AI detects vs. does not detect) - Provided performance data specific to PE findings (sensitivity for Hampton’s hump, Westermark sign) - Warned that AI is not validated for PE detection

3. Who is liable for the missed PE diagnosis?

ED physician issues to examine:

Clinical review: - The presentation required explicit consideration of pulmonary embolism and use of a validated diagnostic pathway - Tachycardia, estrogen exposure, pleuritic pain, mild hypoxemia, and a positive D-dimer increased concern, but the scenario does not justify automatically assigning the subjective Wells criterion that PE is the most likely diagnosis - The appropriate next step depends on the documented pretest probability, assay interpretation, contraindications, and current guidance. A chest radiograph cannot substitute for that pathway. - Management gap: The clinician did not document pretest probability or explain why additional PE imaging was deferred after the positive D-dimer

Plaintiff’s argument: - Potential deviation: The evaluation did not document a validated PE pathway or reconcile the positive D-dimer with the decision to discharge - Automation bias: You over-relied on AI “no abnormalities” output, reduced your independent clinical judgment - Inadequate X-ray review: 18-second viewing time insufficient for careful evaluation - Proximate cause: Missed PE diagnosis → delayed treatment → patient returned 15 hours later in worse condition (hypoxia, tachycardia, right heart strain) → could have had fatal PE

Defense arguments: - Busy ED: Workload provides context but does not establish that reliance on an out-of-scope AI output was reasonable - Subtle findings: Hampton’s hump and Westermark sign are easily missed even by experienced radiologists - AI false negative: The defense could argue that the output influenced a reasonable but mistaken interpretation, although authorization for specified findings does not make reliance reasonable for an out-of-scope diagnosis - Good outcome: Patient ultimately diagnosed, treated, survived without permanent harm

Counter to defense: - Professional duty: The clinician still had to use an appropriate diagnostic pathway and reconcile discordant evidence - AI is adjunct, not replacement: The fictional labeling did not authorize the system to exclude pulmonary embolism - Clinical presentation and pathway: The positive D-dimer and other findings required documented follow-through according to pretest probability and current guidance, even if the chest radiograph appeared normal - Automation bias is foreseeable: Hospital trained you to use AI, should have warned about over-reliance

Outcome boundary: - Liability would depend on jurisdiction, expert evidence, institutional policy, documentation, causation, product warnings, and damages - The exercise cannot predict a verdict, allocation of fault, or settlement range - Automation bias is relevant to system design and human-factors analysis, but it does not by itself decide a legal claim

Hospital-related issues:

Corporate negligence arguments: - Inadequate AI training: Did not warn physicians about AI limitations (no PE detection) - Poor workflow design: AI output shown before physician review → predictable automation bias - Failed to monitor for automation bias: No tracking of physician viewing times, diagnostic error rates post-AI deployment - Created dangerous environment: Deployed AI that increased automation bias risk without adequate safeguards

Legal boundary: - Vicarious liability, institutional negligence, employment status, indemnification, and insurance coverage vary by jurisdiction and contract - The safer conclusion is that the hospital’s selection, training, interface, monitoring, and escalation design require independent review

AI vendor issues:

Product liability arguments: - Failure to warn: Did not adequately disclose AI was not trained for PE detection - Misrepresentation: Marketed as “comprehensive chest X-ray analysis” but was not comprehensive - Design defect: AI should have included PE findings in training objectives

Vendor defenses: - FDA authorization: Clearance for an intended use establishes a regulatory marketing pathway, not clinical utility for PE, freedom from error, or immunity from civil claims - Intended use: AI marketed as adjunct/second reader, not replacement for physician judgment - No direct patient relationship: Vendor sells to hospital; physician responsible for appropriate use

Outcome boundary: - Vendor exposure cannot be inferred without the actual labeling, claims, warnings, contract, defect theory, governing law, and causal evidence - FDA authorization and user responsibility are relevant facts, not automatic defenses - If promotional claims exceeded the authorized intended use or obscured known limitations, the institution should preserve those materials and seek regulatory and legal review

4. How can automation bias be mitigated in clinical practice?

Strategy #1: Workflow Design to Preserve Independent Judgment

Problem: Seeing AI output first creates anchoring bias

Solution - “Independent First” Workflow: 1. Physician reviews image FIRST without AI output visible 2. Physician documents preliminary interpretation in EHR (“Preliminary read: No acute abnormalities” or “Preliminary read: Possible opacity RLL, DDx pneumonia vs. infarct”) 3. THEN physician views AI output 4. Compare physician interpretation to AI: - Agreement: Move forward - Disagreement: Investigate (“Why did I see finding and AI didn’t? Why did AI flag finding I didn’t see?”)

Benefit: Forces independent assessment before AI influence

Strategy #2: Training on Automation Bias Recognition

Include in all AI training: - Definition and examples of automation bias - Cognitive psychology: Why humans over-rely on automation - Real-world cases where automation bias led to errors (e.g., aviation, radiology) - Self-assessment: “Do I spend less time reviewing images when AI says ‘normal’?” (Probably yes → recognize this tendency)

Simulation training: - Present cases where AI is wrong (false positives, false negatives) - Practice detecting AI errors - Build cognitive habit: “Always question the AI”

Strategy #3: Transparency About AI Limitations

Require vendors to provide: - Specific training objectives (AI detects pneumonia, pneumothorax, edema but NOT PE, aortic dissection, subtle masses) - Failure modes (“AI struggles with subtle findings, overlapping structures, unusual presentations”) - Performance data by subgroup (performance in obese patients, elderly, portable X-rays)

Require hospitals to communicate: - Share limitations with all users in clear, concise format (one-page reference guide: “What AI Does/Doesn’t Detect”) - Periodic refresher training on limitations

Strategy #4: Clinical Decision Support Integration

Do not use AI imaging interpretation in isolation

Integrate with clinical risk scores: - Patient with chest pain and dyspnea → EHR prompts the clinician to document a validated pretest-probability pathway - When the documented pathway and D-dimer support additional imaging → Alert: “Pulmonary embolism remains under evaluation. A negative chest radiograph does not exclude PE; consider guideline-concordant imaging.”

Benefit: Clinical context overrides negative imaging when appropriate

Strategy #5: Real-Time Monitoring and Feedback

Track metrics that reveal automation bias: - Physician viewing times before vs. after AI deployment (Are physicians spending less time reviewing images? → automation bias) - Diagnostic concordance: How often physician agrees with AI? (>95% agreement → possible automation bias) - Error rates: Are physicians missing cases AI misses? (Correlation between AI false negatives and physician misses → automation bias)

Feedback to physicians: - Quarterly reports: “Your average X-ray viewing time: 15 seconds (department average: 22 seconds). Consider whether you’re adequately reviewing images independently.” - Case reviews when AI wrong and physician agreed: “Let’s discuss this case where both you and AI missed the finding. How can we avoid this in the future?”

Strategy #6: Promote Healthy Skepticism

Culture change: - “AI is tool, not truth” (AI makes suggestions, physicians decide) - “Trust but verify” (Use AI but always check independently) - “When in doubt, ignore AI and trust your clinical judgment”

Avoid language that promotes automation bias: - “AI read this as normal” - “AI didn’t flag abnormalities, but I’ll review independently”

Strategy #7: Limit AI Use in High-Stakes, High-Workload Settings

When automation bias risk is highest: - Busy ED overnight shifts (fatigue, high patient volume) - High-stakes cases (chest pain, trauma, altered mental status)

Consider: - Double-read by human: High-stakes cases get attending radiologist read in real-time (not just AI + ED physician) - Delayed AI output: In very high-stakes cases, do not show AI output until physician has documented preliminary interpretation

5. Key lessons on automation bias and AI:

Lesson #1: Automation Bias is Predictable and Preventable - Well-documented cognitive bias across multiple domains (aviation, radiology, anesthesia) - Occurs even in experienced, well-trained clinicians - Can be mitigated through workflow design, training, and monitoring

Lesson #2: AI as “Second Reader” Can Alter Diagnostic Accuracy - Intended benefit: AI catches findings physician misses - Potential harm: In some workflows, an unflagged finding can reduce human scrutiny through over-reliance - Net effect must be measured in the actual workflow, not inferred from standalone model accuracy

Lesson #3: Responsibility Is Workflow- and Jurisdiction-Specific - Clinicians retain professional duties when using assistive AI, while institutions, developers, and manufacturers can hold distinct duties - “AI said it was normal” does not resolve whether the evaluation was reasonable - Independent clinical judgment, appropriate escalation, and documentation remain central to safe use

Lesson #4: AI Limitations Must Be Transparently Disclosed - Vendors should clearly state what AI does/does not detect - Hospitals should communicate limitations to users - “Comprehensive” or “94% sensitive” marketing without specifics is misleading

Lesson #5: Workflow Design Matters - Showing AI output BEFORE physician review increases automation bias - “Independent first” workflow preserves physician judgment - Small design changes have large impact on cognitive biases

Lesson #6: Training on AI Should Include Failure Modes and Cognitive Biases - Technical training insufficient (how to use AI interface) - Must include: When AI fails, how to recognize over-reliance, strategies to maintain independent judgment - Simulation training with AI errors valuable

Lesson #7: Busy, High-Workload Environments Amplify Automation Bias - Fatigue, time pressure, cognitive overload increase reliance on automation - Extra safeguards needed for overnight ED, high-volume clinics - Do not assume AI will compensate for physician fatigue (may make it worse)

Lesson #8: Monitoring for Automation Bias Should Be Standard Practice - Track viewing times, diagnostic concordance, error patterns - Identify physicians at high risk for automation bias - Provide feedback and additional training