Healthcare Policy and AI Governance

The FDA has cleared over 950 AI medical devices, 97% via the 510(k) substantial-equivalence pathway, typically without new prospective clinical trials. Epic’s widely deployed sepsis prediction model claimed 85% sensitivity in internal validation, but external testing revealed actual sensitivity of 33%, missing two-thirds of sepsis cases. Regulatory frameworks built for static medical devices struggle to govern AI systems that evolve through continuous learning. The EU AI Act classifies most clinical AI as high-risk requiring rigorous oversight, while U.S. regulation remains permissive. This divergence shapes what AI physicians can access and who bears liability when algorithms fail.

Learning Objectives

After reading this chapter, you will be able to:

  • Understand the evolving FDA regulatory framework for AI/ML-based medical devices
  • Evaluate international regulatory approaches (EU AI Act, WHO guidelines)
  • Recognize reimbursement challenges and evolving payment models for AI
  • Assess institutional governance frameworks for safe AI deployment
  • Navigate liability, accountability, and legal frameworks for medical AI
  • Implement hospital-level AI governance policies

The Regulatory Challenge:

Traditional medical device regulation assumes static products. A pacemaker cleared by FDA in 2020 is the same device in 2025. AI systems challenge this fundamentally: they learn, adapt, and evolve through retraining and algorithm updates. Regulators worldwide are adapting 20th-century frameworks to 21st-century learning systems.

Key Regulatory Developments:

  • FDA (U.S.): Over 950 AI/ML devices authorized as of August 2024, with 97% cleared via the less rigorous 510(k) pathway (FDA AI Device Database; MedTech Dive, 2024). The FDA’s Predetermined Change Control Plan (PCCP) framework allows pre-approved algorithm updates (FDA PCCP Guidance, 2024)
  • EU AI Act (2024): First comprehensive AI regulation globally. Medical AI classified as “high-risk” requiring transparency, human oversight, robustness testing, and bias audits (EU AI Act)
  • WHO: Published global guidance on AI ethics (2021, 2024), but guidelines are aspirational, not binding (WHO Ethics and Governance of AI for Health, 2021)
  • Reimbursement: Most AI lacks dedicated payment codes. IDx-DR diabetic retinopathy screening (CPT 92229) is a rare exception with Medicare coverage (CMS Medicare Physician Fee Schedule, 2022)

What Works and What Doesn’t:

The Epic sepsis model case illustrates regulatory failures: despite FDA 510(k) clearance, external validation at Michigan Medicine found sensitivity of only 33% (not 85% claimed), with 67% of sepsis cases never triggering an alert (Wong et al., JAMA Intern Med, 2021). FDA issued no recall or enforcement action.

The Clinical Bottom Line:

  • FDA clearance ≠ clinical validation. Demand prospective, external validation before adopting AI
  • If your institution lacks a multidisciplinary AI oversight committee, advocate for one
  • “AI told me to” is not a valid malpractice defense. Document AI recommendations and your rationale for following or overriding them

Essential Reading:

Introduction

Medicine operates within complex regulatory and policy frameworks: FDA device approvals, CMS reimbursement decisions, state medical board oversight, institutional protocols, and malpractice liability standards. These structures emerged over decades to protect patients from unsafe drugs, devices, and practices. They assume products are static: a drug approved in 2020 is chemically identical in 2025.

AI challenges this assumption fundamentally. Machine learning systems evolve through retraining on new data, algorithm updates, and performance drift as patient populations change. How should regulators approve systems that change continuously? Who is liable when AI errs: developers who built it, hospitals that deployed it, or physicians who followed its recommendations?

The stakes are high:

  • Patient safety: Poorly regulated AI can harm thousands before problems are detected
  • Innovation: Over-regulation may stifle beneficial AI development
  • Equity: Biased regulatory frameworks may entrench disparities
  • Legal liability: Unclear accountability creates defensive medicine

Part 1: The Major Policy Failure: Epic Sepsis Model

The Case Study

What was promised: Epic’s sepsis prediction model (embedded in EHR) would detect sepsis 6-12 hours before clinical recognition. Vendor claimed 85% sensitivity based on internal validation. FDA cleared the model via 510(k) pathway.

What happened:

In 2021, Wong et al. published an external validation study in JAMA Internal Medicine testing the Epic sepsis model on 27,697 patients at Michigan Medicine (Wong et al., 2021):

  • Sensitivity: 33% (not 85% claimed)
  • 67% of sepsis cases never triggered an alert at any point
  • Positive predictive value: 12% (88% false positive rate among alerts)
  • Area under the curve: 0.63 (poor discrimination)

Why FDA clearance failed to prevent this:

  1. Retrospective validation only: Epic’s 510(k) submission was based on retrospective chart review, not prospective deployment
  2. Look-ahead bias: Training data included labs and vitals ordered after clinicians suspected sepsis, so the model learned to detect suspicion, not actual sepsis
  3. No external validation requirement: FDA did not mandate testing at independent hospitals before clearance
  4. 510(k) predicate pathway: Model cleared as “substantially equivalent” to existing decision support without requiring clinical trial evidence

Regulatory response: FDA issued no recall, no warning letter, no enforcement action. The model remains FDA-cleared as of 2024.

Lessons:

  • FDA clearance ≠ clinical validation
  • Retrospective studies mislead due to look-ahead bias and confounding
  • External validation at independent institutions is essential
  • Post-market surveillance is inadequate

Part 2: FDA Regulation of AI/ML Medical Devices

Regulatory Pathways

510(k) Clearance (Substantial Equivalence):

  • Device is “substantially equivalent” to a predicate device already on market
  • Fastest, least burdensome pathway (median 151 days review time) (MDPI Biomedicines, 2024)
  • 97% of AI devices cleared via 510(k) pathway as of 2024 (MedTech Dive, 2024)

Premarket Approval (PMA):

  • Rigorous review requiring clinical trials demonstrating safety and effectiveness
  • Reserved for high-risk devices (median 372 days for De Novo pathway)
  • Example: IDx-DR diabetic retinopathy screening, the first autonomous AI diagnostic, received De Novo authorization in April 2018 (FDA De Novo Decision Summary)

De Novo Classification:

  • New device type with no predicate
  • Establishes new regulatory pathway for similar future devices

Current State (2024)

By the numbers:

  • 950+ AI/ML medical devices authorized as of August 2024 (MedTech Dive, 2024)
  • 168 devices cleared in 2024 alone, with 94.6% via 510(k) (MDPI Biomedicines, 2024)
  • Radiology dominates: 74.4% of 2024 clearances were imaging-related

Examples of FDA-cleared AI:

Category Examples
Radiology CAD Intracranial hemorrhage (Aidoc, Viz.ai), pulmonary embolism, lung nodules
Cardiology ECG AFib detection (Apple Watch, AliveCor), echocardiogram EF estimation
Ophthalmology IDx-DR/LumineticsCore diabetic retinopathy screening
Clinical Decision Support Sepsis prediction, deterioration algorithms

Predetermined Change Control Plans (PCCP)

Traditional devices are “locked” after approval. The FDA’s PCCP framework addresses this for AI systems that need continuous updates (FDA PCCP Guidance, 2024):

What PCCP allows:

  • Manufacturer specifies anticipated changes (retraining, performance improvements)
  • FDA reviews and approves plan upfront
  • Specified changes proceed without new submissions

Components required:

  1. Description of modifications: Itemization of proposed changes with justifications
  2. Modification protocol: Methods for developing, validating, and implementing changes
  3. Impact assessment: Benefits, risks, and mitigations

Final guidance issued December 2024 broadened scope to all AI-enabled devices, not just ML-enabled devices.

Challenges and Needed Reforms

Problem Evidence Needed Reform
No prospective validation required Epic sepsis model cleared with retrospective data, failed prospectively Mandate prospective validation for high-risk AI
Inadequate post-market surveillance FDA relies on voluntary adverse event reporting Require quarterly performance reports
Generalizability not assessed AI approved on one population may fail in others Require demographic subgroup analysis
Transparency vs. trade secrets Physicians cannot validate black-box AI Mandate disclosure of training data demographics

Part 3: International Regulatory Approaches

EU AI Act (2024)

The EU AI Act is the world’s first comprehensive AI regulation, entering into force August 1, 2024 (European Parliament, 2024).

Risk-based categorization:

Risk Level Requirements Medical AI Examples
Unacceptable (banned) Social scoring, subliminal manipulation Not applicable to medical AI
High risk Strict obligations All medical AI for diagnosis, treatment, or triage
Limited risk Transparency requirements Medical chatbots, symptom checkers
Minimal risk No specific obligations Not applicable to medical AI

High-risk medical AI requirements (npj Digital Medicine, 2024):

  1. Transparency: Disclose training data sources, demographics, limitations
  2. Human oversight: Physicians must retain decision authority and override
  3. Robustness testing: Independent validation across diverse populations
  4. Bias audits: Performance stratified by demographics
  5. Post-market monitoring: Continuous performance tracking, adverse event reporting within 15 days

Compliance timeline: Medical devices qualifying as high-risk AI systems have until August 2, 2027 for full compliance.

Impact: Estimated compliance cost of €500K-€2M per AI system. Small startups may struggle, potentially consolidating market toward large companies.

WHO Guidelines (2021, 2024)

WHO published Ethics and Governance of Artificial Intelligence for Health in June 2021 with six principles (WHO, 2021):

  1. Protect human autonomy: Patients and providers maintain decision-making authority
  2. Promote human well-being and safety: AI must benefit patients, minimize harm
  3. Ensure transparency and explainability: Stakeholders understand AI logic and limitations
  4. Foster responsibility and accountability: Clear assignment of responsibility when AI errs
  5. Ensure inclusiveness and equity: AI accessible to diverse populations, mitigate bias
  6. Promote responsive and sustainable AI: Long-term monitoring, adaptation to changing contexts

In 2024, WHO published additional guidance on large multi-modal models, warning of hallucinations, outdated information, and bias (WHO LMM Guidance, 2024).

Limitation: WHO guidelines are aspirational, not enforceable. Countries adopt them voluntarily.

Other Regions

Region Approach Key Characteristics
Canada (Health Canada) Collaborative Developing adaptive licensing with FDA, UK MHRA
UK (MHRA) Innovation-friendly Post-Brexit independent framework
Japan (PMDA) Conservative Extensive clinical data required
China (NMPA) Rapid approval Data localization requirements limit international collaboration

Part 4: Reimbursement and Payment Models

The Reimbursement Problem

Regulatory approval is necessary but insufficient. Reimbursement drives clinical deployment. If payers do not cover AI, providers will not use it.

Current landscape:

  • Most AI lacks dedicated CPT codes
  • Payers reluctant to cover without clear evidence of clinical benefit
  • Result: High-value AI sits unused; unproven AI deployed where hospitals self-fund

Success Story: IDx-DR Diabetic Retinopathy Screening

IDx-DR represents the rare AI reimbursement success (CMS, 2022; FDA De Novo, 2018):

Milestone Details
FDA authorization April 2018, De Novo pathway (first autonomous AI diagnostic)
CPT code 92229 established 2021 (imaging with AI interpretation without physician review)
Medicare coverage Yes, with CMS establishing national payment in 2022
Payment ~$40-55 per screening depending on carrier

Why it succeeded:

  • Clear clinical benefit (diabetic retinopathy screening reduces blindness)
  • Solves access problem (primary care can screen without ophthalmologist)
  • Cost-effective (cheaper than specialist visit)
  • Prospective validation (randomized trial evidence)

Most AI: Fragmented or No Coverage

AI Category Reimbursement Status
Radiology AI Usually no separate payment; cost incorporated into radiologist fee
Clinical decision support No payment; cost absorbed by hospitals
Ambient documentation Physician/institution subscription ($1,000-1,500/month)

Emerging Payment Models

Fee-for-service does not incentivize AI adoption. Value-based models align incentives:

Value-based care contracts: Providers share risk with payers. AI that reduces hospitalizations and complications directly benefits providers financially.

Bundled payments: Single payment for entire episode of care. AI costs included in bundle; providers incentivized to use cost-effective AI.

Outcomes-based contracts with vendors: Hospital pays vendor based on AI performance, not upfront license. Aligns incentives to reduce false positives.

Part 5: Institutional Governance

Why Hospital-Level Governance Matters

FDA clearance does not guarantee AI works at your hospital:

  • Different patient population, workflows, EHR
  • AI is cost-effective for YOUR budget
  • Physicians will use AI appropriately
  • Patients will not be harmed

Institutional governance fills gaps left by regulation.

Essential Governance Components

1. Clinical AI Governance Committee

Minimum composition:

  • Chair: CMIO or CMO
  • Physicians from specialties using AI
  • Chief Nursing Officer representative
  • CIO or IT director
  • Legal counsel with medical malpractice and AI expertise
  • Chief Quality/Patient Safety Officer
  • Health equity lead
  • Bioethicist
  • Patient advocate

Responsibilities: Pre-procurement review, pilot approval, deployment oversight, adverse event investigation, policy development, bias auditing.

2. Validation Before Deployment

Do not assume vendor validation generalizes to your hospital.

Phase Duration Purpose
Silent mode 2-4 weeks AI generates outputs not shown to clinicians; verify technical stability
Shadow mode 4-8 weeks AI outputs shown as “informational only”; gather physician feedback
Active pilot 3-6 months Limited deployment with pre-defined success criteria

Success criteria should include:

  • Technical: Sensitivity, specificity, PPV thresholds
  • Clinical: Primary outcome improvement vs. baseline
  • User: Physician satisfaction, response rate
  • Safety: Zero preventable patient harm
  • Equity: No performance disparities >10% across demographics

3. Bias Monitoring

Quarterly audits measuring AI performance across:

  • Race/ethnicity
  • Age
  • Sex
  • Insurance status
  • Language

If performance difference >10%: investigate, mitigate, or deactivate.

4. Vendor Contracts

Require:

  • Hospital retains ownership of patient data
  • Performance guarantees with termination rights if thresholds not met
  • Disclosure of training data demographics, validation studies, limitations
  • Vendor indemnification for AI errors and data breaches
  • HIPAA Business Associate Agreement

Part 7: Policy Recommendations from Expert Bodies

AMA Principles for Augmented Intelligence (2019)

The American Medical Association prefers “augmented intelligence” to emphasize physician judgment is augmented, not replaced (AMA AI Principles, 2019).

Six principles:

  1. AI should augment, not replace, the physician-patient relationship
  2. AI must be developed and deployed with transparency
  3. AI must meet rigorous standards of effectiveness
  4. AI must mitigate bias and promote health equity
  5. AI must protect patient privacy and data security
  6. Physicians must be educated on AI

WHO Framework (2021)

Six principles: protect human autonomy, promote well-being and safety, ensure transparency, foster accountability, ensure equity, promote sustainability (WHO, 2021).

Key Professional Society Positions

Society Key Recommendations
American College of Radiology AI-LAB accreditation program for vendor transparency; validate AI at your institution before clinical use
American Heart Association Cardiovascular AI must be validated on populations where deployed
College of American Pathologists Pathologists must review all AI-flagged cases; no autonomous AI diagnosis

Conclusion

AI regulation and policy are evolving rapidly. The frameworks designed for static products do not fit dynamic, learning systems that update continuously. Challenges include unclear evidence standards, insufficient post-market surveillance, reimbursement barriers, unsettled liability, and fragmented international regulations.

Key principles for physician-centered AI policy:

  1. Patient safety first: Prospective validation, external testing
  2. Evidence-based regulation: Demand prospective trials for high-risk AI
  3. Transparent accountability: Clear liability when AI errs
  4. Equity mandatory: Performance tested across demographics; biased AI not deployed
  5. Physician autonomy preserved: AI supports, never replaces judgment
  6. Reimbursement aligned with value: Pay for AI that improves outcomes

What physicians must do:

Individually: Demand evidence, validate AI locally, document AI use meticulously, report errors, maintain clinical independence.

Institutionally: Establish AI governance committees, implement bias audits, create accountability frameworks, provide training.

Professionally: Engage specialty societies, lobby for evidence-based regulation and reimbursement, publish validation studies.

The future of AI in medicine will be shaped by the choices made today: the regulations demanded, the reimbursement models advocated for, the governance structures built, and the standards held.

References