flowchart TD
A[Artificial Intelligence<br/>Broadest category: machines performing intelligent tasks] --> B[Machine Learning<br/>Systems that learn from data]
B --> C[Deep Learning<br/>Neural networks with many layers]
style A fill:#fee2e2,stroke:#dc2626,stroke-width:2px
style B fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style C fill:#dbeafe,stroke:#2563eb,stroke-width:2px
AI Fundamentals for Clinicians
Clinical AI claims often begin with a single performance number, such as 95% accuracy, while omitting the population, prevalence, threshold, comparator, and workflow. Evaluation requires understanding what the model learned, which task it performs, how the metric was calculated, and whether use of the configured system improves care. A model, an interface, and a clinical workflow are related but distinct parts of the intervention.
Learning Objectives
After reading this chapter, clinicians should be able to:
- Understand the difference between AI, machine learning, and deep learning
- Recognize the major types of machine learning relevant to medicine
- Grasp how neural networks learn from medical data
- Evaluate claims about AI performance using clinical metrics
- Identify when AI is appropriate (and when it is not) for clinical problems
- Understand the limitations and failure modes of medical AI systems
Introduction
Clinical AI includes learned models, rules, statistical scores, language systems, and combinations of software and workflow. Diabetic-retinopathy screening, sepsis prediction, and radiology decision support therefore cannot be evaluated through one generic definition of AI. Understanding what the configured system does, what data and rules it uses, and how its outputs enter care is increasingly important for clinical practice.
Current Adoption Landscape: Healthcare AI adoption lags behind other sectors but is accelerating. Approximately 8.3% of healthcare firms use AI in producing goods or services, compared with 11.6% in finance, 15.1% in education, and 23.2% in information services (Nguyen et al., 2025). Ambulatory care settings reported 8.7% adoption compared with 4.5% among nursing facilities. These firm-level figures measure organizational deployment; individual physician surveys measure a different unit and should not be directly compared. The AMA 2026 Physician Survey on Augmented Intelligence found that 81% of surveyed physicians reported using AI in their practices, more than double the 38% reported in 2023, with an average of 2.3 use cases (up from 1.1). More than three-quarters believed AI improved their ability to care for patients. Physicians prioritized safety and efficacy validation (88%) and data privacy (86%) as prerequisites for broader adoption, and 70% viewed AI as a burnout-reduction tool (AMA, March 2026). A separate vendor-sponsored Athenahealth survey of 1,045 physicians reported that 54% were comfortable with AI in practice, 38% used AI for documentation, and 30% believed AI would complicate healthcare, compared with 42% in its 2024 survey (Athenahealth, March 2026). For a guide to major reports and their methodological differences, see Industry Reports & Benchmarks.
The Central Concept:
Traditional medical software follows explicit rules programmed by humans:
IF temperature > 38°C AND WBC > 12,000 AND systolic BP < 90
THEN flag for sepsis evaluationMachine learning systems instead learn patterns from data:
Show algorithm 50,000 patient cases labeled "developed sepsis" or "no sepsis"
Algorithm identifies patterns (subtle vital sign trends, lab trajectories, timing relationships)
Apply learned patterns to predict sepsis risk in new patientsThis distinction explains both the value of learning-based systems (representing complex patterns) and their risks (learning shortcuts, lacking faithful explanations, and changing performance on unfamiliar cases).
Clinical and AI Terminology
If you trained in evidence-based medicine, you already understand most AI concepts under different names. This translation table maps familiar clinical research terminology to machine learning equivalents:
| Clinical Research / EBM | Machine Learning | Notes |
|---|---|---|
| Predictor, covariate, independent variable | Feature | Identical concept |
| Outcome, dependent variable | Label, target | What the model predicts |
| Model fitting, estimation | Training | Same process, different name |
| Internal validity | Internal validation | Performance estimated within the development data source under a prespecified split or resampling design |
| External validity | External validation | Performance on new populations |
| Confounding | Confounding or shortcut learning | Predictive models may exploit associations that do not support causal interpretation |
| Selection bias | Training data bias | Who’s included shapes what’s learned |
| Effect modification | Subgroup performance variation | Different results in different populations |
| Prospective study | Prospective evaluation | Design and comparator determine whether it measures workflow, diagnostic, or patient outcomes |
| Sample size | Development and evaluation sample | Adequacy depends on prevalence, labels, task, operating point, and precision |
| Overfitting a regression | Overfitting | Model captures noise, not signal |
The key insight: You already have the mental models for critical AI evaluation. The vocabulary is new; the concepts are not.
AI, Machine Learning, and Deep Learning: A Hierarchy
These terms are often used interchangeably but have distinct meanings:
Artificial Intelligence (AI)
The broadest category: Any computer system performing tasks that typically require human intelligence.
Medical examples: - Expert systems like MYCIN (rule-based, 1970s) - Machine learning algorithms (data-driven, 1990s-present) - Natural language processing for clinical notes - Learning-enabled functions within some surgical systems (teleoperation alone is not AI) - Clinical decision support systems
Key point: Not all AI is machine learning. MYCIN used hand-coded rules, not learning from data.
Machine Learning (ML)
Narrower category: Algorithms that learn patterns from data rather than following explicit rules.
Medical examples: - Predicting which patients will develop sepsis based on EHR data - Classifying skin lesions as benign or malignant from images - Extracting structured information from clinical notes - Forecasting disease progression from longitudinal data
Why ML matters for medicine: Medical data is too complex and nuanced for rule-based systems. ML can find subtle patterns across thousands of variables that no human could code explicitly.
Deep Learning (DL)
Narrowest category: Machine learning using artificial neural networks with many layers (hence “deep”).
Medical examples: - Detecting diabetic retinopathy from retinal fundus photographs Gulshan et al., 2016 - Identifying pneumonia on chest X-rays Rajpurkar et al., 2017 - Analyzing pathology slides for cancer detection Nagpal et al., 2019 - Generating radiology reports from images
Why DL changed medical imaging: Deep networks can learn complex features directly from pixels without requiring every feature to be specified by hand. On selected imaging tasks and datasets, this produced large performance gains, but benchmark performance does not establish expert-level care across settings.
The clinical reality: Deep learning is prominent in imaging, pathology, waveform analysis, and language applications, while many clinical prediction systems use regression, gradient-boosted trees, rules, or other methods. Understanding the model family matters because its data requirements, failure modes, and validation needs differ.
Machine Learning Fundamentals: A Clinical Analogy
Think about how you learned to diagnose pneumonia:
- Training: You saw hundreds of chest X-rays during residency, with attending physicians pointing out infiltrates, consolidations, effusions
- Pattern recognition: Your brain learned to recognize visual patterns associated with pneumonia
- Generalization: You can now diagnose pneumonia in new patients you’ve never seen
- Refinement: You get better with experience, especially on edge cases
Machine learning works similarly:
- Training Data: Algorithm sees thousands of chest X-rays labeled “pneumonia” or “normal” by radiologists
- Pattern Learning: Algorithm learns visual features associated with pneumonia (infiltrates, consolidation patterns, location preferences)
- Prediction: Algorithm can classify new chest X-rays it’s never seen
- Optimization: Algorithm improves by adjusting how much weight it gives different features
The key difference: Physicians can communicate clinical reasoning (“I see right lower lobe consolidation with air bronchograms”). A deep model’s output does not inherently provide a faithful, clinically meaningful explanation of the computation that produced it (Rudin, 2019).
Types of Machine Learning Relevant to Medicine
Supervised Learning (Most Common in Medicine)
What it is: Algorithm learns from labeled examples.
Medical applications: - Classification: Diagnosis tasks with categorical outputs (benign vs. malignant, pneumonia vs. normal, sepsis vs. no sepsis) - Regression: Predicting continuous outcomes (estimated survival time, predicted blood pressure, risk scores 0-100%)
Requirements: - A labeled sample adequate for the task, prevalence, model complexity, target precision, and intended subgroups - High-quality labels (accurate diagnoses from expert physicians) - Labeled examples representing the diversity of real clinical cases
Limitations: - Label quality matters enormously: If training labels are inaccurate or biased, algorithm learns inaccurate/biased patterns - Generalization is uncertain outside the development distribution: A model developed without pediatric data should not be assumed to perform adequately in children without relevant evaluation - Expensive: Labeling requires physician time
Example: Diabetic Retinopathy AI:
- Training data: 128,000 retinal images labeled by ophthalmologists as “no DR,” “mild DR,” “moderate DR,” “severe DR,” “proliferative DR” Gulshan et al., 2016
- Learning: Algorithm identifies patterns (microaneurysms, hemorrhages, exudates, neovascularization) associated with each severity level
- Deployment: Can classify new retinal images into appropriate categories
- Performance: On one prespecified validation set, the model reported 97.5% sensitivity and 93.4% specificity for referable diabetic retinopathy; the second validation set produced different operating characteristics (Gulshan et al., 2016)
Prediction vs. Inference: A Critical Distinction
Regression coefficients are associational unless a causal question, identification assumptions, design, and analysis justify a causal interpretation. Statistical adjustment does not automatically control all confounding.
In machine learning, model weights are usually prediction parameters, not causal estimates. A sepsis prediction model might weight “time since last antibiotic order” heavily because it correlates with clinical suspicion, not because delaying antibiotics causes sepsis. Multicollinearity can make coefficient estimates unstable and imprecise, but it does not by itself determine whether causal identification is valid.
Key implications for physicians:
- ML finds correlations, not causes. A model predicting readmission identifies correlated factors, but changing those factors may not reduce readmissions
- Confounding is not automatically addressed. ML may use confounders as predictive features rather than controlling for them
- For causal questions (“Does X cause Y?”), use a design and method appropriate to the estimand and assumptions. Randomization is strongest when feasible; observational approaches require explicit causal identification, not merely a named adjustment technique
If you need to explain why something happens, use epidemiological methods. If you need to predict what will happen, ML often excels.
Unsupervised Learning (Less Common Clinically)
What it is: Algorithm finds patterns in unlabeled data.
Medical applications: - Clustering: Identifying patient subgroups with similar characteristics (disease phenotypes) - Dimensionality reduction: Simplifying complex multi-omic data for interpretation - Anomaly detection: Flagging unusual patients who do not fit typical patterns
Limitations: - Hard to validate (no ground truth labels) - Clinical utility often unclear - Requires careful interpretation
Example: Disease Phenotyping:
Unsupervised clustering of EHR data might identify distinct COPD subtypes based on symptoms, comorbidities, medication responses. These subtypes could be more clinically meaningful than traditional classifications.
Reinforcement Learning (Emerging in Medicine)
What it is: Algorithm learns by trial-and-error, receiving rewards for good actions and penalties for bad ones.
Medical applications (mostly research): - Optimizing treatment strategies for chronic diseases - Personalizing chemotherapy dosing - Controlling mechanical ventilation
Limitations: - Online exploration on patients can be unsafe or unethical, so many systems rely on retrospective, simulated, or tightly constrained learning - Reward functions can encode the wrong objective or reproduce historical treatment patterns - Off-policy evaluation, confounding, distribution shift, and prospective validation remain difficult
Why it matters: Reinforcement learning has demonstrated selected medical control tasks, including a 2026 randomized trial of automated drug infusion during low-risk endoscopy, but most proposed treatment policies remain retrospective or simulated. Evidence must distinguish an evaluated controller from broader claims of autonomous treatment optimization.
Neural Networks: The Engine Behind Deep Learning
What is a Neural Network?
A neural network is a machine learning model loosely inspired by biological neurons, consisting of layers of interconnected nodes that transform inputs into outputs.
Clinical analogy: Think of diagnostic reasoning as information flow:
Patient symptoms → Your brain processes information → Differential diagnosisNeural networks work similarly:
Input data → Hidden layers process information → Output predictionThe “deep” in deep learning: Multiple hidden layers allow learning complex, hierarchical patterns.
flowchart LR
subgraph Input["Input Layer"]
I1[Symptom 1]
I2[Lab Value 1]
I3[Imaging Feature 1]
I4[Vital Sign 1]
end
subgraph Hidden["Hidden Layers<br/>(Process Information)"]
H1[Node]
H2[Node]
H3[Node]
H4[Node]
H5[Node]
H6[Node]
end
subgraph Output["Output Layer"]
O1[Disease A Probability]
O2[Disease B Probability]
O3[No Disease Probability]
end
I1 --> H1 & H2 & H3
I2 --> H1 & H2 & H3 & H4
I3 --> H3 & H4 & H5 & H6
I4 --> H4 & H5 & H6
H1 & H2 & H3 --> O1 & O2 & O3
H4 & H5 & H6 --> O1 & O2 & O3
style Input fill:#dbeafe,stroke:#2563eb
style Hidden fill:#fef3c7,stroke:#f59e0b
style Output fill:#d1fae5,stroke:#10b981
How Neural Networks Learn
Training process:
- Initialize: Start with random weights (connections between nodes)
- Forward pass: Input data flows through network, producing prediction
- Calculate error: Compare prediction to actual label (ground truth)
- Backward pass: Adjust weights to reduce error (backpropagation)
- Repeat: Iterate through thousands/millions of examples until error minimizes
Clinical translation:
Like a radiology resident reviewing cases with an attending: - Resident makes diagnosis (forward pass) - Attending provides correct answer (ground truth label) - Resident learns from mistakes (backpropagation) - Resident improves with practice (iterative learning)
Key difference: Neural networks can process large datasets and learn representations that were not manually specified. Whether those representations are clinically meaningful, transportable, or superior to a simpler model remains an empirical question.
Convolutional Neural Networks (CNNs): For Medical Imaging
Special architecture for images:
- Convolutional layers: Detect local patterns (edges, textures, shapes) before combining them into complex features
- Hierarchical learning: Early layers learn simple features (edges), deep layers learn complex features (anatomical structures)
- Translation invariance: Can detect findings regardless of location in image
Why CNNs revolutionized medical imaging:
Traditional computer vision required manually designing features (“look for round objects with size 2-5mm with density >X”). CNNs learn features automatically from raw pixels, discovering patterns human programmers would not have designed.
Medical applications: - Chest X-ray interpretation - CT/MRI analysis - Pathology slide review - Retinal imaging - Dermatology photo classification
Recurrent Neural Networks (RNNs): For Sequential Data
Special architecture for time-series data:
- Memory: Maintains information about previous inputs
- Sequential processing: Processes data in order (like reading a clinical note sentence by sentence)
Medical applications: - Analyzing EHR data over time (predicting deterioration from vital sign trends) - Processing clinical notes (understanding context and relationships) - Forecasting disease progression
Modern variant: Transformers:
Power large language models like GPT-4, Med-PaLM. Better than RNNs at capturing long-range dependencies and parallel processing.
Identifying AI Types: A Quick Reference
When vendors describe their AI system, use this guide to understand what architecture they’re likely using and what questions to ask:
flowchart TD
A[What type of data does the AI analyze?] --> B[Medical Images<br/>X-rays, CT, MRI, pathology]
A --> C[Sequential/Time-Series<br/>Vital sign trends, EHR trajectories]
A --> D[Clinical Text<br/>Notes, reports, literature]
A --> E[Tabular Data<br/>Demographics, labs, structured EHR]
B --> F[CNNs / Deep Learning<br/>Ask about: training data diversity, edge cases]
C --> G[RNNs / Transformers<br/>Ask about: temporal validation, data leakage]
D --> H[LLMs / Transformers<br/>Ask about: hallucination rates, grounding]
E --> I[Gradient Boosting / Logistic Regression<br/>Ask about: feature importance, interpretability]
style A fill:#f3f4f6,stroke:#6b7280,stroke-width:2px
style F fill:#dbeafe,stroke:#2563eb
style G fill:#fef3c7,stroke:#f59e0b
style H fill:#d1fae5,stroke:#10b981
style I fill:#fee2e2,stroke:#dc2626
Why this matters: Deep learning is not automatically the best choice for tabular risk prediction. Gradient-boosted trees are a strong benchmark for many medium-sized tabular datasets, but model ranking depends on representation, sample, tuning, missingness, and validation design. A conference-supplement study of antimicrobial-resistance prediction in sepsis reported AUROC 0.85 for deep learning on structured EHR data and 0.74 for an LLM using notes, with no measured benefit from combining them (Hixon et al., 2026). Architecture should be selected by comparative evidence on the intended task, not by prestige or novelty.
Evaluating AI Performance: Clinical Metrics
Vendors tout impressive performance metrics. How do you evaluate them critically?
Confusion Matrix: The Foundation
Every binary classification task (disease vs. no disease) can be summarized:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actually Positive | True Positive (TP) | False Negative (FN) |
| Actually Negative | False Positive (FP) | True Negative (TN) |
From this come all common metrics:
Sensitivity (Recall, True Positive Rate)
\[\text{Sensitivity} = \frac{TP}{TP + FN}\]
Clinical meaning: Of all actual cases of disease, what % did AI detect?
When it matters most: Screening (cancer detection), rule-out tests, situations where missing a case is dangerous
Example: Sepsis prediction with 85% sensitivity misses 15% of patients who develop sepsis
Specificity (True Negative Rate)
\[\text{Specificity} = \frac{TN}{TN + FP}\]
Clinical meaning: Of all patients without disease, what % did AI correctly identify as negative?
When it matters most: Avoiding false alarms, situations where false positives cause harm (unnecessary biopsies, psychological distress, treatment side effects)
Example: Sepsis prediction with 70% specificity incorrectly flags 30% of patients who do not have sepsis
Positive Predictive Value (PPV, Precision)
\[\text{PPV} = \frac{TP}{TP + FP}\]
Clinical meaning: If AI says “positive,” what’s the probability it’s actually correct?
Why it matters enormously: PPV depends on disease prevalence. A test with excellent sensitivity/specificity can have poor PPV if disease is rare.
Example: - Disease prevalence: 1% - Sensitivity: 95% - Specificity: 95% - PPV: Only 16%! (84% of “positive” predictions are false alarms)
Critical for physicians: Ask for the PPV at the intended threshold and in a population with prevalence comparable to the local setting, not just overall accuracy.
Negative Predictive Value (NPV)
\[\text{NPV} = \frac{TN}{TN + FN}\]
Clinical meaning: If AI says “negative,” what’s the probability it’s actually correct?
Accuracy
\[\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}\]
Clinical meaning: Overall, what % of predictions are correct?
Warning: Misleading for imbalanced datasets. An AI that always predicts “no cancer” achieves 99% accuracy if cancer prevalence is 1%, but is clinically useless.
AUC-ROC (Area Under Receiver Operating Characteristic Curve)
What it measures: Overall discrimination ability across all possible thresholds
Range: 0.5 (no better than chance) to 1.0 (perfect discrimination)
Interpretation: No universal labels such as “excellent” or “poor” determine clinical utility. The relevant judgment depends on the comparator, confidence intervals, operating threshold, prevalence, calibration, consequences of error, and feasible alternative.
Limitations: - Does not tell you performance at specific clinical threshold - Can be high even when PPV is poor at relevant prevalence - Does not capture calibration (are probabilities accurate?)
Calibration
What it measures: Do predicted probabilities match observed frequencies?
Example: If AI predicts “30% risk of mortality” for 1000 patients, do ~300 actually die?
Why it matters: Poorly calibrated models produce misleading probabilities, making clinical decision-making difficult.
How to assess: Calibration plots comparing predicted vs. observed outcomes
Vendor Requirements and Red Flags
- Evidence matched to the claim: External validation for transportability, prospective workflow evaluation for use in practice, and comparative outcome studies for patient-benefit claims
- Performance metrics at the intended threshold and local prevalence
- Subgroup analysis: Does performance differ by age, sex, race, insurance status?
- Calibration assessment: Are probabilities accurate?
- Failure mode documentation: When and how does the system fail?
- Independent evaluation: Prefer evidence produced outside the development team, with accessible methods and prespecified endpoints
Translating Vendor Claims
When evaluating AI products, vendor marketing language often obscures rather than clarifies. This translation table helps interpret common claims:
| Vendor Says | What It Actually Means |
|---|---|
| “AI-powered” | Uses some form of pattern recognition (could be simple or sophisticated) |
| “Clinically validated” | An undefined phrase until the design, population, comparator, version, and endpoint are supplied |
| “FDA-cleared” | A specific device was found substantially equivalent to a predicate for its labeled intended use; this is not a general efficacy guarantee |
| “99% accuracy” | On their test set, with their prevalence, under their conditions |
| “Proven in 50+ hospitals” | Deployed, but performance at each site may vary dramatically |
| “Real-time predictions” | Fast output, but speed says nothing about accuracy |
| “Explainable AI” | Provides some rationale, but explanations may not match actual reasoning |
| “Trained on millions of cases” | Large dataset, but representativeness matters more than size |
| “Outperforms physicians” | In one narrow task, under controlled conditions, often retrospectively |
| “Continuously learning” | Verify whether the marketed model actually changes after deployment and how updates are authorized, validated, versioned, and monitored |
Marketing language should be translated into a version-specific, testable claim. Request the labeling, peer-reviewed evidence, independent evaluation, threshold-specific results, subgroup analysis, and update history.
When to Use AI (and When Not To)
AI Works Well When:
Well-defined task with clear output: Binary classification, risk score, structured prediction
Adequate, high-quality labeled data are available: Sample requirements depend on the task, prevalence, model complexity, intended subgroups, and precision needed at the operating threshold
Pattern recognition from complex data: Images, longitudinal data, multi-dimensional inputs
A credible reference standard exists: Pathology, longitudinal outcomes, expert adjudication, or another prespecified standard can support evaluation, while still carrying measurement error and disagreement
Task is repetitive and time-consuming: Screening, triage, routine interpretation
Human performance has known limitations: Fatigue, variability, rare findings
Examples: - Diabetic retinopathy screening from retinal photos - Detecting pneumothorax on chest X-rays - Identifying metastases in pathology slides - Predicting no-show appointments - Extracting structured data from clinical notes
AI Struggles When:
Task requires general medical reasoning: Synthesizing information across domains, considering patient preferences, navigating uncertainty
Training data is limited or biased: Rare diseases, underrepresented populations, novel clinical scenarios
Ground truth is subjective or uncertain: Ambiguous diagnoses, prognosis depends on unmeasured factors
Explanation is essential: Medical-legal situations, teaching, situations requiring physician buy-in
Errors cannot be safely contained: High-consequence decisions without effective human oversight, fallback, or recovery procedures
Examples: - Diagnosing complex multi-system diseases - Navigating goals-of-care discussions - Handling completely novel presentations - Replacing physician judgment entirely
Common Failure Modes and Limitations
Distribution Shift (External Validity)
The problem: AI trained on Hospital A’s data may perform differently at Hospital B
Why: Different patient demographics, imaging equipment, clinical documentation, disease prevalence, treatment protocols
Example: Chest X-ray AI trained on data from academic medical centers performed poorly at community hospitals due to different patient populations and equipment Zech et al., 2018
What physicians should do: Require evidence relevant to the intended population, site, equipment, threshold, and workflow.
Overfitting (Memorization vs. Learning)
The problem: Algorithm memorizes training data instead of learning generalizable patterns
Why: Too complex model + too little data = memorization
How to detect: Excellent performance on training data, poor performance on new data
What physicians should do: Ask about validation strategy (hold-out test sets, cross-validation, external validation)
Confounding and Spurious Correlations
The problem: Algorithm learns correlations that do not represent causal relationships
Famous example: COVID-19 chest X-ray AI that learned to detect the word “portable” in image metadata (sicker patients get portable X-rays) rather than actual lung findings DeGrave et al., 2021
What physicians should do: Question HOW the AI makes predictions, not just WHETHER it’s accurate. Ask about confounding analyses.
Adversarial Attacks
The problem: Tiny, imperceptible changes to inputs can completely fool AI
Example: Adding noise invisible to human eyes can make AI misclassify malignant lesions as benign Finlayson et al., 2019
Clinical implications: Potential safety and security risks, especially for critical diagnoses
Algorithmic Bias
The problem: If training data under-represents certain populations, AI performs worse for those groups
Documented example: A widely used population-health algorithm used healthcare cost as a proxy for need. At the same risk score, Black patients were sicker than White patients, reducing the number of Black patients selected for additional care (Obermeyer et al., 2019).
Why it happens: Historical inequities → biased data → biased algorithms → perpetuate inequities
What physicians should do: Demand subgroup analyses by race, sex, age, insurance status. Question fairness metrics.
Label Leakage
The problem: Algorithm uses information that would not be available at prediction time, or learns from clinician responses rather than patient physiology.
Example: A sepsis prediction model trained on EHR data might use antibiotic orders as input features. But antibiotics are ordered because clinicians suspect sepsis. The model is not predicting sepsis, it’s detecting that someone already suspected sepsis.
Why it’s dangerous: Model appears to predict early, but actually requires information that only exists after the clinical decision has been made.
What physicians should do: Ask vendors: “Walk me through exactly when each input feature becomes available relative to when the prediction is made.” If features include treatment orders, diagnostic tests ordered, or specialist consults, be suspicious.
Temporal Leakage
The problem: For time-series predictions (deterioration, readmission, mortality), using future information to predict past events.
Example: A 30-day readmission model trained on patients who were eventually readmitted, using data from their entire initial hospitalization, including the final discharge summary that mentions “patient at high risk for readmission.”
Why it’s dangerous: Retrospective performance looks excellent, but the model cannot access future data when deployed prospectively.
What physicians should do: Ask about temporal validation. The training/test split should respect time: train on earlier data, test on later data. Random shuffling of time-series data is a red flag.
Automation Bias
The problem: Clinicians over-rely on AI recommendations, even when their own judgment or other evidence contradicts it.
Evidence: Systematic review found automation bias increased risk of commission errors by 26% when using incorrect decision support compared to working without decision support Goddard et al., 2012.
Why it happens: AI systems are perceived as objective and tireless. Cognitive load makes it easier to accept recommendations than to critically evaluate them.
What physicians should do: Treat AI outputs as one input among many, not as authoritative answers. Develop workflows that require active engagement with AI recommendations, not passive acceptance.
The Black-Box Problem
Many modern deep learning systems do not provide a faithful, clinically meaningful account of how an output was produced. A model can expose features, saliency, or generated rationales without those artifacts representing its actual decision process.
Why it matters: - Clinical justification: The clinician still needs reasons grounded in patient data and accepted evidence - Trust calibration: Users may overtrust a fluent explanation or reject a valid output without understanding its limits - Safety: Post hoc explanations may fail to reveal shortcuts, missing inputs, or distribution shift - Teaching: Generated rationales require verification before use as educational explanations
Approaches to explainability: - Saliency maps: Highlight image regions influencing prediction (but often do not match clinical reasoning) - Attention mechanisms: Show which words/features model focused on - LIME/SHAP: Explain individual predictions (but computationally expensive, not always accurate)
Current reality: Explainability remains an active research area. Interpretability should be evaluated for the intended user and task, and an explanation interface should not be treated as proof of model validity.
What physicians should do: Use a workflow that makes verification, escalation, and override possible at the clinical consequence level.
Large Language Models: A Special Case
Terminology: LLM vs. LMM
Large Language Models (LLMs) accept text input and produce text output (ChatGPT, Claude). Large Multi-Modal Models (LMMs) accept multiple input types, including text, images, and audio, and can generate diverse outputs (GPT-4V, Gemini, Med-PaLM M). As medicine increasingly uses image + text models for radiology, pathology, and dermatology, the LMM distinction matters. WHO’s 2025 guidance uses “LMM” as the preferred term for these general-purpose foundation models (WHO, 2025).
What makes LLMs different: - Trained on massive text corpora (internet, books, journals) - Can perform diverse tasks without task-specific training (few-shot learning) - Generate human-like text (including medical documentation, patient education, literature summaries)
Medical applications: - Clinical documentation assistance - Literature synthesis - Patient question answering - Medical education - Clinical reasoning support (with caveats)
Critical limitations: - Unsupported output: A fluent response can contain fabricated or weakly supported statements. Language models estimate token sequences from learned patterns; that objective does not guarantee factuality, source fidelity, or calibrated uncertainty. Retrieval, tool use, and reasoning procedures can reduce some errors but do not eliminate the need for verification (WHO, 2025). - Pattern matching vs. reasoning: High benchmark scores may reflect pattern recognition rather than genuine clinical reasoning. When answer patterns are disrupted, LLM accuracy drops 9-38% depending on the model, with reasoning models the most resilient (Bedi et al., 2025) - Data-access boundaries: A model can use current data only when the configured system is authorized and technically connected to those sources - Accountability: Responsibility remains with clinicians, organizations, developers, and other actors according to their roles and applicable law - Privacy and security: Sending patient data to an external service requires approved data handling, access controls, contracts, and configuration
Detailed coverage: See Large Language Models in Clinical Practice
AI Agents: From Chatbots to Autonomous Systems
Traditional AI tools respond to single queries: you ask a question, the system provides an answer. AI agents go further. They combine large language models with the ability to use tools, execute code, search databases, call APIs, and chain multiple actions together to accomplish complex goals with varying degrees of autonomy.
What distinguishes agents from chatbots:
| Characteristic | Chatbot/LLM | AI Agent |
|---|---|---|
| Input/Output | Single question → single answer | Goal → multi-step plan → execution |
| Tool use | None | Web search, code execution, API calls, database queries |
| Memory | Limited context window | Can maintain persistent memory across sessions |
| Autonomy | Responds to prompts | Can act independently on goals |
| Iteration | One-shot response | Plans, acts, observes results, adjusts approach |
How agents work in clinical contexts:
- Goal specification: Clinician defines a task (“Find recent trials for this patient’s cancer type and stage”)
- Planning: Agent breaks the task into steps (search clinical trial databases, filter by eligibility, rank by relevance)
- Tool use: Agent executes searches, reads results, processes information
- Iteration: Agent refines approach based on what it finds
- Output: Structured recommendation with sources
Current clinical applications:
- Oncology: Multi-agent systems for drug discovery, treatment planning, and clinical trial matching. In a preliminary validation of 20 simulated multimodal cases, an autonomous oncology agent reached correct clinical conclusions in 91% of cases and accurately cited guidelines in 75.5%, though evidence remains limited to small case series without prospective RCT validation (Ferber et al., 2025; Truhn et al., 2026)
- Prescription-renewal pilot: Utah authorized a temporary, conditional Doctronic regulatory-mitigation pilot for renewal of previously prescribed, noncontrolled medications. In Phase 1, every recommended renewal requires licensed-practitioner authorization, and the state reports that robust benefit evidence is not yet available (Utah Office of AI Policy, 2026). This is not general autonomous prescribing authority.
- Prior authorization: Agents navigating payer requirements and compiling documentation
- EHR tasks: Stanford’s MedAgentBench found Claude 3.5 Sonnet achieved 69.67% success on 300 clinical EHR tasks (Jiang et al., 2025)
Critical safety considerations:
Agents amplify both capabilities and risks. When an LLM hallucinates, it generates incorrect text. When an agent hallucinates, it may take incorrect actions: submitting wrong orders, searching incorrect databases, or iterating on flawed assumptions.
- Regulatory status is function-specific: Determine whether a software function is a device, non-device clinical decision support, or another regulated service, then verify the exact record and labeling. The FDA AI-enabled device list is not a list of general-purpose agents
- Hallucination risks compound across multi-step workflows
- Liability is fact- and jurisdiction-specific: Duties may attach to clinicians, institutions, developers, and other actors according to the workflow, representations, contracts, and applicable law
- Human oversight is essential for any patient-affecting decisions
The autonomy spectrum:
Not all agents are fully autonomous. Clinical agents range from:
- Supervised assistants: Generate drafts for human review (documentation, summaries)
- Bounded autonomy: Act independently within narrow constraints (scheduling, information retrieval)
- Full autonomy: Make and execute clinical decisions without human review (rare, regulatory barriers remain high)
Patient-affecting autonomy should be bounded by the evidence, authorization, reversibility, monitoring, and escalation pathway. Higher autonomy raises the required assurance level; a generic human-in-the-loop statement is not enough.
Rapid democratization of agent capabilities:
Open-source agent frameworks are accelerating access to systems that browse, call tools, store information, and act across services. OpenClaw is one example. Popularity counts and social-network registration figures are volatile and do not establish clinical validation, privacy safeguards, authorization, or safe handling of patient data. Clinicians may encounter patients using general-purpose agents for symptom research, medication reminders, or scheduling without a validated clinical workflow. The relevant question is what data and actions the configured agent can access, not how popular its framework is.
Detailed coverage: See AI Agents in Oncology and Multi-Agent AI Systems for specialty-specific applications and emerging developments.
Clinical Questions and Answers
What is the difference between AI, machine learning, and deep learning?
Artificial intelligence is the broad category of computer systems performing tasks associated with intelligence. Machine learning is a subset that estimates patterns from data, and deep learning is a subset of machine learning built from multilayer neural networks.
What is sensitivity vs specificity in medical AI?
Sensitivity is the proportion of reference-positive cases detected at a chosen threshold. Specificity is the proportion of reference-negative cases correctly classified. Both require confidence intervals, a defined reference standard, and the intended population.
Why does medical AI fail when deployed at different hospitals?
Performance can change across hospitals because patient mix, prevalence, equipment, acquisition, documentation, treatment, and workflow differ. External and local validation estimate transportability; monitoring detects change after deployment.
What is the black-box problem in medical AI?
Many deep learning models do not provide a faithful, clinically meaningful explanation of how an output was produced. Post hoc explanation tools can be useful, but they may be unstable or fail to represent the model’s actual decision process.
Can AI replace physicians?
AI can automate or support defined tasks, but capability depends on the model, data, workflow, and operating conditions. Patient preferences, uncertainty, accountability, and decisions outside the validated task remain clinician and health-system responsibilities.
What questions should physicians ask AI vendors?
Ask for the exact intended use and version, independent and local validation, threshold-specific sensitivity and specificity, predictive values at local prevalence, calibration, subgroup results, failure modes, update controls, and evidence that using the system improves the claimed workflow or patient outcome.
What are AI agents and how do they differ from chatbots?
AI agents combine a model with tools, state, and an execution loop to pursue multistep goals. Their autonomy varies from supervised drafting to bounded action, and every accessible tool expands both capability and risk.
How should physicians evaluate general-purpose AI agents used for health tasks?
General-purpose agents can search, call tools, store information, and take actions across services. Their popularity does not establish clinical validation, privacy safeguards, regulatory status, or safe use with patient data.
Key Takeaways for Physicians
Essential Concepts
Many AI systems learn from data, while others use rules or hybrid methods: Learning can represent complex patterns, but it can also encode shortcuts and produce outputs without faithful explanations
Many medical AI systems learn from labeled data: Deep learning is prominent in images, signals, and language, while regression, boosted trees, rules, and other methods remain important
Performance metrics are nuanced: Accuracy and AUC alone do not establish clinical utility. Ask about predictive values at local prevalence, calibration, subgroup performance, and the intended operating threshold
Black boxes require trust but limit understanding: Maintain human oversight, do not blindly follow unexplainable recommendations
Distribution shift is expected but variable: Performance can change across sites, time, equipment, prevalence, and workflows. Require relevant validation and monitoring
AI augments, does not replace: Think “AI-assisted physician” not “physician-less AI”
Bias requires measurement: Training data can encode inequities, but the direction and magnitude of harm must be evaluated for the target, subgroup, threshold, and intervention
Evidence should match the claim: Retrospective accuracy does not guarantee prospective utility, and prospective deployment without a comparator does not establish patient benefit
The bottom line:
You do not need to build neural networks to evaluate medical AI critically. You need to understand: - What AI can and cannot do - How to interpret performance metrics - What questions to ask vendors - What failure modes to watch for - When human oversight is essential
With these foundations, you’re prepared to evaluate AI tools for your specialty, covered in Part II.