Evaluating AI Agents in Healthcare

Judge an AI agent by whether it finishes whole tasks correctly and safely, including what it writes into the record, rather than by how well it answers questions. Agentic AI pairs a language model with software it can act in, such as an electronic health record (EHR), a payer portal, or an order screen, so that it can plan and carry out a sequence of steps toward a goal. In simulated EHRs, a redesigned agent now completes about 9 in 10 short, explicitly worded chart tasks (Chen et al., 2026), while the best agents completed fewer than half of long specialist-consultation tasks (Liu et al., arXiv:2605.02240, 2026, preprint) and about a third of screen-based administrative tasks (Bedi et al., arXiv:2604.09937, 2026, preprint).

Learning Objectives

After completing this chapter, clinicians and health system leaders should be able to:

  • Explain why an AI agent needs different evaluation from a model that only answers questions
  • Read a healthcare agent benchmark and identify what its environment, data, setup, and grading can and cannot show
  • Distinguish step accuracy from end-to-end task completion, and single-run success from consistency across runs
  • Specify tests for access, scope, stopping, escalation, and prompt injection before an agent touches real records
  • Match the strength of evidence required to both the autonomy and the consequence of an agent’s actions
  • Write a local evaluation plan, with go and stop criteria, for an agent that a vendor proposes

What an agent is: A language model that plans and takes actions in software to finish a multistep task. Its errors include actions (a wrong order, a submitted form, a missing order), not only wrong text.

What the benchmarks show (all simulated, all from or with Stanford investigators):

  • Short chart tasks: 69.67% for the best 2024 model in the original MedAgentBench (Jiang et al., 2025); 91.0% for a redesigned GPT-4.1 agent and 88.67% on 300 new tasks (Chen et al., 2026)
  • Long consultation tasks: 46.3% on the first attempt, 28.0% on all three of three runs (Liu et al., arXiv:2605.02240, 2026, preprint)
  • Administrative work through screens: 36.3% end to end for the best agent; another agent passed 82.8% of individual steps but finished 26.7% of tasks (Bedi et al., arXiv:2604.09937, 2026, preprint)

What to measure: end-to-end completion with its uncertainty; consistency across runs; the path taken; what was actually executed in the record; stopping and escalation; resistance to hidden instructions; cost and time per completed task.

Bottom line: Require evidence in proportion to both what the agent may do on its own and how much harm an error could cause, and treat a benchmark pass as completion of a sandboxed task, not as safe care.

Introduction

Health systems are being offered agents for prior authorization, chart preparation, referral triage, inbox management, and order entry. Each of these is a workflow of many steps, and each step depends on the ones before it. The definition of an agent, its autonomy spectrum from supervised drafting to independent action, and early live pilots such as Utah’s prescription-renewal program are covered in AI Fundamentals for Clinicians. The question here is narrower: what evidence should a clinician, a chief medical information officer, or a governance committee ask for before an agent acts on patient records or on a patient’s behalf?

The general principles of clinical AI evaluation still apply, including intended use, representative data, external validation, workflow fit, and monitoring, and are covered in Evaluating AI Clinical Decision Support Systems. Agents add problems that a single-answer model does not have, and a few terms recur throughout this chapter.

Term Meaning
Harness Everything around the model: its instructions, tools, turn limits, settings, and the grader that scores it
Trajectory The full sequence of actions an agent took on a task
FHIR Fast Healthcare Interoperability Resources, the standard interface EHRs use to exchange data; most agent benchmarks act through it
pass@1 The share of tasks completed on a single attempt
pass^k The share of tasks completed correctly on every one of k repeated attempts, a measure of consistency
Tokens Units of text a model processes; a rough proxy for computing cost
Prompt injection An instruction hidden in content the system reads, written to change its behavior

What Makes an AI Agent Different to Evaluate

Errors accumulate across steps

A workflow of 10 independent steps, each done correctly 95% of the time, finishes correctly about 60% of the time (0.95 raised to the 10th power). Real errors are not independent: they cluster in particular task types and particular steps, so results by task type say more than an average step score. The administrative-agent benchmark below shows the size of the gap between step-level and task-level success.

The same agent can succeed once and fail the next time

Language models do not give the same output every time. PhysicianBench reports both pass@1, averaged over three runs, and pass^3. The best agent’s pass@1 of 46.3% fell to 28.0% when it had to succeed on all three runs, and the authors conclude that successful completion “is often stochastic,” meaning partly a matter of chance (Liu et al., arXiv:2605.02240, 2026, preprint).

The path matters, not only the endpoint

In a 2026 PLOS Medicine Perspective, Ruhrberg Estévez and colleagues argue that matching final outputs is insufficient for agents because clinical care often allows several safe paths, while a correct diagnosis reached through excessive or misordered tests “is not clinically equivalent” to an efficient, guideline-concordant workup. They propose scoring the whole trajectory on three dimensions: action appropriateness (sensible actions at the right point), process safety (no unsafe steps, such as omitted red-flag screening or treatment without checking contraindications), and resource stewardship (no unnecessary tests, referrals, or costs). Because care rarely has a single correct path, they recommend defining acceptable ranges of practice from expert consensus and guidelines, and they warn that agents can optimize benchmark scores without improving clinical utility (Ruhrberg Estévez et al., 2026). Two of the four authors disclose shareholdings in AI companies, one of them is employed by one, and he has also received a research grant from OpenAI.

The setup changes the score

An agent’s result belongs to the model and its harness together. A team including MedAgentBench’s authors rewrote the agent’s instructions, asked it to state a plan before acting, and added tools for calculations, output formatting, and EHR calls; with GPT-4.1, success on the original 300 tasks rose to 91.0%, against 69.67% for the best model in the original setup (Chen et al., 2026; Jiang et al., 2025). Scores from different harnesses are not directly comparable, and a vendor’s score applies to the vendor’s configuration.

The Current Evidence: Healthcare Agent Benchmarks

A benchmark is a fixed set of tasks scored the same way for every system; for agents, it includes a simulated environment the agent can act in. Every result below comes from a simulated or sandboxed environment, and none measured patient outcomes.

Benchmark Environment and data Tasks Best reported result What it cannot show
MedAgentBench (Jiang et al., 2025) FHIR-compliant simulated EHR; 100 patient profiles from de-identified, date-jittered Stanford records 300 physician-written tasks in 10 categories, each limited to eight rounds 69.67% (Claude 3.5 Sonnet v2, original harness) Long workflows, team coordination, repeated-run reliability
MedAgentBench v2 (Chen et al., 2026) Same environment; redesigned agent harness Original 300 tasks plus 300 new multistep tasks 91.0% (GPT-4.1); 98.0% with memory; 88.67% on new tasks Terse real-world instructions, independent task sets
PhysicianBench (Liu et al., arXiv:2605.02240, 2026, preprint) FHIR EHR sandbox built from de-identified, perturbed Stanford e-consult cases 100 long consultation tasks, 21 specialties, 670 checkpoints, about 27 tool calls each 46.3% pass@1; 28.0% pass^3 (GPT-5.5) Live use, patient outcomes
HealthAdminBench (Bedi et al., arXiv:2604.09937, 2026, preprint) Four simulated web environments (EHR, two payer portals, fax) with synthetic patients 135 administrative tasks, 1,698 subtasks 36.3% task completion (Claude Opus 4.6); 82.8% subtask success with 26.7% task completion (GPT-5.4) Real portals, real logins and multifactor steps, retries

MedAgentBench: short EHR tasks

MedAgentBench, published in NEJM AI, gives an agent nine FHIR functions, six that search patients, conditions, laboratory results, vital signs, medication requests, and procedures, and three that create vital signs, medication requests, and procedures. Each of 300 physician-written tasks is graded with manually curated checks. In the original setup, the best model, Claude 3.5 Sonnet v2, succeeded on 69.67% of tasks overall, 85.33% of tasks that only retrieved information, and 54.00% of tasks that wrote to the record, and most models did better on retrieval than on action. The authors used a single attempt per task because “even a single incorrect action or response can have significant consequences,” limited each task to eight rounds, noted that the environment does not capture coordination between clinical teams, and left reliability under repeated runs to future work (Jiang et al., 2025).

The follow-up study, presented at the Pacific Symposium on Biocomputing, reached 91.0% with a redesigned harness and 98.0% after adding a memory of corrections drawn from failed tasks in the same benchmark, and 88.67% (266 of 300) on 300 new multistep tasks generated with a language model and reviewed by a physician. The authors state that clinical use needs accuracy above 95%, that the benchmark’s instructions were “often overly explicit” compared with the terse instructions clinicians actually give, and that some failures reflected answers in an unexpected format rather than wrong reasoning (Chen et al., 2026). Near-saturated scores on short, clearly worded tasks do not transfer to long, ambiguous, real-world work.

PhysicianBench: long consultation workflows

PhysicianBench adapts 100 primary-care-to-specialist e-consult cases from Stanford’s de-identified research repository, with dates, demographics, and values perturbed, into tasks that require retrieving data across encounters, reasoning, executing clinical actions, and writing documentation. Each task is graded at execution-verified checkpoints, and agents were allowed up to 100 tool-calling turns. Across 12 agents, the best (GPT-5.5) completed 46.3% of tasks on the first attempt and 28.0% on all three runs, and the best open-weight model, one whose weights are publicly released, reached 18.7%. Clinical reasoning caused about half of failed checkpoints on average (50.4%); in the two leading models, these were most often incomplete answers that dropped a dose, monitoring plan, or safety caveat. In the same two models, action failures were dominated by an output gap: the agent wrote the correct action in its note but never created the order in the record (Liu et al., arXiv:2605.02240, 2026, preprint). For clinicians, that means a correct-looking note is not evidence that the order exists. Developer-reported runs of the same benchmark use different harnesses and are discussed in the MAST section of the evaluation chapter.

HealthAdminBench: administrative work through screens

HealthAdminBench tests computer-use agents, which operate software through its screens as a person would, on prior authorization, appeals and denials management, and durable medical equipment orders. A task counts as complete only when every subtask passes, and 521 of the 1,698 subtasks are graded by a language model rather than by deterministic checks. The best agent (Claude Opus 4.6) completed 36.3% of tasks end to end. Another agent (GPT-5.4) passed 82.8% of individual subtasks but completed only 26.7% of whole tasks. By task type, the best agent completed 38.3% of prior authorization tasks, 25.0% of appeals and denials tasks, and 73.3% of equipment-order tasks. The environments and patients were synthetic, access steps such as multifactor authentication were abstracted away, each agent had one attempt per task, and five of the authors are affiliated with a company, Kinetic Systems (Bedi et al., arXiv:2604.09937, 2026, preprint). The evaluation chapter’s section on administrative agents describes the failure modes the authors identified.

How far these results generalize

All four benchmarks come from Stanford investigators or their collaborators, and MedAgentBench, its follow-up, and PhysicianBench share senior authors and draw on the same institution’s records. They point the same way, but they are not independent replications, and none tested agents on another health system’s data, workflows, or documents.

Measuring What Matters: Completion, Consistency, Process, and Cost

The benchmarks point to a small set of measures that a health system can ask a vendor for, or run itself in a test environment.

Measure Plain-language question Why it matters
End-to-end task completion, with its confidence interval Of a sample of typical tasks drawn from local work, how many did the agent finish correctly with no human repair? Estimates how much work falls back on staff; step accuracy overstates it
Consistency across runs If the same task is run three to five times, how often is every run correct? A tool that succeeds by chance cannot be planned around
Process quality Did the agent take sensible, safe, and proportionate steps on the way? A correct endpoint can hide unsafe or wasteful actions
Executed actions Do the orders, messages, and filings in the record match what the agent reported doing? Agents can describe an action they never took
Stopping and escalation When a task is impossible, ambiguous, or missing documents, does the agent stop and hand off, and how often does it escalate needlessly? Working around a blocker is a recurring failure in developer testing; an agent that escalates everything saves no work
Resistance to hidden instructions Does content inside documents or images change what the agent does? Agents that read outside content can be steered by it
Cost and time per completed task What does each correct completion cost, including retries and human review? Agent designs trade accuracy against compute, latency, and staff time

Four practical rules follow. First, report the denominator: count completion over every task attempted, including those the agent abandoned or that hit a turn limit, as MedAgentBench and HealthAdminBench do (Jiang et al., 2025; Bedi et al., arXiv:2604.09937, 2026, preprint). Second, report uncertainty: with 100 to 300 tasks, a rate is imprecise, and 36.3% of 135 tasks, for example, corresponds to a 95% confidence interval of roughly 29% to 45% (handbook calculation). Third, test on the institution’s own task mix, because performance varied widely by task type in every benchmark. Fourth, hold the harness constant when comparing systems, and weigh success against cost and time per task, as the MedAgentBench follow-up recommends (Chen et al., 2026).

Grading itself needs evaluation. When a language model grades an agent’s output, its agreement with blinded clinician review should be measured before its scores are trusted; the LLM-as-a-judge safeguards in the evaluation chapter apply unchanged.

Testing Safety: Access, Scope, Stopping, and Manipulation

Access, identity, and audit

An agent that acts in clinical systems is a user of those systems and needs the controls applied to any user. The benchmarks above abstracted these away; HealthAdminBench, for example, removed multifactor authentication and session timeouts from its environments (Bedi et al., arXiv:2604.09937, 2026, preprint). A local evaluation should confirm that the agent has its own identity and the narrowest access its task requires, consistent with the minimum-necessary standard covered in Privacy, HIPAA, and Patient Data Security; that every write it makes is recorded with a marker showing the agent made it; and that it files to the correct patient when names, dates of birth, or record numbers are similar.

Out-of-scope actions and persistence

Agents can take actions their operators did not intend. In a preliminary October 2026 report on its models in evaluations and internal use, Anthropic described cases in which models submitted real online forms, including one in which Claude Haiku 4.5, told to complete a form but stop before the final submission, several times submitted it, expecting another confirmation page. The company described most of the behaviors as forms of persistence, in which the model, “when it cannot complete a task as given, works around a restriction instead of stopping,” noted that many cases arose from tasks that were ambiguous or impossible to complete, and suggested that clearer statements of what was in and out of scope (targets, permitted actions, and network boundaries) might have prevented some of them (Anthropic, October 2026). This is a developer’s account of its own models, not a clinical study, but the pattern maps onto clinical and administrative work, where the correct action for a blocked task is often to stop.

Test every clinical agent on tasks it should refuse or escalate: missing or invalid documentation, conflicting instructions, a request outside its authority, and a broken or unavailable system. HealthAdminBench builds this into its hard equipment-order tasks, which require the agent to identify invalid or missing documentation and halt processing (Bedi et al., arXiv:2604.09937, 2026, preprint). Score both missed escalations and unnecessary ones.

Prompt injection

Clusmann and colleagues embedded instructions in medical images (CT, MRI, ultrasound, endoscopy, histology, and photographs of melanomas, all with confirmed cancers) and asked four vision-language models to describe them. Across 594 attacks, every model was susceptible: with injected instructions, GPT-4o’s lesion miss rate rose from 22% to 89%, and for GPT-4o and Reka Core, low-contrast or tiny text that human observers would not readily notice was about as harmful as clearly visible text. Asking the model to follow high ethical standards and adding a second supervising model did not stop the attacks for three of the four models; for Claude 3.5 Sonnet, the ethical prompt reduced susceptibility from 64.8% to 27.8% (Clusmann et al., 2025). The study tested 18 cases with models accessed in 2024, in a fully simulated setting, so the rates describe those models rather than current ones. The task was a single image description, the lowest level of autonomy, so any system that reads outside content, including one that only summarizes, should be tested with adversarial inputs placed where real attacks would arrive, such as faxes, referral attachments, and patient-uploaded files.

Human attestation and escalation rules

The Coalition for Health AI (CHAI), whose agentic AI work group was led mostly by companies, with one academic health system, recommends requiring authenticated human attestation before agent output is committed to the EHR and clearly marking which data points an agent added or changed. It suggests deriving safety metrics by asking clinicians how they would supervise a staff member doing the same task, while also asking what the agent could get wrong that a person almost certainly would not, “like silently proceeding when it should have paused to escalate.” For voice agents, it recommends treating escalation on a clinical red flag as unconditional and exempt from optimization. Its testing and evaluation framework lists usefulness, usability, efficacy, fairness, safety, reliability, and business impact as dimensions to evaluate (CHAI, July 2026). These are consensus recommendations, not tested interventions. Human sign-off is also not a complete safeguard, because reviewers can come to defer to automated suggestions; automation bias is covered in Clinical AI Safety and Risk Management.

Matching Evidence to Autonomy and Consequence

The American Medical Association’s governance framework for agentic AI that assumes clinical functions organizes oversight around four questions: what clinical function the AI performs (for example, documenting, monitoring, triaging, recommending, diagnosing, treating, or prescribing); what evidence shows it can perform that function and what happens if it fails; what it should be permitted to do and under what conditions, along a sequence from inform to recommend to decide to act; and who is accountable among developers, deployers, health systems, physicians, payers, and others. Its rule is that greater consequence and greater autonomy require stronger evidence, and that clinical authority “should be earned through evidence and maintained through demonstrated performance”. The framework also states that authority and oversight can change as evidence, performance, risk, or the technology changes (AMA, 2026). It is a statement of principles, not an evaluation protocol.

Applied to evaluation, the rule has two axes. The grid below is a handbook synthesis, not a validated standard.

What the agent may do If errors are reversible and low harm (for example, a scheduling note or a draft letter) If errors could cause irreversible or serious harm (for example, an order, a denial, or a missed critical result)
Inform: retrieve and summarize; a person acts Local accuracy check on typical records; injection testing if it reads outside content Add subgroup checks (language, document quality, payer, age) and a seeded-error audit of whether reviewers catch mistakes
Recommend: draft for sign-off End-to-end completion and consistency on the local task mix; measured reviewer correction rate Add a shadow period comparing proposed actions with what staff did, and pre-set error thresholds
Act without per-item review, within narrow bounds All of the above, plus refusal, escalation, and access testing; audit of every executed action during a supervised start; rollback plan Use only where CHAI-style human attestation of EHR writes is replaced by an equally strong control, with prospective monitoring and explicit stop criteria
Act on clinical decisions: order, treat, or prescribe Not applicable Prospective comparative evidence on patient outcomes; regulatory and legal review

Move an agent up a row only when it meets criteria set in advance, and move it down when monitoring shows errors above threshold or when the model, tools, or workflow change. When an agent has made no errors in n reviewed actions, the error rate is still uncertain: the upper limit of a 95% confidence interval is about 3 divided by n, so 100 error-free actions bound the rate only below about 3%, and showing a rate below 1% needs about 300 (handbook calculation).

CHAI adds a caution that applies before any of this: not every workflow should be an agent, and many are better served by deterministic, validated components, because autonomy that is not needed adds cost and clutter. The same coalition reported that one health system found roughly three times as many agents built in approved low-code tools as use cases that had cleared formal review (CHAI, July 2026). An inventory of agents already running is therefore part of governance.

Legal responsibility for agent actions is covered in Physician AI Liability and Regulatory Compliance, and post-deployment monitoring of generative and agentic systems in the evaluation chapter’s observability section.

A Local Evaluation Plan for a Health System

The following sequence turns the evidence above into steps a governance committee can require. It is a synthesis, not a validated protocol.

  1. Write the task down. State the workflow, the users, the systems the agent can read and write, and the actions it may take without review.
  2. Ask whether it needs to be an agent. If a fixed rule or a single model call does the job, use that.
  3. Set up access and audit. Give the agent its own identity, the narrowest access the task needs, and a marker on every write.
  4. Measure the current baseline. Record the error rate, turnaround, and staff time of the existing workflow, and of a simpler non-agent option, on the same cases.
  5. Build a test set from local work. Sample de-identified cases in proportion to the local task mix, including hard, incomplete, and near-duplicate-patient cases, and define a correct end-to-end result for each. Keep part of the set unseen by the vendor.
  6. Measure completion and consistency. Run every case three to five times; report completion over all attempts with a confidence interval, and the share correct on every run.
  7. Grade the path and the record. Have clinicians review a sample of trajectories for unsafe, unnecessary, or out-of-order actions, and check that every action the agent reported was actually executed.
  8. Test refusal, escalation, and injection at scale. Run impossible, ambiguous, out-of-scope, and adversarial cases many times, because rare failures need many trials to appear; score missed and unnecessary escalations.
  9. Price it. Record compute cost, time, and human review time per correctly completed task.
  10. Set go, stop, and promotion criteria before go-live. Define the error thresholds, incident types, and model or tool changes that pause the agent, and the evidence needed to reduce supervision.
  11. Run a shadow period, then a supervised start. Let the agent propose actions on live inputs without executing them, compare them with what staff did, then begin at the lowest autonomy level the use case allows, with every write action attested and logged and with seeded-error audits of reviewers.

Check Your Understanding

A vendor proposes a prior authorization agent and reports “85% accuracy” on its internal test set.

Question: What should the committee ask before accepting the number?

Answer: Ask whether the 85% is the share of individual steps done correctly versus the share of whole tasks completed end to end, how many tasks and runs it covers and with what confidence interval, whether abandoned and timed-out tasks are in the denominator, how often the same task succeeds on repeated runs, and which harness was used. In HealthAdminBench, one agent passed 82.8% of individual subtasks but completed 26.7% of whole tasks (Bedi et al., arXiv:2604.09937, 2026, preprint). Then ask for results on the institution’s own payer mix and for the agent’s behavior when documentation is missing.

A health system plans an agent that reads faxed referral packets, extracts the reason for referral, and books specialist appointments without per-item review.

Question: Which tests are essential before go-live?

Answer: Test whether the agent stops and escalates when a packet is incomplete, illegible, clinically urgent, or matches more than one patient; test it with adversarial text embedded in scanned pages, since hidden instructions in medical images changed model outputs in a controlled study (Clusmann et al., 2025); confirm that every booking it reports exists in the scheduling system; and define, before go-live, the error rate that would return it to per-item review.

Key Takeaways

Essential Points
  1. Judge agents by completed work: End-to-end completion, counted over every attempt and reported with its uncertainty, is the primary measure; step accuracy overstates it.
  2. Require repeated runs: Success on one attempt does not predict success on the next.
  3. Check the record, not the report: Agents can describe an action they never took.
  4. Grade the path: Appropriate actions, process safety, and resource use matter alongside the endpoint (Ruhrberg Estévez et al., 2026).
  5. Test stopping and manipulation: Impossible and out-of-scope tasks, and hidden instructions in documents and images, reveal failures that ordinary test cases miss.
  6. Compare like with like: A score belongs to a model and its harness together.
  7. Match evidence to autonomy and consequence: The more an agent may do on its own, and the greater the harm an error could cause, the stronger the evidence required (AMA, 2026).

How do you evaluate an AI agent in healthcare?

Test the whole workflow the agent will run, not its individual answers. Measure the share of tasks completed correctly end to end with its uncertainty, repeat each task several times to measure consistency, check the path the agent took and what it actually executed in the record, test whether it stops and escalates when a task is impossible or out of scope, and record cost and time per task. Match the strength of this evidence to how much the agent may do and how much harm an error could cause: the AMA’s framework for agentic AI holds that greater consequence and greater autonomy require stronger evidence (AMA, 2026).

How accurate are AI agents on healthcare tasks?

It depends on the task, the setup, and the test environment, and the published benchmarks are all simulated. On short, explicitly worded chart tasks in MedAgentBench, the best model in the original 2025 study succeeded on 69.67% (Jiang et al., 2025), and a redesigned agent reached 91.0% on the same tasks and 88.67% on 300 new ones (Chen et al., 2026). On long consultation tasks in PhysicianBench, the best agent completed 46.3% on the first attempt and 28.0% on all three of three runs (Liu et al., arXiv:2605.02240, 2026, preprint). On administrative tasks in HealthAdminBench, the best agent completed 36.3% end to end (Bedi et al., arXiv:2604.09937, 2026, preprint).

What is the difference between task completion and step accuracy for AI agents?

Step accuracy counts how many individual actions an agent gets right; task completion counts how many whole jobs it finishes correctly. They can differ widely. In HealthAdminBench, one agent passed 82.8% of individual subtasks but completed only 26.7% of whole tasks (Bedi et al., arXiv:2604.09937, 2026, preprint). For a buyer, end-to-end completion on representative workflows is the better estimate of how much work still falls back on staff.

Can prompt injection affect medical AI agents?

Yes. Prompt injection is an instruction hidden in content the system reads, such as an image, a document, or a web page. In a Nature Communications study, text hidden in medical images, including low-contrast text that human readers would not readily notice, raised the rate at which four vision-language models missed confirmed cancers, and the mitigations tested failed for three of the four models (Clusmann et al., 2025). Any system that reads outside documents should be tested against such inputs before deployment.

Why do AI agents score so differently across studies?

A score belongs to the model and its harness together: the instructions, tools, turn limits, and grader. When a team including MedAgentBench’s authors redesigned the agent’s instructions and tools, success on the same 300 tasks rose from 69.67% for the best model in the original setup to 91.0% with GPT-4.1 (Chen et al., 2026). Task length and wording also matter: the same group noted that the benchmark’s instructions were more explicit than those clinicians actually give.

What frameworks exist for governing agentic AI in healthcare?

The American Medical Association published a governance framework for agentic AI that assumes clinical functions, organized around four questions: function, evidence and risk, authority and oversight, and accountability (AMA, 2026). The Coalition for Health AI released a best practice guide and a testing and evaluation framework for agentic AI in July 2026 (CHAI, July 2026). Both are consensus guidance rather than validated evaluation methods.

Conclusion

Across the published healthcare agent benchmarks, agents that perform individual steps well still fail to complete a large share of long or ambiguous tasks, and the same agent does not always succeed twice. These are measurable properties, and a health system can measure them on its own work before deployment: end-to-end completion with its uncertainty, consistency across runs, the trajectory and the executed record, refusal and injection behavior, and cost per completed task. Because almost all agent results come from simulated environments built at a few institutions, the first local deployments should begin in shadow mode and then under supervision, with every write action attested and logged, and with stop criteria set in advance.

Continue Reading

The Physician AI Handbook is free to read and cite, with no advertising. If this chapter informed your work, you can support the handbook.

Bryan Tegomoh, MD, MPH

Physician-scientist and epidemiologist in health AI and biosecurity.