---
title: "Hospital Operations and Management – The Physician AI Handbook"
source: "https://physicianaihandbook.com/implementation/hospital-operations.html"
author: "Bryan Tegomoh, MD, MPH"
last_updated: "2026-11-13"
doi: "10.5281/zenodo.18251404"
license: CC BY 4.0
generated-by: handbook-markdown-export
---

# Hospital Operations and Management

Judge an operational AI tool by whether the hospital runs better after staff act on it, not by prediction accuracy alone. Hospitals use AI to schedule operating rooms, predict missed appointments, forecast admissions and discharges, and assign billing codes. The strongest study reviewed here is a randomized trial at one cancer center: machine learning estimates of surgical case length reduced scheduling error, and patients scheduled after the day’s first case waited less ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)). In a benchmark of code lookup, the best of four language models tested in December 2023 returned the exact ICD-10-CM code for a code’s official description about one time in three ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)).

Learning Objectives

After completing this chapter, clinicians and health system leaders should be able to:

- Name the main operational uses of AI in hospitals and the kind of evidence this chapter reviews for each
- Tell a study of prediction accuracy apart from a study of operational outcome
- Interpret the randomized trial of machine learning surgical case-length prediction and its limits
- Explain why a no-show prediction model is only as useful as the action attached to it
- Describe what is and is not known about large language models for medical coding and administrative tasks such as prior authorization
- Name a balancing measure for an operational change
- List the questions to ask before a hospital buys or builds an operational AI tool, and what to measure after go-live

Chapter Summary (TL;DR)

**Operating room scheduling:** In a randomized trial of 683 surgical patients at one cancer center, machine learning case-length predictions lowered mean absolute scheduling error from 59.3 to 49.5 minutes. Patients scheduled after the day’s first case waited a mean of 16.3 minutes instead of 49.4, with no increase in time between cases; these were secondary outcomes, reported without confidence intervals ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)).

**Missed appointments:** In a before-and-after study at one hospital, staff used a model to choose which quarter of patients to call before an MRI, in place of calls made without a set method and with about the same staff effort. No-shows fell from 19.3% to 15.9% ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)).

**Medical coding:** In a benchmark, the best of four base language models returned the exact ICD-10-CM code for a code’s official description 33.9% of the time ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)). In a retrospective case study, physician reviewers preferred GPT-4’s emergency department primary diagnosis code, chosen with retrieval from past emergency visits, over the provider-assigned code more often than the reverse ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)).

**Bottom line:** A study of prediction accuracy does not show that a hospital runs better. Before scaling a tool, ask for the operational outcome, a comparison group, what got worse, and the effect on each patient group.

### Evidence at a Glance

| Use | Study | Design | Setting | What was measured | Main result | Main caveat |
|----|----|----|----|----|----|----|
| Operating room case length | [Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361) | Randomized trial | One cancer center, two services | Scheduling error; waiting of the patient after the first case | Mean absolute error 59.3 to 49.5 minutes | Waiting was a secondary outcome, without a confidence interval |
| MRI no-shows | [Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594) | Before-and-after quality improvement study | One hospital, Singapore | No-show rate | 19.3% to 15.9% | No concurrent control; rates were rising beforehand |
| Code lookup | [Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040) | Benchmark | Codes from one health system’s records | Exact match from a code’s official description | Best 33.9% for ICD-10-CM | Not coding from notes; December 2023 models |
| Emergency primary diagnosis coding | [Klang et al., 2025](https://doi.org/10.1056/AIcs2401161) | Retrospective case study | One health system, 500 visits | Physician reviewer preference against the provider-assigned code | 447 versus 277 accuracy judgments for GPT-4 | Primary diagnosis only; no payment outcomes |
| Administrative agents | [Bedi et al., 2026](https://arxiv.org/abs/2604.09937), preprint | Preprint benchmark | 135 tasks in simulated systems | Whole-task completion | Best 36.3% | Not live portals |

## Introduction

Physicians feel hospital operations every day: a case that runs long and leaves the next patient waiting, a full emergency department waiting for beds, an empty clinic slot, a denied claim. Some AI tools target this work of running a hospital rather than diagnosis or treatment. They may be bought by operations, finance, or information technology leaders without the clinical review a diagnostic tool would get.

Physicians still have a stake. An operating room schedule decides how long patients wait while fasting. A no-show model can decide who gets a reminder call or whose slot is double-booked. A coding tool turns physician documentation into the codes that drive payment, quality metrics, and research data. Physicians who sit on governance committees, lead departments, or simply document care should know what the evidence shows.

This chapter reviews evidence on operating room scheduling, appointment no-shows, and revenue cycle and medical coding, and sets out what to ask about patient flow tools. Payer use of algorithms in coverage and prior authorization has its own section in [Healthcare Policy and AI Governance](../future/policy.html#sec-payer-algorithms), and billing compliance for AI-drafted notes is covered in [AI-Assisted Clinical Documentation](../practical/documentation.html#sec-documentation-billing).

| Term | Meaning |
|----|----|
| Mean absolute error | The average size of a prediction’s miss, ignoring whether it was too high or too low |
| Signed mean error | The average miss, counting direction; here, a planned time that was too short counts as negative |
| Area under the curve (AUC) | How well a model ranks cases that have an outcome above those that do not; 0.5 is chance and 1.0 is perfect |
| Before-and-after study | A comparison of one period with an earlier one, without a group that did not use the tool over the same period |
| Concurrent control | A comparison group that did not use the tool during the same period |
| Balancing measure | A measure tracked to check that an improvement does not make something else worse |
| Revenue cycle | The administrative steps from scheduling and registration through coding, claims, payment, and denials |
| ICD-10-CM | The diagnosis code set used in the United States for billing and records |
| Exact match | The model returned the identical code, not a broader or related one |
| Nonbillable code | A valid code that cannot be used on a claim, usually a category with more specific codes beneath it |
| Retrieval-augmented generation | Giving a language model relevant reference material, here data from past emergency visits, before it answers |

## Prediction Accuracy Is Not an Operational Outcome

A model that predicts well changes nothing until someone acts on its output, and the action is what the hospital should evaluate. A study that reports how accurately a model predicted admissions, length of stay, no-shows, or case duration on past data has taken a necessary first step. It leaves three questions open: whether staff used the prediction, whether the operational measure improved, and whether the improvement came at a cost elsewhere, such as longer waits for another group of patients.

Two designs differ sharply in how well they answer those questions. A randomized trial assigns cases or time periods to the tool or to usual practice, so changes happening at the same time affect both groups alike, on average. A before-and-after study compares a period with the tool to an earlier period without it, so any concurrent change, such as new staff, a seasonal pattern, or another improvement project, can explain part of the difference. At minimum, an evaluation should show a year or more of data before the change, which reveals whether the measure was already moving. A good evaluation also tracks a balancing measure, something that could get worse because of the change. The two studies in the next sections show the difference.

## Operating Room Scheduling

Operating room schedules depend on estimates of how long each case will take. When a case runs longer than planned, the next patient waits, often while fasting. When it runs shorter, the room and team sit idle.

### A randomized trial of case-length prediction

At Memorial Sloan Kettering Cancer Center, a team built a machine learning model that used more than 300 data features from the electronic health record (EHR), including patient characteristics such as age, sex, race, body mass index, and comorbidities, along with surgeon, procedure, and operational factors, to predict case duration. The team tested the model in a randomized clinical trial on two campuses from April to June 2018. Colorectal and gynecology cases were randomly assigned to one of two estimates. For one group, a scheduler entered the model’s prediction into the live schedule the day before surgery. The other group used the usual estimate from the EHR, supplemented by the surgeon or scheduler ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)).

| Outcome | Measured in | Usual estimate | Machine learning estimate |
|----|----|----|----|
| Mean absolute error of planned duration (primary outcome) | All cases | 59.3 minutes | 49.5 minutes (P = .03) |
| Planned duration off by more than 60 minutes | All cases | 33.7% | 25.4% |
| Mean wait of the patient scheduled after the day’s first case in a room | Patients scheduled after a first case | 49.4 minutes | 16.3 minutes |
| Mean time between the end of that first case and the start of the next | Same patients | 70.6 minutes | 69.1 minutes |
| Mean time from registration to entering the operating room | Same patients | 173.3 minutes | 148.1 minutes |

683 patients (356 usual estimate, 327 machine learning estimate). Rows 2 to 5 are secondary outcomes; the paper reports no P values or confidence intervals for them. Source: [Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361).

In this trial, scheduling with the model’s estimates cut the mean wait of patients scheduled after the day’s first case by 33.1 minutes, without lengthening the time surgeons waited between cases. The authors describe the drop as significant but report no P value or confidence interval for it ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)). On average, the usual estimates were 33.7 minutes too short, compared with 3.6 minutes for the model. When a case runs past its planned end, the next patient waits.

The trial also tracked a balancing measure: time between cases, when surgeons wait. It did not lengthen (70.6 versus 69.1 minutes). The authors argue that implementation, not model building, has long been the main obstacle to predictive modeling, and this trial put the predictions into the live schedule ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)).

Several limits apply. The trial covered two surgical services at one cancer center. It was sized to detect a difference in mean absolute error across all cases, not within each service. The error reduction was significant for colorectal surgery (74.0 to 55.7 minutes, P = .004) but not for gynecology (51.9 to 46.9 minutes, P = .64), which does not show that the model failed in gynecology; there, the model reduced the systematic underestimate from 24.6 to 7.5 minutes (P = .01). Waiting and the other operational outcomes were secondary, measured only for patients scheduled after the first case in a room, and reported without P values or confidence intervals. Predictions were made one day ahead and entered by hand, cases booked after the predictions were made were excluded, and the authors note that predictions may be inaccurate when the booked procedure differs from the one performed ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)). A hospital considering a similar tool should ask whether it has been tested on its own services and booking patterns, and how predictions reach the schedule.

## Appointment No-Shows

Missed appointments waste scarce slots and lengthen waits for other patients. A no-show prediction is only as useful as what the clinic does with it.

### A deployed MRI no-show model: a before-and-after study

At Changi General Hospital in Singapore, a radiology team trained a model on 32,957 outpatient MRI appointments from 2016 to 2018 using 21 features, none of them clinical diagnoses. On a held-out set of 1,080 appointments, the model distinguished appointments that would be missed from those that would be kept with an AUC of 0.738. The factors that contributed most to its predictions were patient age and the wait for the appointment, followed by the number of reschedulings, male sex, certain postal districts, and certain weekdays ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)).

Before the change, technicians made reminder calls without a set method. From March 2019, staff instead called the 25% of patients the model ranked at highest risk, one business day before the scan. The cutoff was chosen so that calling took about the same staff effort as before ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)).

The no-show rate in the 6 months after the change (March to August 2019) was 15.9% (961 of 6,027 appointments), compared with 19.3% (2,112 of 10,967) in the preceding 12 months: an absolute drop of 3.3 percentage points (95% CI, 2.1 to 4.5), or 17.2% in relative terms ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Among patients the model flagged as high risk, those reached by phone missed 17.5% of appointments and those who could not be reached missed 40.3%. This comparison is not randomized: patients who answer the phone may differ in other ways from those who do not, so the gap cannot be credited to the call alone. Of the 930 high-risk patients staff reached, 192 rescheduled or cancelled. The study did not count these as no-shows, so some of the improvement may reflect advance cancellations rather than kept appointments.

This was a before-and-after quality improvement study, not a trial. The authors note that a randomized design may better account for variation in baseline workflow, and no-show rates had risen from 16.4% in 2016 to 19.1% in 2018, before the change. ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)) Without a concurrent control group, the study cannot separate the effect of targeted calls from that trend, from seasonal differences (the after period covered only March to August), or from a return toward earlier rates. The authors also note that the model will need retraining as patterns change over time.

### What the intervention should be

The same prediction can drive very different actions. A reminder call or transport help adds support for patients at risk of missing care. Double-booking the slot of a patient predicted to miss puts the cost of a wrong prediction on that patient if they arrive, and on everyone booked after them. In the MRI study, even among flagged patients staff could not reach, 40.3% missed their appointment, so most of them arrived ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Before choosing an action, check who the model ranks as high risk. If high-risk scores cluster in particular patient groups, for example through a predictor such as postal district, the action attached to the score decides who gains and who bears the cost of a wrong prediction. Before deploying a no-show model, decide what action it will trigger and measure that action’s effect on attendance for each patient group, not only overall.

## Patient Flow: Admissions, Beds, and Discharge

Tools that forecast emergency department arrivals, predict admissions, estimate length of stay, or flag likely discharges are another operational use of AI. This version of the chapter does not review that evidence. The same test applies: a forecast that predicts admissions well does not show that boarding, crowding, or patient outcomes improved.

When evaluating a flow tool, ask what decision it informs (calling in staff, starting a bed request earlier, prioritizing discharges), who acts on it and when, and what happened to boarding hours and length of stay against a comparison group. A model that flags more patients than staff can act on delivers less than its accuracy suggests; see [Capacity to Act Limits Realized Benefit](../implementation/evaluation.html#sec-evaluation-capacity).

## Revenue Cycle and Medical Coding

Automated coding research, AI-generated documentation queries, and coding audit risk are covered in [AI in Medical Coding and Billing](../implementation/medical-coding-billing.html).

The revenue cycle covers registration, eligibility checks, prior authorization, documentation, coding, claims submission, payment, and appeal of denials. The studies reviewed here concern medical coding and administrative agents. They measure code accuracy and task completion, not payments, denials, or audit findings.

### Language models alone: a code lookup benchmark

At Mount Sinai, researchers gave four large language models the official descriptions of 7,697 ICD-9-CM, 15,950 ICD-10-CM, and 3,673 Current Procedural Terminology (CPT) codes drawn from the health system’s records, and asked for each matching code. The models were queried in December 2023. GPT-4, the best model tested, returned the exact code for 45.9% of ICD-9-CM, 33.9% of ICD-10-CM, and 49.8% of CPT descriptions; Llama 2 70B returned under 3% for each set. Models did better on common, shorter codes. Of GPT-4’s wrong ICD-10-CM answers, 18.5% were codes that do not exist and 12.7% were nonbillable codes ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)).

The authors concluded that, in their base form, these models are not suitable for medical coding without further research. No model exceeded 50% exact match, and wrong answers tended to be either too general or falsely specific. ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)) This benchmark tested matching a code’s official description to the code, not coding from clinical notes, and it did not test prompt engineering, retrieval, tools, or fine-tuning. Newer models may perform better.

### Language models with retrieval: an emergency department case study

A Mount Sinai team then tested retrieval in a retrospective case study of 500 randomly selected emergency department visits. Nine language models assigned each visit’s primary ICD-10-CM diagnosis code with help from a retrieval system that drew on data from more than 1 million earlier emergency visits. Four physicians, masked to the source, compared the retrieval-supported GPT-4 code with the provider-assigned code, and two language models also reviewed the codes. The physicians’ judgments favored the GPT-4 code over the provider-assigned code for accuracy 447 times versus 277, and for specificity 509 times versus 181 (P \< 0.001 for both). These counts are reviewer judgments, not visits ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)). Smaller open models also improved with retrieval.

The comparison was with provider-assigned codes rather than with a professional coding audit, and the study measured reviewer preference for the primary diagnosis code, not claim acceptance or payment. ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)) It suggests that giving a model real, locally used codes to choose from can produce primary diagnosis codes that physician reviewers prefer. It does not show that the approach can code full encounters without review.

### Administrative agents: a preprint benchmark

AI agents are also being built to work through EHR screens, payer portals, and fax systems for prior authorization, appeals, and equipment orders. In a benchmark of 135 such tasks in four simulated environments, the best agent, a Claude Opus 4.6 computer-use agent, completed 36.3% of tasks from start to finish, including 38.3% of prior authorization tasks and 25.0% of appeals and denials tasks. Agents completed many individual steps correctly but far fewer whole tasks: the best subtask success rate was 82.8%, for a GPT-5.4 computer-use agent ([Bedi et al., 2026](https://arxiv.org/abs/2604.09937), preprint). How to evaluate such agents is covered in [Evaluating AI Agents in Healthcare](../implementation/ai-agent-evaluation.html). Payer use of algorithms in coverage decisions, the error rates of human prior authorization review, and the 2026 federal prior authorization rules are covered in [Healthcare Policy and AI Governance](../future/policy.html#sec-payer-algorithms).

### What physicians should watch

Coding tools sit between documentation and payment. A tool that suggests more specific or higher-paying codes can raise revenue, and it can also raise audit risk if the documentation does not support the code. A more specific code is better only when the note supports it. In the code lookup benchmark, one common error was false specificity ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)), and the retrieval case study measured reviewer preference, not documentation support ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)). Who is accountable for an AI-suggested code is covered in [AI-Assisted Clinical Documentation](../practical/documentation.html#sec-documentation-billing). Ask a vendor how its codes are checked against the note, how often human coders change them, what happens to denial and audit rates after go-live, and whether the tool ever suggests a code without citing the supporting text.

## Evaluating an Operational AI Tool

The questions below adapt [Evaluating AI Clinical Decision Support Systems](../implementation/evaluation.html) to operational tools. Ask the first set before buying or building and the second after go-live.

**Before buying or building**

| Question | A good answer | Example in this chapter |
|----|----|----|
| 1\. What decision and action does the output drive, and who acts? | A named role, action, and time | A scheduler entered case-length predictions the day before surgery |
| 2\. How does the output reach the workflow? | Inside the system staff already use, or a tested manual step with an owner | Predictions were entered by hand each morning |
| 3\. What operational outcome changed, and by how much? | Absolute and relative change in a measure such as waiting minutes, missed appointments, denials, or audit findings | No-shows fell 3.3 percentage points (17.2% relative) |
| 4\. Against what comparison? | A randomized or concurrent control, and at least a year of trend before the change | Randomized trial (operating room) versus before-and-after (MRI) |
| 5\. What could get worse? | A balancing measure tracked alongside the main outcome | Time between cases did not lengthen |
| 6\. Who ran the evaluation, and where? | Independent or peer-reviewed results from a site like yours | Each study of hospital data here comes from one center or health system |
| 7\. Can staff act on every flag? | Flag volume matched to staff capacity | The call list was sized to the existing calling effort |

**After go-live**

| Question | A good answer |
|----|----|
| 8\. Who benefits and who bears the cost? | Results by patient group: who is called, who waits, who is double-booked |
| 9\. Is performance holding? | A named owner and schedule for rechecking on local data, with retraining when patterns shift |
| 10\. Who reviews, overrides, or switches it off? | A named reviewer, an override path, and a stop rule agreed before launch |

If a model’s inputs include race, postal district, or payer, question 8 should also ask how those inputs were handled.

## Check Your Understanding

Scenario 1: A Vendor’s No-Show Model

A vendor offers a no-show prediction model with an AUC of 0.80 and proposes double-booking the slots of patients it rates at high risk.

**Question:** What should the clinic ask before agreeing?

**Answer:** Ask what operational result the model produced at a comparable site, and whether it came from a randomized or concurrently controlled comparison; an AUC answers neither question. Then ask how many flagged patients still arrive. In the MRI study in this chapter, 40.3% of high-risk patients whom staff could not reach missed their appointment, so most did not ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Had those slots been double-booked, most would have had the original patient arrive as well. That study used reminder calls, and its result came from a before-and-after comparison. Require results by patient group, and start with a supportive action such as a call or transport help.

Scenario 2: An AI Coding Assistant

The revenue cycle team proposes a large language model that assigns diagnosis codes for emergency department visits, with coders reviewing only a sample.

**Question:** What evidence and safeguards should physicians ask for?

**Answer:** Ask whether the tool restricts its output to valid, locally used codes. In a benchmark, base models given a code’s official description returned the exact code at most half the time, and some answers were codes that do not exist ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)). Ask how its codes compare with professional coders’ codes on the institution’s own notes, how each code links to supporting text, and what happened to denials and audit findings after go-live. The retrospective case study of retrieval in this chapter covered only the primary diagnosis, compared GPT-4 with provider-assigned codes by reviewer preference, and did not measure payment outcomes ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)). Before moving coders to sample review, measure agreement on full review of local cases.

## Key Takeaways

Essential Points

1.  **Outcome, not accuracy alone:** An operational model helps only when acting on it improves an operational measure without making another one worse.
2.  **One randomized trial in this chapter:** At one cancer center, machine learning case-length estimates reduced scheduling error in a randomized trial; waiting fell for patients scheduled after the first case, a secondary outcome reported without a confidence interval ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)).
3.  **Targeting is the intervention:** Using a model to choose whom staff called, with about the same calling effort, was followed by a 3.3 percentage point drop in the MRI no-show rate in a before-and-after study ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Double-booking on the same scores would put the cost of wrong predictions on patients who arrive; the studies here did not test it.
4.  **Base language models could not look up codes reliably:** Given a code’s official description, the best of four models tested in December 2023 returned the exact code at most half the time and sometimes gave codes that do not exist, in a benchmark ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)).
5.  **Retrieval looks promising; payment outcomes are untested:** With retrieval from past emergency visits, physician reviewers preferred GPT-4’s primary diagnosis code to the provider-assigned code more often than the reverse, in a retrospective case study; claims, payments, and audits were not measured ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)).
6.  **Report results by patient group:** Show who waits, who is called, and who is double-booked, for each patient group.

Questions About Hospital Operations

### Does AI improve hospital operations?

It can, in the few uses tested against operational outcomes. In a randomized trial at Memorial Sloan Kettering, machine learning estimates of surgical case length reduced scheduling error, and patients scheduled after the day’s first case waited a mean of 16.3 minutes instead of 49.4 ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)). In a before-and-after study in a Singapore radiology department, using a model to choose which quarter of patients received a reminder call was followed by a fall in the MRI no-show rate from 19.3% to 15.9% ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Both are single-site results. A study that measures only prediction accuracy does not show that a hospital ran better.

### Can AI predict surgical case duration?

Yes, better than the usual estimate in at least one randomized trial. At Memorial Sloan Kettering, the planned duration of cases scheduled with a machine learning prediction was off by 49.5 minutes on average, compared with 59.3 minutes with the usual estimate. Fewer were off by more than an hour (25.4% versus 33.7%) ([Strömblad et al., 2021](https://doi.org/10.1001/jamasurg.2020.6361)). The gain was significant in colorectal surgery but not in gynecology; the trial was not sized to test each service separately, and it covered two services at one cancer center.

### Can AI predict patient no-shows?

Moderately well, and the value comes from what staff do with the prediction. At Changi General Hospital in Singapore, a model distinguished MRI appointments that would be missed from those that would be kept with an area under the curve of 0.74. In a before-and-after study, using it to choose which quarter of patients received a reminder call, in place of calls made without a set method, was followed by a fall in the no-show rate from 19.3% to 15.9% ([Chong et al., 2020](https://doi.org/10.2214/AJR.19.22594)). Without a concurrent control group, part of the drop may have other causes.

### Are large language models accurate at medical coding?

Not on their own. In a benchmark at Mount Sinai, GPT-4, the best of four models tested in December 2023, returned the exact ICD-10-CM code for a code’s official description 33.9% of the time, and some wrong answers were codes that do not exist ([Soroush et al., 2024](https://doi.org/10.1056/AIdbp2300040)). In a retrospective case study from the same health system, physician reviewers preferred GPT-4’s primary diagnosis code, chosen with retrieval drawing on over a million past emergency visits, over the provider-assigned code in 447 judgments versus 277 ([Klang et al., 2025](https://doi.org/10.1056/AIcs2401161)). Neither study measured claim denials, payments, or audit findings.

### What should a hospital ask before buying an operational AI tool, and measure after?

Before buying, ask for the operational result the tool produced at a site like yours, against a comparison group, with results by patient group. After go-live, measure the outcome the tool is meant to change (minutes of patient waiting, missed appointments, overtime, denials, or days in accounts receivable) against your own trend before the change, track what could get worse, and recheck performance as workflows change.

## Conclusion

The evidence reviewed in this chapter is narrow: one randomized trial in operating room scheduling, one deployed no-show model evaluated before and after, two coding studies that measure code accuracy rather than payment, and one preprint benchmark of administrative agents. For every operational tool, physicians can ask which action it triggers, which outcome it changed against a comparison group, what got worse, and which patients were affected.

Continue Reading

- [Evaluating AI Clinical Decision Support Systems](../implementation/evaluation.html): evidence design, local validation, and the capacity to act on a model’s output
- [Evaluating AI Agents in Healthcare](../implementation/ai-agent-evaluation.html): how to test agents that work through portals and EHR screens
- [Healthcare Policy and AI Governance](../future/policy.html#sec-payer-algorithms): payer algorithms, prior authorization error rates, and federal rules
- [AI-Assisted Clinical Documentation](../practical/documentation.html#sec-documentation-billing): billing and compliance for AI-drafted notes
- [Integration into Clinical Workflow](../implementation/workflow.html): fitting AI tools into daily work
- [Healthcare Administration](../implementation/healthcare-administration.html): billing costs, drafted patient messages, and prior authorization from the practice side

![](../images/bryan-tegomoh-author.jpg)

[Bryan Tegomoh, MD, MPH](https://bryantegomoh.com/)

Physician-scientist and epidemiologist in health AI and biosecurity.

[Follow @BryanTegomoh](https://x.com/intent/follow?screen_name=BryanTegomoh) [Follow on LinkedIn](https://www.linkedin.com/comm/mynetwork/discovery-see-all?usecase=PEOPLE_FOLLOWS&followMember=bryan-tegomoh-md-mph) [Support](../support.html)

The Physician AI Handbook is free to read and cite, with no third-party advertising. If this chapter informed your work, [you can support the handbook](../support.html).