The Art of Evals
On June 4, 1996, the maiden flight of the European Space Agency’s Ariane 5 rocket lifted off, closing out a suspenseful hour of delay due to visibility concerns in the coastal region of Kourou, French Guiana. Just 37 seconds after ignition, the rocket strayed from its flight path, ultimately self-destructing. The Ariane 5 had reused incompatible code from its predecessor rocket, the Ariane 4, and a conversion error between data types doomed the mission1. It has since become known as one of the most expensive software bugs in history, with a loss of over $370 million in 1996 dollars—at this point infamous to the point of cliché.
The Ariane anecdote is classic supporting evidence for the philosophy of test-driven development (TDD) in the world of software 1.02. Write good tests; avoid catastrophic outcomes. But over the last five years, we’ve seen a marked shift away from software 1.0, where all code is written by hand, to this new, weird regime of software 3.0. We now command computers through natural language. What becomes of testing philosophy now?
The equivalent is what AI practitioners today unglamorously call “evals.” Whereas software 1.0 bugs reduce down to some simple line of code, models and agents fail statistically and distributionally, the stuff of data. There’s an art to building agentic systems, and there’s an art to testing and safeguarding them as well. We don’t want to wait around for the rocket explosions of the AI world as the technology becomes more pervasive. It’s important we get the art of the eval right.
What is an eval
An AI evaluation consists of up to three key components.
- Data: Some set of tasks or problems to probe a model and gauge its performance. Like the question set in an exam.
- Graders: The mechanism by which we measure goodness or correctness over a model’s responses. Like the grading rubrics for an essay. Designing an optimizing metric is an important subproblem of grader construction.
- Environments: Some tasks and graders are defined relative to some stateful execution substrate, like a particular app. You need a track in order to take some fitness exams. Not every eval needs an environment; for some exams, a standard test paper is enough.
What makes an eval good
The fundamental specification unit in software 1.0 is imperative logic. If this, do that. Save this under some name. A well-designed test suite breaks down that logic into its verifiable bite-sized chunks and ensures they all work independently.
The specification unit of an agent is data, which is drawn from some distribution. Given some input, such as “Write me a function that…”, we expect some output, such as “def post_train(*args, **kwargs): …”. A well-designed evaluation has several characteristics:
- Coverage: Do we have enough test (input, output) pairs spread across the different domains in the data we care about? Are less common failure modes represented?
- Alignment: Does improving on this evaluation mean our system actually gets better at a task we care about? Is our metric adequately aligned with what we define as good? What if our essay rubric doesn’t have the right criteria?
- Realness: Probing model behavior on a diverse set of examples is only useful if they are representative of real queries the model will get in production. Tweak the value for gravity in a physics exam and the questions will look like physics and feel like physics, but in the real world all your calculations will be wrong. This is a primary concern if your eval contains synthetic data.
- Unsaturation: Less important for a regression test suite, more important for hill-climbing. If you’re trying to measure quality gains in a model, it’s important for the corresponding eval to have room for improvement in order to see that lift.
A real (synthetic?) example
We walk through the construction of one of our evaluations: a benchmark for detecting medical billing errors. This is probably more detail than necessary for how to build evals generally, but we include enough precision to shed light on some of the real design decisions and trade-offs we faced when building a benchmark for a non-toy example.
The problem
According to 2022 research by the Kaiser Family Foundation3, 43% of all U.S. adults report receiving a medical or dental bill they thought contained an error. Worse yet, 51% of all adults with healthcare debt who received a bill they thought was erroneous were unable to successfully dispute it or did not start a dispute at all. Financial pressure and fear of collections coerce consumers into paying first and redressing later. Complicated insurance legislation and multiple sets of clinical nomenclature (ICD codes4, CPT codes5, HCPCS codes6, DRG codes7, and so on)—often indecipherable to those outside healthcare—all increase the burden of disputing a bill for the consumer.
Enter modern foundation models. Major providers like OpenAI and Anthropic are advancing into the consumer healthcare space, and mainstream news records a few nascent success cases of AI-empowered patients disputing billing errors8. A question arises: how good are they, really?
Why hasn’t this been done
It turns out this is a fairly sophisticated problem. Data availability is one of the archnemeses of progress in the AI for healthcare space, and the billing niche is no exception. Medical bills contain PHI (protected health information), and neither hospitals nor consumers are eager to share them. To our knowledge, there is not a single public evaluation online that addresses medical billing error detection, likely due to some combination of data availability and attention. A few tangential datasets exist, like MedHELM’s MIMIC-IV Billing Code dataset9, but this probes for the ability to map between clinical notes and ICD-10 billing codes, overlooking the possibility of billing errors by treating available records as ground truth. Worse yet, MIMIC is not freely available. Some healthcare datasets are credential-gated on platforms like PhysioNet10, posing further barriers to entry for practitioners.
To remedy this, we opt to go synthetic. Implication: we need to build the dataset from the ground up.
Realism, part one: clinical
# Evaluation Task
Review the after-care visit summary and the bill side by side. Decide which
billed items, if any, should be disputed.
## Side-By-Side Record
### After-Care Visit Summary
```json
{aftercare_summary_json}
```
### Billing Information
```json
{bill_json}
```
## Instructions
- Focus on whether the bill conflicts with the care actually documented
in the after-care summary or contains billing behavior that should be
disputed.
- If no billed items should be disputed, return an empty `disputed_items`
list and explain briefly why.
- If the bill should be disputed but the issue applies to the whole bill or
patient identity rather than a specific billed item, return
`summary_decision` as `dispute` with an empty `disputed_items` list.
- If billed items should be disputed, include only valid `item_id` values
that appear in the bill.
- Explanations should be concise and specific to the disputed items.
- Do not mention hidden labels, benchmark metadata, or ground-truth
categories.We cannot simply prompt an LLM to generate hundreds of medical billing code clusters, including diagnoses, procedures, and billing codes. Their clinical plausibility would be in question, as would the diversity of the distribution if entropy was a pure function of temperature-based sampling stochasticity.
As a mitigation, we upgrade from synthetic to semi-synthetic. We anchor our synthesis in 5% of real-world Medicare beneficiary claims data from 2008 to 2010, compiled by the U.S. Centers for Medicare & Medicaid Services (CMS)11. Aggressive de-identification and obfuscation by CMS compromised the clinical plausibility of individual encounter bundles, with dates perturbed, variables like expenditures and birth years coarsened, and columns shuffled, though overall dataset statistics like the marginal prevalence of diagnoses or procedures were preserved.
This shuffling necessitated the usage of LLMs for clinical reconstruction. A primary clinical direction is selected from each billing and diagnosis bundle we sample, with seeded codes that are irrelevant to that direction discarded and plausible supporting codes added where needed. Encounter dates are deterministically shifted forward, and all diagnosis codes returned by the LLM are ICD-10.
Add on subsequent LLM- and rules-based passes for semantic cleaning and demographic repair, such as no diagnoses or procedures that conflict with the patient sex or encounter type, and we have the beginnings of a medical billing dataset, split into balanced inpatient and outpatient cohorts.
Realism, part two: pricing
Next challenge: the prices are distributed according to 2008-era amounts. Preliminary data exploration reveals a median total bill of $80 and a p99 value of $17,000. All dollar amounts are in USD and represent the summed total of raw claim payments per encounter. Both numbers are unrealistically low compared to contemporary data. We can do better.
Recent U.S. legislation, namely the Hospital Price Transparency Rule, mandates hospitals release transparent pricing charts for their services, and we use this data to overwrite every bill’s billable services and inpatient facility components with figures from 2025. We assign an in-network scenario with probability 0.85, inspired by the 84.1% share of medical claims that were in-network according to 2019 data12, and sample one of five payers: United Healthcare, Blue Cross Blue Shield, Aetna, Humana, or Medicare Advantage if the patient is 65 or older. A repricing fallback waterfall is executed, since not all hospitals have all procedures, and final priced items in each bill carry multiple accounting fields for provenance. We add natural-language descriptions for codes where missing and generate a corresponding after-care summary. The care summary is required for allowing the agents we evaluate later to identify contextual billing discrepancies, like phantom billing13.
Realism, part three: perturbations
| Code | Description | Charge |
|---|---|---|
| 99284 | Emergency dept visit, level 4 | $1,842.00 |
| 71046 | Chest x-ray, 2 views | $312.00 |
| 82947 | Glucose, quantitative | $48.00 |
| 84520 | Urea nitrogen (BUN) | $42.00 |
| 82565 | Creatinine, blood | $45.00 |
| 80051 | Electrolyte panel | $86.00 |
| 82310 | Calcium, total | $41.00 |
| 82040 | Albumin, serum | $38.00 |
| 84450 | Transferase, AST | $44.00 |
| 84460 | Transferase, ALT | $44.00 |
Unbundling. These eight tests are components of a comprehensive metabolic panel and should be billed together under a single code. The lines total $388.00, with an estimated allowed amount of $232.80, compared with $124.00 for bundled code 80053. | ||
| 36415 | Venous blood collection | $28.00 |
| J1885 | Ketorolac 15 mg, injection | $67.00 |
Illustrative estimate assumes allowed charges equal 60% of billed charges and the plan pays 80% of the allowed amount.
| # | Error class | Scope |
|---|---|---|
| 1 | Upcoding | Shared |
| 2 | Unbundling | Shared |
| 3 | Phantom billing | Shared |
| 4 | Mismatched diagnosis | Shared |
| 5 | Ignored contractual adjustment | Shared |
| 6 | Duplicate billing | Shared |
| 7 | Quantity inflation | Shared |
| 8 | Wrong patient demographics | Shared |
| 9 | Date of service mismatch | Shared |
| 10 | DRG mismatch with diagnoses | Inpatient only |
| 11 | Physician procedure without facility support | Inpatient only |
| 12 | Wrong CPT for inpatient setting | Inpatient only |
Our bills are starting to look realistic, but the point of this exercise was generating realistic medical billing errors. Thankfully, it’s easier to perturb than it is to synthesize. We take a random 80% of our clean, golden bills, chosen to ensure sufficient coverage rather than real-world prevalence; sample one billing error class from a predefined taxonomy; and mutate the bill accordingly. After-care summaries are kept unmodified so that the clinical narrative functions as ground truth for error detection. To maintain consistency and financial realism, mutated bills are renormalized and repriced. Some errors, like unbundling14, lead to the addition of new billing line items.
Coverage
Generally, synthetic data can sometimes lower the burden of coverage and alignment in exchange for a higher realism bar, as we saw here. We control the exact set of billing errors we care about by defining the error taxonomy from the start. There are more billing error categories we could have considered, particularly those related to payer contracts, such as No Surprises Act violations, but that would complicate an already synthetic dimension and create another burden of proof to be cleared for realism. We accept this as a tradeoff in the v0 dataset.
Rooting our generation in CMS claims also induces a demographic and clinical skew toward the Medicare population; this means our billing examples are comorbidity-dense and highly complex. We argue this is appropriate. Models that can successfully navigate dense billing clusters are more likely to generalize to the broader, typically less complex rest of the population, though this is not well-tested and the skew remains a coverage gap we must accept.
Alignment
Grading is relatively simple for this task: does the model appropriately flag erroneous billing items? The selection of evaluation metrics required some care. We reported recall and precision of disputable funds15, since these directly operationalize the financial efficiency of a given model from the patient’s perspective. If a model catches ten $12 errors but misses the $4,000 one, that’s far worse for the patient despite winning on a line-item basis. We also tracked bill and line-item recall, precision, and F1 metrics. High false-positive rates translate to increased administrative burden and unnecessary frictional costs to the consumer.
Unsaturation
A preliminary evaluation across 11 models in early May 2026 reveals a wide variance in performance across models, none of which saturate the benchmark.
Recall of Disputable Funds
Reflection on the example
Looking back at our example, this was a lot of design and care just to produce an eval artifact that is, truthfully, only half-decent. The gold standard to certify this eval as useful for rigorous public consumption would be to conduct an expert audit study with certified billing experts on the data, and perhaps collect a human baseline as well. Short of that, the final records remain intrinsically bound by the zero-shot clinical accuracy of the generating models.
We also do not address several complicating dimensions that plague real-world medical and billing practices, such as the lack of standardization between facilities. Realism strikes again! In practice, different hospitals have different facilities and physicians with varying expertise, which may translate to different operating procedures and billing frequencies from place to place. Our generation procedure does not account for variation like that.
Nevertheless, this work is not for naught. The real world is messy, and data is locked away and hard to collect. We need to start somewhere, and in a world where billing evaluations are nonexistent, this is a good first step.16
Conclusion
A good eval need not be exhaustive as long as it is treated with appropriate caution and uncovers useful insights. For the healthcare benchmark, despite its many limitations, we notice a number of useful trends among the examined models: some are more conservative and others are more liberal when flagging errors. One is clearly the best, and we walk away with some insights into why.
Evals are an art precisely because there is no standardized handbook for them. At Whitney AI, we build ours according to a set of best practices we’ve honed after years of crafting them across multiple scales and industries. Their importance is often overlooked by the companies we talk to, but as the old adage goes: what can be measured can be improved. As we’ve argued, getting that measurement right can be tough, but often a rock-solid eval is all that’s stopping you from that next level of product or agent quality.
If you’re building something where being wrong is expensive, or you’re struggling to see ROI, we should talk.