RadMatch: Making Radiology Report Evaluation Auditable
September 1, 2026
Every time an AI model drafts a radiology report, someone has to answer a simple but hard question: is it correct? Not "does it sound fluent," not "does it use the right words," but did it catch the pneumothorax, get the location right, and not invent a finding that isn't there?
Today most tools that try to answer this question give you a single number. A 0.6. An 82%. It feels precise, but it doesn't tell you what actually went wrong, or whether the mistake was a missed pneumothorax or a rewritten sentence about a benign calcification. Those are very different problems, and a single score treats them the same.
We built RadMatch to fix that. RadMatch is an evaluation framework for AI-drafted radiology reports: it counts how many clinically significant mistakes a draft contains, and keeps a record of each penalty. As a result, a model builder can pinpoint exactly what to fix while radiologists can verify a score down to the individual finding rather than taking it on faith.
As with any evaluation method, a handful of examples can't capture the full range of what a metric will see in practice, but looking closely at specific cases is what makes an evaluation method trustworthy rather than a black box itself. That's the spirit behind the walkthrough below.
I. The problem with one number
Historically, evaluation methods borrowed from general language tasks (BLEU, ROUGE, BERTScore) compare word overlap, compare sentence similarity, and/or count matching phrases; however, these methods are blind to clinical stakes. On one hand, they will reward a report for reusing familiar phrasing. On the other hand, they penalize a harmless rewording just as heavily as a critical mistake such as changing "no pneumothorax" to "pneumothorax present," even though only one of those changes could change how a patient’s treatment.
To bridge this gap, more recent approaches ask a language model to read both reports and simply output a judgment. While this strategy tracks radiologist opinion much better, it still remains a black box: you get an aggregate score, but not underlying reasoning. As a result, if a model builder wants to know why a system is struggling, or if a radiologist wants to verify a number before trusting it, there's nothing tangible to inspect. Take, for instance, GREEN and CRIMSON, two such LLM-based metrics we compare RadMatch against below. Though both leverage language models to judge report accuracy, they ultimately collapse that evaluation into a single number rather than offering a traceable, finding-by-finding breakdown.
At its core, evaluating an AI-drafted report requires comparing it against a reference report, i.e. the ground-truth report written by a radiologist that we treated as correct. Converselt, the draft being evaluated against that reference is the candidate report.
In this context, every evaluation metric performs a version of the same task: reading both reports and measuring how far apart they are. Ultimately, the disagreement across the field is not whether tocompare them, but how to define what “far apart” truly means.
II. What RadMatch does differently
RadMatch breaks the comparison into three distinct, traceable steps:
Extracting atomic findings: Each report, the reference and the candidate, is broken down into individual, atomic observations: "small right pleural effusion," "no pneumothorax," and so on.
Clinical matching: Findings describing the same anatomy and the same condition in both reports are paired together, even when they're phrased completely differently or described at different levels of detail.
Multidimensional scoring: Every match is checked across seven dimensions, including location, severity, and certainty, into five distinct outcomes: correct (COR), partial (PAR), or incorrect (INC), with any remaining findings flagged as missing (MIS), or spurious/hallucinated (SPU).
Consequently, the resulting RadMatch score is a count of clinically significant errors: a missed pneumothorax counts far more than a reworded description of a stable, low-priority finding. Because every single penalty traces back to a specific pair of findings, the final score can be inspected and verified rather than taken on faith (Figure 1).
Beyond evaluating individual reports, this same structural framework scales up to provide a dataset-wide perspective on specific pathologies. Because every finding is evaluated across the same seven attribute dimensions, including location, severity, morphology, certainty, status, comparison to a prior report, and measurement, teams can pool results across an entire cohort to identify precise performance bottlenecks. For instance, when analyzing simple renal cysts with an internal model, RadMatch might reveal 32.5% of model errors coming from location, 15% from measurement, with severity (5.0%) and morphology (2.5%) accounting for smaller shares while the remaining dimensions remain accurate (Figure 2). Ultimately, this granular breakdown points directly at what to fix, rather than leaving a developer to guess why a pathology's overall score is lagging.
III. Does Radmatch actually match what radiologists think?
To validate these findings, we tested RadMatch against two benchmarks built directly from radiologist judgment. Most notably, on ReXVal benchmark, RadMatch's agreement with the panel reached a correlation of 0.79. Remarkably, this matches the same level of agreement radiologists show with one another, effectively proving that RadMatch evaluates reports with human-level clinical oversight.
Next, we turned to the second benchmark, RadEvalExpert. Here, the candidate reports come from actual AI report-writing systems trying to describe the scan from scratch, meaning they make the kind of varied, unpredictable mistakes real-world AI drafts make. Because this represents a harder, more realistic test, it is precisely where RadMatch's advantage over existing methods proved better. In fact, Radmatch achieved a correlation of 0.58, more than doubling the agreement of the next-best method we compared it against (0.24). Fig. 3 shows this comparison across both benchmarks, alongside how closely radiologists agree with each other.
Equally important, RadMatch stayed reliable across a wide range of underlying models, including smaller, open models that can run on a single consumer-grade GPU. This is a crucial advantage for clinical teams who need to keep sensitive patient data strictly on their own infrastructure rather than routing it through an external cloud service.
IV. In practice: two real cases
To understand what an agreggate score leaves out in practice, here are two case studies that demonstrate how high level scores can mask crucial diagnostic errors.
Case 1: A fluent report with three clinically significant errors
To see this in action, Fig. 4A shows this pair in full. At first glance, this candidate reads fluently and repeats two correct normal statements verbatim, which is exactly the pattern that flatters a single opaque score. Consequently, GREEN and CRIMSON, rates it 0.40 and 0.08 respectively, both suggesting a rather correct report. However, RadMatch's finding-level record tells a different story: 3 actionable errors, including a status inversion (the candidate says "no pulmonary edema" against a reference describing edema that is only improving, not resolved), a second inversion that drops the reference's pneumonia concern entirely, and a hallucinated feeding tube. Ultimately, because two of these touch urgent findings, the report's triage precision and recall both come out at zero, a result that a single, aggregated number would never surface.
Case 2: A candidate report that scores well but reverses a worsening finding
Take, for example, the case in Fig. 4B. Here, GREEN and CRIMSON both rate the candidate as largely sound, at 0.50 and 0.83 respectively. Behind those sound scores, however, the candidate calls a new, large effusion "no significant change," while also overstating the atelectasis as "severe." Unpacking the clinical details, RadMatch flags both as errors on urgent findings, specifically, a longitudinal inversion and a severity overcall, yielding 2 actionable errors overall. Ultimately, this represents the exact failure a scalar score is most likely to miss: fluent, confident language masking a clinically important reversal.
Crucially, these two examples aren't cherry-picked outliers; rather, they are representative of what happens once report generation models get good enough that a naive error count starts to disagree with what a radiologist would actually flag. To that end, the interactive dashboard we're releasing alongside RadMatch lets anyone pull up this same kind of finding-level view for their own reports, so a score is never just a number to take on faith.
V. Releasing RadMatch as open source
From the start, RadMatch was never intended as an internal scoring tool kept to ourselves. Instead we designed it to extend to new anatomies and imaging modalities from just a handful of examples. To that end, we are releasing RadMatch as an open-source code alongside an interactive dashboard, ensuring that anyone, from model builders and researchers to radiology teams, can inspect a score down to the individual finding that produced it.
Because RadMatch doesn't stop at a single, aggregate score, this finding level transparency unlocks vital safety specific views directly from the data. Specifically, because every finding, match, and verdict is kept on record, teams can immediately extract two critical metrics: how many of the truly urgent findings did the candidate get right (recall), and how many reported findings were actually warranted (precision). Ultimately, this is the exact kind of granular oversight we believe matters most for teams deciding whether an automated reporting system is ready to be trusted in clinical practice.
We believe this kind of transparency matters beyond our own use case. As more research groups advance report-drafting, everyone benefits from an evaluation method a practicing radiologist can actually read and challenge, rather than a black box number that asks to be trusted. For this reason, Radmatch serves as the foundational evaluation layer we rely on internally as our own report-generation research moves toward more complex, multi-study, longitudinal cases.
Looking ahead, we are excited to share that RadMatch has been accepted at the MedFMB workshop at ECCV 2026.
VI. Where this fits
First and foremost, RadMatch is an evaluation tool, not a diagnostic one. Specifically, it is built for research and model-development use to help teams systematically measure and improve report-drafting systems; as such, it is not a substitute for radiologist review of any generated report in a clinical setting. Built to extend across imaging modalities and anatomies, RadMatch's published benchmark results are currently chest X-ray, since that is where expert-annotated benchmarks with radiologist error counts exist today. We've since evaluated RadMatch on chest CT and abdominal CT as well, with those results coming in upcoming research.
In the meantime, if you work in report-generation research and want to try it on your own model's output, we’d welcome hearing from you.
Ready to explore Radmatch?
Acknowledgements
Thank you to our partner Segmed
For more updates as we continue to push the boundaries of AI in precision radiology: