The first census of real errors in deployed medical AI scribes has been published: 31.3% of clinical notes that had already passed final physician signature contain a confirmed error. The authors checked 565 notes from three commercial ambient AI scribes on the same set of 142 consultations — most often the systems made errors in allergy and medication data, fabricated patient identification, and described an “examination” that physically could not have taken place during a phone consultation. Along with the article, the OmissionBench dataset and pipeline verification code are open, so the measurement can be repeated independently.
What happened
On August 31, 2026, the article “One note in three: a verified census of three deployed AI scribes, and the instrument that counted it” (2608.31017, authors Sebastian Fox, Luke Markham, Ryan Lail, and Michael Karotsieris, 47 pages) was published on arXiv. The authors ran notes from three commercial ambient AI scribes — systems that automatically draft clinical records from dictation of a consultation — through the same set of 142 consultations: a total of 565 notes from British primary care, American outpatient practice, and author-created scenarios. A confirmed error was found in 31.3% of notes (95% confidence interval 27.0–35.6). The verification pipeline was multi-stage: 12 search passes proposed 13,678 error candidates, an importance filter left 5,898, and an adversarial panel of two models from different families confirmed 618 findings. A companion article (arXiv 2608.31016) about the measurement instrument itself was published in parallel: it shows that the error rate estimate varies from 28% to 97% depending on the chosen verification standard.
Context
Ambient AI scribe is an already deployed product category: the service listens to a consultation recording and automatically drafts a ready-made clinical note, while the physician formally remains the final authority and signs the document. The main vendor selling point is built precisely on this — “the doctor will check it anyway.” The second layer is methodological: the error rate depends on the measurement instrument just as much as on the product itself, so honest reporting is considered to be in the spirit of “pipeline plus confidence interval,” not a single percentage from marketing. From this also comes a lesson for everyone building LLM judges: the metric “judge found an error” says almost nothing about the judge’s ability to notice an omission, so in this work verification was assembled as an adversarial panel of different models, not as a single judge.
Why this matters for the industry
For the industry, this is a blow to the key argument of the category and simultaneously a map of a ready-made product gap. Since notes with a confirmed defect in this sample reached final physician signature, the signature did not work as a filter — and typical omissions concern precisely allergies and medications, i.e., classes that lead to clinical incidents. Teams need new control surfaces: a QA layer on top of existing scribes, a review interface with risk-ranked diff, and judges that catch omissions, not just extra facts. Clinic buyers get grounds to demand independent audits instead of demo numbers: the pipeline is open, the measurement can be repeated in days, and “error rate X%” without methodology loses comparability. A wave of third-party audits on open pipelines is likely: measurements will either confirm the order of magnitude of about 30% or show a spread — in both cases vendors will have to respond with methodology. The window for QA middleware, benchmark services for judges, and judge prompt tooling opens earlier than vendors implement their own built-in checks.
Why this matters for users
All materials are open and can be used today: the OmissionBench dataset with 618 verified findings is on Hugging Face under the CC BY 4.0 license, the pipeline code is published in the MIT repository composo-ai/omission-bench, and the measurement can be repeated via OpenRouter. Teams with a similar “meeting — structured note” pipeline can run their judges through this benchmark and find out whether their metrics catch omissions, not just factual errors. For the ordinary reader, the lesson is simpler: auto-generated medical records should be read selectively and carefully — first of all the blocks on allergies, medications, and the examination performed — and not signed on trust.
What is still unknown / limitations
The census was made on a single sample of 142 consultations and three specific deployed systems, so the obtained share cannot be automatically transferred to all products in the category. The headline “one in three” is just one point on the curve: the error rate estimate strongly depends on the verification standard. The census lists typical defect classes, but does not give their ranking by clinical severity and frequency, so considering this list a ready-made specification for review-UX is premature. Forecasts that adversarial audits and omission checking will become a standard of eval suites are an interpretation of a trend, not an established consequence of the article.
Sources
- One note in three: a verified census of three deployed AI scribes, and the instrument that counted it (arXiv:2608.31017)
- OmissionBench — dataset of 618 verified findings and paired tests for LLM judges (ComposoAI, CC BY 4.0)
- composo-ai/omission-bench — audit pipeline and judge benchmark code (MIT)
Author
Look at AI, editorial team