Adoption of AI scribes across Australian specialist practices is moving faster than the evidence underneath it. Vendors point to falling documentation times and lower burnout, and those claims are real. But a 2026 perspective in the ANZ Journal of Surgery makes a point that procurement teams and specialists should sit with before signing anything: the tools being sold here have not been tested here.

This matters because the way an AI scribe is built, and where a human sits in the workflow, turns out to be more important than the headline time saving. Here is what the current evidence actually says, and what it means when you are choosing a tool.

The validation gap nobody is advertising

In their review of the two large randomised controlled trials on ambient AI scribes, Dudi-Venkata and colleagues note that none of these tools have been externally validated in Australia or New Zealand. The trials that the industry leans on – Lukac et al. and Afshar et al., both published in NEJM AI in 2025 – were run in US health systems on predominantly younger, female, medical-specialty cohorts. The authors are explicit that this limits how far the findings generalise, particularly to surgical and late-adopting populations.

For an Australian specialist, that gap is not academic. Local validation has to cover more than documentation efficiency. It needs to account for governance and privacy obligations under Australian frameworks, transparency about where data is stored, and the auditability of AI-generated notes. A tool that performs well in California has not, on that basis alone, demonstrated anything about its fit for an ENT clinic in Melbourne or a renal practice on the Gold Coast.

The trials themselves were also short. Lukac et al. ran for two months and Afshar et al. for 24 weeks, and utilisation varied widely (between roughly 30 percent and 71 percent of eligible encounters). Short trials with self-selected early adopters are a starting point, not a verdict.

The three stages of AI documentation – and where the evidence actually sits

The ANZ paper frames the technology as evolving through three stages. Stage one is human-led, with tools like dictation and templates supporting a clinician who does most of the work. Stage two is mixed-initiative: AI converts the encounter into a draft, and a human reviews and edits it. Stage three is computer-led, where the system handles documentation autonomously and only asks for human input on exceptions.

The authors are clear that clinical practice is firmly in stage two, and that this is where the evidence supports it being. Stage three is not validated for routine use, and there are good reasons it should not be rushed into specialist settings.

This is the part of the conversation that gets lost in marketing. A workflow that keeps a human reviewing the output is not a transitional compromise on the way to full automation. For high-stakes documentation, it is the current evidence-based standard. Treating human review as friction to be engineered out is getting the evidence backwards.

Why “fully automated” is the wrong goal for specialist documentation

Specialist and surgical documentation carries decisions that do not tolerate quiet errors: consent discussions, procedural detail, perioperative plans, medication choices. The ANZ authors recommend additional verification safeguards specifically for these contexts, and argue that clinicians should retain direct responsibility for checking AI-generated content before sign-off.

Even in the favourable trial conditions, the notes were not flawless. Inaccuracies were reported as occurring occasionally, and the most common error types were omissions and structural problems rather than obvious transcription slips. An omission is exactly the kind of error that survives a quick glance and surfaces later, when a referring clinician acts on a letter that left something out.

There is also a liability point that no software disclaimer removes. The review is blunt about it: whoever signs the document is the one who carries responsibility for it, not the software. Whatever the tool generates, the doctor who signs owns it. A workflow built around that reality, rather than against it, is the safer commercial and clinical choice.

What this means when you choose a tool

Two practical distinctions follow from the evidence.

First, capture method matters. A 2025 systematic review in BMC Medical Informatics and Decision Making found word error rates ranging from around 0.087 in controlled dictation settings to over 50 percent in conversational, multi-speaker scenarios. Ambient scribes that listen to a live consultation are working in the high-error end of that range by design. Dictation-based input sits in the favourable end. If accuracy is the concern, how the audio is captured is not a detail.

Second, human-in-the-loop is a feature, not a flaw. The trials show benefit; they also show the need for ongoing oversight. A tool that produces a draft ready to proof, with the practice team completing and approving it, is aligned with what the research recommends rather than betting against it.

Where Dict8ion fits

Dict8ion is not an ambient AI scribe. It is built for patient letters, taking a doctor’s dictation and producing a formatted draft ready to proof, with a custom medical dictionary and a human-in-the-loop workflow. Doctors keep using their existing dictaphone, voice app, or the Dict8ion mobile app, so nothing about the way they work has to change.

That design is not a coincidence. Dictation-based capture sits in the low-error end of the evidence. Human review keeps a clinician accountable for the final document. And as an Australian-built product, Dict8ion is designed for the regulatory and clinical environment that the imported tools have not been validated against.

The evidence does not say AI has no place in specialist documentation. It says the current standard is AI that drafts and humans who verify – and that the local validation work still has to be done. Choosing a tool built around that reality is the cautious read of the research, not the cautious read of the marketing.

Not an AI scribe. smarter. faster. more organised. patient letters. To see how Dict8ion works in practice, get in touch

Dict8ion.ai | AI speed. Human accuracy.

References

• Dudi-Venkata NN, Hastie I, Warrier S, Heriot A, Reddy S. Ambient Artificial Intelligence Scribes: A Reality Check – Insights From Two RCTs. ANZ Journal of Surgery. 2026. DOI: 10.1111/ans.70699
• Lukac PJ, Turner W, Vangala S, et al. Ambient AI Scribes in Clinical Practice: A Randomized Trial. NEJM AI. 2025;2(12). DOI: 10.1056/AIoa2501000
• Afshar M, Baumann MR, Resnik F, et al. A Pragmatic Randomized Controlled Trial of Ambient Artificial Intelligence to Improve Health Practitioner Well-Being. NEJM AI. 2025;2(12). DOI: 10.1056/AIoa2500945
• Ng JJW, Wang E, Zhou X, et al. Evaluating the performance of artificial intelligence-based speech recognition for clinical documentation: a systematic review. BMC Medical Informatics and Decision Making. 2025;25(1):236. DOI: 10.1186/s12911-025-03061-0