Will regulation redraw the AI medical documentation landscape?
A wave of AI-based scribes is flooding the healthcare market. Will regulation shake it up? An educated guess.
AI scribes are having their VC moment. Just since posting our scribe market map 2 weeks ago, we’ve added another 10 startups to our list. That’s 60+ AI scribe startups based in Europe alone, not counting scale-ups and incumbents.
Everyone agrees on ambient note-taking as the first LLM use case to be adopted in clinical practice. Yet, one risk both founders and customers are divided about is regulation. AI scribes currently operate in a grey zone and a single policy shift could redraw the map.
So, what are the odds of regulation?
The past months, I spoke with over 30 scribe founders, more than 20 customers and dozens of VCs about this. Views were heterogenous - and let’s be honest, most startups don’t truly prep for regulation.
In theory, scribes are medical devices
To cut through the fog, I had a chat with Hugh from Hardian Health last week. They were one of the earliest voices advocating for regulatory oversight of AI scribes and btw write their own newsletter on regulatory topics.
Here’s a quick overview of the arguments.
Contra Regulation:
Most vendors offer human-in-the-loop products, prompting physicians to check the output. This is positioned as the ultimate risk mitigation measure.
To underline this argument, some vendors use RAG or explainability / transparency measures to point to the underlying sources.
Besides, clinicians have always used Google or other non-compliant tools informally to access knowledge. They assume the risk of false information themselves - and it is broadly accepted.
In a pragmatic view, the incredible surge of LLM tools will prevent regulators from catching up anyways. As an AI scribe startup, you could simply bet on this lag.
Pro Regulation:
Despite advertised as pure “documentation” solutions, LLM-based scribes usually filter and select information. Otherwise they wouldn’t save time. Any filtering of information can indirectly influence clinical decisions.
LLMs make frequent and sometimes subtle transcription or content errors (see also one of my recent posts). These errors can translate into clinical decisions. And no physician really checks every word. What if the scribe notes the wrong medication dose?
Even if LLMs achieve human performance: Doctors and human scribes are already heavily regulated. LLMs performing similar functions shouldn’t be exempt.
By nature LLMs are general-purpose tools, which are difficult to restrict to one use case and unpredictable by nature.
The conclusion: Let’s be honest, AI scribes are technically (low-risk?) medical devices. They make mistakes and influence clinical workflows and clinical decisions.
All counter-arguments are mostly of pragmatic nature and overestimate user’s time to double check LLM outputs…
Practically, regulation might not matter
After a long time in a legal grey area, regulators recently started moving.
Silently, the MHRA in the UK published its first formal guidance on the topic last month - originally about digital mental health, just covering scribes on the side. They classified them as medical devices stating their “high functionality” combined with a medical purpose.
Now, AI scribes are officially medical devices in the UK. First startups have reacted accordingly btw, both Tortus and Heidi have recently self-certified as class I.
To assess this regulation’s impact, we need to ask a follow-up question: Will risk class I be sufficient or will regulators double down and demand (and enforce) class II and above?
Here’s the catch: If you assume a low-risk profile, class I and thus self-certification is sufficient for now. That means no waiting for notified bodies, no large trials upfront, and very limited oversight. You’re expected to document performance and they can audit you, but even if they do, you will be granted time to correct. It means a magnitudes smaller entry barrier (perhaps 1-2 years and several 100k in costs) than a class II or class III medical device.
What about other regions? In the US, the FDA is quiet, but self-certification seems a likely path, especially under the current regulation-averse administration. The EU is often stricter, but still vague on how they will handle scribes. With the EU AI act kicking in, a certain regulatory effort will be required anyway.
Startups are placing their bet
It’s still difficult to assess market impact. From my conversations, I can deduct two scenarios:
A) The “self-serve” scenario
It’s possible that regulators de facto accept class I regulation or maintain a grey area for years, choosing a pragmatic approach to save resources. In this scenario, regulation will continue to play a subordinated role. The better startups are already tracking technical performance to convince enterprise buyers and can use it to file for class I. Under-prepared companies might take a hit, but it’s not a market-changing moment. Some startups intentionally bet on this scenario and focus resources on distribution.
B) The “purge” scenario
But what if class II+ becomes a necessity? We would see a wash-out of AI scribes that have ignored regulatory requirements or were lazy in their documentation. In-house solutions and “tourist” scribes from other verticals would need to pull back. I know that some European startups are prepping for exactly this scenario. They see regulation as their long-term moat, and some have begun filing for class II.
Either way, regulatory strategy has become a must-have DD item for any VC entering the space.
Startups and investors on both sides are placing their bets and I’m here to watch the race.
Happy Monday,
Lucas