Human in the Loop: Process Design for AI-Assisted SaMD

AI coding assistants and agentic development tools have arrived in software engineering, including among manufacturers of Software as a Medical Device, or SaMD. Expectations of rapid efficiency gains now meet a differentiated evidence base. McKinsey reports that almost 30% of companies saw productivity decline after introducing agentic AI tools [1]. At the same time, a small group achieves clear quality and speed gains. Their lever is consistent process redesign [2].

For SaMD manufacturers, the message is clear: the regulatory risk does not lie in AI-generated code itself. It lies in a development process designed for human authorship that merely adds AI on top. Human in the Loop needs to be embedded as a defined decision architecture in the quality management system.

Executive Summary

1. The productivity paradox: more code, not more software

The gap between activity and outcome is now well documented. McKinsey cites a study in which AI tools increased coding activity by 180%, while shipped releases rose by only 30% [1]. Acceptance among developers also remains limited. Globally, 46% of developers distrust the accuracy of AI tools, 33% trust them and only 3% report high trust in their results [1].

That scepticism has a technical basis. A CodeRabbit analysis of 470 open-source pull requests, including 320 AI co-authored and 150 created solely by humans, reported the following findings [3]:

The study did not directly confirm authorship. It identified AI involvement through pull request signals and assumed human authorship for the remaining pull requests [3]. The direction of the finding remains important. Missing null checks, guardrails and complete exception handling are critical in safety-relevant medical device software.

2. What does not change from a regulatory perspective

Manufacturers do not need a new legal framework. They need consistent application of the existing one. IEC 62304 defines lifecycle requirements for medical device software. The standard does not assess who wrote a line of code. It assesses whether requirements, architecture, verification and release are controlled and traceable [14]. The Johner Institute has made the same point for conventional code generators: generated code can undergo the same quality assurance measures as handwritten code [4].

There is also a tooling layer. LLM-based systems count as computerised systems. Where manufacturers use them in QMS processes, they fall under ISO 13485 requirements [5]. The object of validation is not the model as an abstract technology. It is the computerised system in its defined intended use, context and data flow. Validation effort is set on a risk basis.

Using a coding assistant does not automatically make a SaMD an AI system under Regulation (EU) 2024/1689. The decisive question is whether the product itself contains AI functionality. For AI-based medical devices, the MDR or IVDR and AI Act apply in a complementary way. The joint AIB and MDCG guidance recommends integrating necessary testing, reporting and documentation processes into existing MDR or IVDR procedures wherever appropriate [13].

Standardisation will not provide near-term relief. According to the Johner Institute, the next milestone for IEC 62304 is expected to be a second Committee Draft, rather than the second edition. An FDIS is expected only in 2028 and publication no earlier than 2029 [6]. The draft contains an informative annex on AI-enabled health software [7]. Manufacturers should therefore establish their answers in the QMS now.

3. Human in the Loop in practice: where people decide

Human in the Loop is often understood as the broad idea that a developer checks the output. The approach becomes regulatory robust only when the organisation defines which qualified person takes which decision at which lifecycle point and how that decision is documented. The European Commission explicitly identifies human oversight as a requirement for high-risk AI in healthcare [12].

This view aligns with McKinsey's observation that roles are changing. Developers need judgement, code-review and supervision skills to steer agents. The emphasis moves towards architectural coherence, domain modelling and AI supervision [8].

4. Process design: five building blocks

The companies that genuinely benefit differ less in their choice of tools than in their operating model. The top quintile achieves 16% to 30% improvements in productivity, time to market and customer experience, together with 31% to 45% quality gains. Simply providing AI tools changes little [2]. Leading organisations build verification, control and measurement systems that keep pace with faster ways of working [9]. For SaMD, this translates into five building blocks:

The fifth building block is frequently underestimated. The AI guideline developed by the Johner Institute together with TÜV SÜD asks manufacturers to identify and document competency requirements for each role within the QMS scope that is directly or indirectly connected with AI [10].

5. Deployment matrix by safety class

Not every software component requires the same degree of restraint. The following matrix provides orientation for an internal procedure. In view of the forthcoming edition of IEC 62304, the draft is expected to replace classes A, B and C with Levels I and II [11].

Conclusion

AI-generated code is neither taboo nor a shortcut for SaMD manufacturers. The standards are technology-neutral enough to accommodate it. They require specified intent, traceable decisions and robust verification. McKinsey warns that agentic software development without a systematic approach can lead to unintended results [1]. In a regulated setting, that creates audit findings, rework in technical documentation and avoidable risks for patients.

Manufacturers should prioritise three actions:

Organisations that establish these foundations use AI through their quality management system. That is where the competitive advantage lies.