AI Governance in Healthcare: Navigating the Agentic Era
Image: Unsplash
As AI’s capabilities race ahead and clinicians and health managers start to experiment with them, it’s vital that leaders put in place AI governance and oversight that balance the desire to leverage technology for greater productivity with the requirement to safeguard patient safety, confidentiality, and the motivation and skills of the workforce. Otherwise, well-meaning pilots will backfire.
Governance needs stretch from the obvious—such as preventing identifiable patient data from being entered into LLMs—to the more subtle requirement of ensuring human clinicians remain firmly in the loop on high-consequence decisions, maintaining full accountability rather than delegating by default to a machine.
The Emerging Era of Agentic AI
The emerging era of agentic AI in healthcare adds to these leadership challenges:
Beyond Short Demos: AI agents must be thoroughly assessed for both technical and clinical reliability across a wide range of real-world scenarios.
Independence & Accountability: With greater autonomy comes greater accountability; regardless of how intuitive the tools become, humans must bear final responsibility.
Multi-Agent Orchestration: In the future, as multiple agents interact around a single patient, orchestrating their workflow while keeping the patient's best interests at the heart will demand clear transparency and strict guardrails.
Regulatory Sandboxes & Data Liberation
To ensure oversight keeps pace with emerging technology, clinical AI sandboxes have an important role to play. The NHS in London and the UK regulator, the MHRA, have recently taken a valuable initiative in this direction.
As observed in traditional digital health, deep study of specific use cases—evaluating both clinical and economic outcomes—is essential to persuade sceptical funders and providers to allocate necessary budgets and clinical time as technology scales.
Releasing health data from silos presents a similar leadership hurdle:
Beyond Technical Hurdles: Unlocking EMR data is less about technical middleware and more about making a compelling professional and public case.
Overcoming Protectionism: While patients typically embrace de-identified data sharing for broader medical benefit, [healthcare workforce transformation] is often held back by professional protectionism, which professional bodies must actively address.
Preparing the Healthcare Workforce for Human-Machine Partnership
The evolution of the medical workforce driven by AI is now gaining critical attention:
Relieving Operational Burden: AI has the potential to eliminate drudgery and free up clinical time for high-value care (the rapid adoption of ambient scribes being a prime example).
Reinventing Medical Education: Medical school curricula and continuing professional education (CPE) must adapt to incorporate AI tools, alert professionals to inherent drawbacks, and reinforce a true human-machine partnership.
Currently, very few healthcare leaders are fully shaping up to this organizational challenge.
The Change Management Imperative
In principle, the advent of AI should accelerate both the generation of new healthcare technologies and the emergence of consistent frameworks for real-world evidence and value-based AI adoption.
However, adopting new technology is ultimately an exercise in behaviour change. Innovation destroys old, comfortable routines. Executive leaders must actively guide their organisations through this change management journey rather than delegating the entire process to technical leadership—be it CIOs, CCIOs, or the newly minted Chief Clinical AI Officers (CCAIOs).
