What happens when ‘AI kills a patient’?

Abstract

What happens when problems with an Artificial Intelligence (AI) system are implicated in the harm, or death, of one or more patients? What will we do? The news headlines almost write themselves: ‘AI kills patient’. Although the headline may be hyperbole, the scenario is not. AI technologies are being rapidly deployed across healthcare and are taking on key parts of clinical work, from treatment planning to ambiently capturing clinical conversations and automatically generating medical records.¹ In England, accelerating this roll-out is a national priority²: healthcare organisations are required to implement technologies such as AI scribes ‘at pace’³ as the Government strives ‘to give the NHS the most AI-enabled workforce in the world’.² Considerable benefits are likely to be realised from these technological shifts, particularly if they provide opportunities to reimagine how entire systems of care are organised and delivered.⁴ But all new technologies bring with them new forms of risk and the potential for unwelcome surprise (such as, for instance, the unexpected harms associated with the introduction of metal-on-metal hip implants). Realising the benefits of AI in healthcare will therefore involve carefully managing risks and uncertainties – both the direct impacts that AI will have on the safety of care and the indirect impacts that may arise from high-profile AI safety events and the resulting damage to public and professional trust (Box 1).

Box 1.

Exploring and imaging the risks of AI in healthcare

Plausible predictions: systemic risks of ambient voice technology

The race to deploy

Ambient Voice Technologies (AVTs), or ‘AI scribes’, use AI models to listen to clinical conversations and produce various forms of clinical records. AVTs are one of the most rapidly expanding applications of AI in healthcare, and NHS England has set out formal expectations that ‘providers should deploy ambient voice technology (AVT) at pace’.³ Hopes are high that this will lead to considerable efficiencies in the production of clinical documentation: the NHS ‘10 Year Plan’ makes the ambitious claim that implementing AVT in all emergency departments across England could create capacity to see over 9000 more patients every day.²

AI reliability gaps

AVTs use sophisticated Large Language Models (LLMs) to automatically transcribe clinical conversations and to generate clinical records in various formats. LLMs are sophisticated statistical algorithms trained on vast quantities of text data to generate plausibly representative language. These models have inherent and widely acknowledged reliability issues. They are non-deterministic, so even identical inputs will generate different outputs. They are subject to omissions, so key information discussed in a consultation may be left out of the clinical note generated by an AVT. And they are subject to hallucinations, where plausible but entirely fabricated clinical information (e.g. non-existing conditions or patient history) or incorrect details (e.g. wrong medication doses or location of pain) may be generated and included in the output produced by the AVT.⁵

Sociotechnical risks

AVT outputs must be carefully reviewed (and typically edited) by clinicians to ensure accuracy and completeness. Relying on human verification ensures a clinician remains responsible for the clinical record but also creates a number of risks. Human cognition is vulnerable to vigilance decrements, where performance can degrade rapidly on repetitive monitoring or checking tasks.⁶ Similarly, people are vulnerable to automation bias, a tendency to over-trust and under-scrutinise the outputs of automation, and confirmation bias, which is a tendency to seek evidence that confirms current beliefs and memories and downplay discrepancies. Moreover, the hallucinations and errors produced by LLMs can also be particularly hard to identify: the text produced is typically plausible, detailed, professional and authoritative, irrespective of its accuracy, so any errors are hidden within a fluent and credible narrative. Instances where patients have been incorrectly referred for serious conditions they do not have due to incorrect AI-generated documentation have already been reported in the press⁷ (explained away, of course, as an isolated human error).

A plausible recipe for risk incubation

Combine all this with the time pressures, resource constraints, interruptions, distractions and fatigue that can be common features of clinicians’ working environment; mix with strong policy incentives to rapidly implement AI and save time and costs; garnish with the tantalising promise of offloading much of the cognitive and administrative burden of clinical documentation and leave to simmer. Patients are also, of course, increasingly likely to be consulting consumer-facing AI tools before, after (and perhaps even during) clinical consultations, adding further AI-generated material into the processes of care. Without careful and proactive management of these complex sociotechnical risks, this is potentially a recipe for the future widespread manufacture and invisible insertion of inaccuracies, errors and fictions throughout the clinical record. In the short term, this poses risks to individual patients, who may receive inappropriate care or suffer harm based on inaccuracies in their records. It also poses risks to clinicians, who become liable for capturing and correcting the errors produced by AI. In the long term, this also poses a more fundamental and systemic risk: a gradual erosion in the integrity of and trust in electronic health records, the ‘ground truth’ of a healthcare system and the care it provides.

These challenges have already been confronted in other sectors: the first pedestrian fatality involving a self-driving car in 2018 prompted a high-profile media frenzy and organisational crisis and fed wider public distrust for many years.⁸ Waiting for disaster to strike before fully engaging with the risk implications of a new breed of technology is a hazardous strategy. In healthcare, it risks the potential benefits of AI being curtailed by the damaging distraction of hotly contested blame games when things go wrong, and the poorly formulated knee-jerk ‘Pavlovian policy’ that can follow.⁹ This is why leaders of prominent autonomous vehicle companies now publicly discuss the challenge of ‘when’, not if they will need to manage the consequences of adverse events.¹⁰ Taking safety seriously means pre-emptively and openly engaging with these issues upfront and building the organisational and institutional capabilities that will be needed when (not if) bad things happen.

What happens when ‘AI kills a patient’ is therefore a critical question that deserves serious consideration right now, well before clinicians, healthcare leaders, policymakers, regulators and patients are forced to confront the scenario in real time. So, what will happen? There are at least four key sets of interconnected questions that need to be carefully considered and prepared for, and which are complicated by the complexities of AI (Table 1). Firstly, how will healthcare systems reliably identify and detect when AI technologies are implicated in safety events in the first place? Secondly, how will stakeholders collaborate, disclose and share relevant information? Thirdly, how (and by whom) will the events, technologies and underlying risks be investigated, analysed and explained? And fourth, how will accountability be negotiated and improvement be coordinated to ensure system-wide learning?

Table 1.

Challenges in the development of AI safety and learning capabilities.

Principles, challenges and capabilities for AI safety management and learning
Risk management principle	AI safety and capability challenges
1. Identification and detection	How will we know if an AI technology has contributed to patient harm, or is posing an ongoing risk to the safety of patients? a. How will continuous monitoring of AI safety and performance be performed and organised, and who should coordinate and supervise these processes? b. How will specific sources, patterns and indicators of AI risk be defined and monitored, and who will determine and detect appropriate thresholds for further action? c. How will healthcare staff and service users be supported to understand, recognise and report concerns related to AI safety and performance, and who will review these signals?
2. Collaboration and information	What data will we need to understand AI risks, who will have it and how will we collaborate to share and use it? a. How will all the sources and forms of relevant AI safety data be defined, and how and by whom will these data be routinely collected and maintained? b. How will AI safety data and evidence be shared and accessed by all the actors who collect, hold and need it, and what mechanisms and agreements will assure access? c. How will stakeholders collaborate and contribute expertise to collate and integrate AI safety data, and who will critically interrogate and validate this evidence?
3. Explanation and investigation	How will we understand the sociotechnical patterns of risk associated with AI in complex care processes, and what analytical strategies and expertise will be needed? a. How will the complex sociotechnical sources of risk (human, technical, organisational and clinical) be routinely and rigorously analysed across different levels and arenas of the healthcare AI ecosystem? b. How will multidisciplinary teams expert in AI safety investigation and analysis be created and supported, how will these teams be organised and where will they be based? c. How will systemic AI safety analysis be conducted in ways that integrate complex domains of knowledge, and develop clear and actionable explanations of sources of risk?
4. Accountability and improvement	How will we determine responsibility and accountability for AI safety, and how will improvement be organised and coordinated across the healthcare system? a. How will responsibility for AI risks and safety improvement be allocated, monitored and governed across all relevant stakeholders, and who will oversee this? b. How will AI safety improvement and risk mitigation activities be targeted and coordinated across the system, and who will support these improvement efforts? c. How will AI risks and safety insights be shared across the system, and who will ensure that improvements and mitigations are effectively implemented?

Identification and detection

Determining whether an AI technology may pose an ongoing risk to patient safety, or has contributed significantly to a safety incident, will be challenging. Healthcare organisations and the work of delivering care are enormously complex, both socially and technologically. Harmful events, and risks to patient safety, are similarly complex and are never the result of a single, simple cause. Moreover, safety issues associated with the use of technology are already widely under-reported.¹¹ Detecting the signs of emerging problems with AI – and identifying when AI is implicated in safety events – is therefore unlikely to be obvious. It will depend on carefully defining and monitoring specific patterns or signals of risk which may represent, for example, indications of degrading clinical vigilance, the implementation of immature automation or gaps in validation and governance.¹² The precise nature of these sociotechnical patterns and associated indicators will depend on how particular AI technologies are implemented and used in different care settings, as well as a clear understanding of the types of harmful outcome that need to be protected against in each specific context (Box 1). Reliably detecting and identifying AI-related sources of risk will therefore require moving beyond safety management approaches organised around the episodic and ad hoc reporting of incidents to systems that support continuous monitoring of the performance and use of AI in real-world settings – allowing both the prospective detection of early signs of emerging problems, and the retrospective examination of how AI may have contributed to serious failures, akin to the use of ‘black box’ data recorders in other high-automation sectors.¹³

Collaboration and information

Understanding whether and how AI technologies are creating serious safety risks – or have contributed to actual harm – will depend on a variety of different groups and organisations collaborating to share and analyse a diverse array of data.¹⁴ The evidence needed to properly analyse the safety impact of AI in real-world healthcare settings will encompass clinical data such as clinical outcomes and care processes, experiential data such as insights of patients and practices of staff, management data such as audit logs and governance records, organisational data such as clinical pathways and local policies, technical data such as model version history and training data specifications and regulatory data such as validation studies and hazard analyses. These data will all be essential to understanding how the design, governance, implementation and use of AI impacts the safety of patients. A considerable challenge is that these data are held by many different parties, some of whom may be reluctant or fearful to share data – particularly in the aftermath of a serious event. The requisite expertise and knowledge needed to interpret these data are similarly widely distributed. Effectively understanding and addressing AI-related risks will therefore require formal agreements, organisational mechanisms and cultural norms that support genuine collaboration and shared access to all relevant information,⁸ while managing confidentiality concerns, commercial sensitivities and fears of blame and liability.

Explanation and investigation

Explaining how and why AI has contributed to patient harm, or is posing an ongoing threat to safety, will require sophisticated analytical capabilities and investigative strategies. The algorithms and models that underlie advanced AI are inherently hard to explain.¹⁵ Analysing the performance of AI requires close technical scrutiny (such as of training data and model architecture) and detailed clinical analysis of AI outputs (such as of case-level validation and sub-group performance). Explaining the real-world safety impacts of AI also critically requires extensive investigation of the ‘real-world’ that AI is embedded within: that is, the complex sociotechnical systems that AI is necessarily a part of.⁸ Analysis will have to disentangle many complex issues: how an AI tool was designed and implemented in a specific setting; how performance was monitored over time; how AI influenced and shaped human judgement and decision-making; and how organisational and cultural issues such as time pressure, training and staffing levels affect use of and reliance on AI. Fully explaining the safety of AI – or how and why AI has contributed to harm – will therefore depend on extensive investigative and analytical work. This work will need to be conducted by expert multidisciplinary teams experienced in sociotechnical systems analysis and systemic investigation, and capable of interpreting and integrating complex clinical, technological, statistical, psychological, organisational and regulatory evidence; and will need to genuinely engage with and represent patient experiences and insights. Given the scale and complexity of many AI technologies, it is also likely that AI tools will be key in supporting these analysis and learning processes: these AI tools will in turn require careful review and poses further complex questions around trustworthiness and AI explainability.^12,13

Accountability and improvement

Improving safety after AI-related harm occurs, or when new risks are discovered, will depend on accurately determining which actors, activities and technologies contribute to and are responsible for specific risks, and allocating clear accountabilities for change and improvement. The work of organising and accounting for improvement is often difficult and complicated in healthcare.¹⁶ AI will complicate it further.¹⁷ Responsibility for AI safety will necessarily be distributed within a complex network that includes clinicians using it, organisations governing it, managers purchasing it, regulators assuring it and manufacturers designing and building it. Focusing on any single party – such as simply blaming frontline staff and turning them into convenient ‘liability sinks’¹⁷ – will prevent the inherently systemic and integrated approach that will be needed to coordinate learning and improvement across the healthcare system. Addressing AI risks, and improving AI safety, will depend on clear allocation of active accountabilities for improvement¹⁸ that encompasses the whole AI ecosystem: spanning design, regulation, validation, implementation, governance and use. It will be equally important to organise system-wide learning from AI risks, particularly given the ambitious plans to rapidly deploy AI at scale.² Risks or harms that emerge at one site should be learnt from before they are repeated elsewhere. This will require an integrated approach, and coordinating bodies or agencies, to access and aggregate risk data, investigate and analyse systems, convene and connect stakeholders and target improvement-focused recommendations to any and all relevant actors across the system, and hold them to account for achieving those improvements.¹⁶

Building an AI learning infrastructure: collaborative, systemic and integrated

How can healthcare build the organisational and institutional capabilities for managing AI safety that will both minimise the risk, and prepare for the reality, of bad things happening? All healthcare organisations and agencies, at all levels of the system, will need to engage with these challenges to some greater or lesser degree. The questions set out in Table 1 provide an initial structure for healthcare organisations, leaders, regulators and policymakers to explore the key challenges involved in managing the safety of AI in healthcare, and for assessing current and future capabilities for governing systemic AI risks.¹⁹ The questions provide a framework that could guide assessments of what capabilities may be needed in future, and how current capabilities may be found lacking in the face of a major AI safety crisis: an organisational ‘pre-mortem’ or ‘premeditatio malorum’ to assist in imagining and preparing for future challenges. Importantly, individual organisations and the healthcare system as a whole will need to build joined-up, system-wide and proactive capabilities for learning that match the considerable speed, scale, complexities and risks of AI deployment. Learning from one incident, one device, one patient and one organisation at a time will be both too slow and grossly insufficient to meet the scale of the systemic challenges – as well as the transformative potential and policy ambitions – associated with integrating AI across the healthcare system.

So, what might a learning infrastructure for AI safety look like? Many of the core capabilities can be built on and extend existing safety infrastructure.⁸ Current safety occurrence reporting programmes can be expanded to better accommodate and encourage the detection and disclosure of AI-related safety issues such as the UK’s Yellow Card scheme,²⁰ while safety investigation processes and frameworks (such as England’s Patient Safety Incident Response Framework) could be more purposefully oriented to the analysis of the sociotechnical sources of AI risk and resilience. ^8,12 However, new capabilities will also need to be developed, based on a clear understanding that embedding AI throughout healthcare will generate new forms of systemic and sociotechnical risk (Box 1). These systemic AI risks will need to be recognised and engaged with as distinct objects of governance, which require dedicated institutional attention and specialised roles and resources that reach beyond the silos currently separating the regulation of healthcare products, providers and professionals. The safety challenges associated with AI necessarily involve complex and systemic interactions between technologies, organisations and people. AI governance capabilities similarly need to be systemic, integrated and focused on assuring the safety of entire sociotechnical systems – not just one component or one element of those systems at a time.

This points to many important practical implications which require further intensive and rapid exploration.^12,14,19 One implication, for example, is that the deployment of AI technologies within healthcare organisations should be considered the beginning of safety assurance processes, not the end. The real-world performance, use and impact of AI will need active and continuous safety monitoring that is attentive to the complex human, sociotechnical and organisational context it is embedded in, much as airlines continuously monitor the complex sociotechnical reality of aircraft operations.¹³ Another implication is that the competencies, skills and performance of AI technologies and healthcare professionals will increasingly need to be considered in combination, as distributed cognitive systems – just as other sectors that incorporate sophisticated technologies of automation have long designed, trained, maintained and assessed professional competencies and technological systems as complementary and mutually supportive contributors to the same sociotechnical system.^14,21 A further implication is that learning infrastructures that support system-wide engagement, rapid sharing and coordinated action will need to be rapidly expanded and supported. Learning infrastructure is developing in the form of large-scale collaborative AI communities, such as the Ambient Voice Innovation and Development (AVID) community which supports open learning and peer-to-peer sharing around practical challenges with ambient voice technology.²² And it is developing in the form of national bodies with a dedicated remit for system-wide learning, such as England’s independent Health Services Safety Investigations Body (HSSIB) which is uniquely empowered to conduct learning-focused safety investigations that can carefully examine the role of every actor across the healthcare system, from technology developer to regulator to healthcare provider to training body.²³ The challenge will be to rapidly scale and support these learning infrastructures in ways that match the ambitious pace of AI deployment across the healthcare system – as well as avoiding inadvertent policy decisions² which threaten to degrade or remove the limited systemic investigation and learning infrastructures that are starting to develop.²⁴ Without strong and well-established capabilities for systemic investigation, expert analysis and collaborative learning, the enormous potential of AI in healthcare may be derailed when the first AI safety events that harm a patient begin to hit the headlines.

Footnotes

Acknowledgements

Views expressed here are the author’s own and do not reflect the beliefs or opinions of any organisations the author is affiliated with. CM is supported by the NIHR Yorkshire and Humber Patient Safety Research Collaboration.

Declarations

ORCID iD:

Carl Macrae

References

National Academy of Medicine. Generative Artificial Intelligence in Health and Medicine: Opportunities and Responsibilities for Transformative Innovation. Washington, DC: The National Academies Press, 2025.

Department of Health and Social Care. Fit for the Future: The 10 Year Health Plan for England. London: Stationery Office, 2025.

NHS England. Medium Term Planning Framework: Delivering Change Together 2026/27 to 2028/29. London: NHS England, 2025.

Wachter

A Giant Leap: How AI Is Transforming Healthcare and What That Means for Our Future. New York: Portfolio, 2026.

Asgari

Montaña-Brown

Dubois

, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. NPJ Digit Med 2025; 8: 274.

Adler-Milstein

Redelmeier

Wachter

RM.

The limits of clinician vigilance as an AI safety bulwark. JAMA 2024; 331: 1173–1174.

Nolan

UK health service AI tool generated a set of false diagnoses for one patient that led to him being wrongly invited to a diabetes screening appointment. Forbes. See https://fortune.com/2025/07/20/uk-health-service-ai-tool-false-diagnoses-patient-screening-nhs-anima-health-annie/ (last checked 27 October 2025).

Macrae

Learning from the failure of autonomous and intelligent systems: accidents, safety and sociotechnical sources of risk. Risk Anal 2022; 42: 1999–2024.

Lodge

Hood

Pavlovian policy responses to media feeding frenzies? Dangerous dogs regulation in comparative perspective. J Conting Crisis Manag 2002; 10: 1–13.

10.

O’Kane

Waymo’s co-CEO on the challenge of scaling robotaxis safely. TechCrunch. See https://techcrunch.com/2025/10/27/waymos-co-ceo-on-the-challenge-of-scaling-robotaxis-safely/ (last checked 27 October 2025).

11.

Martin

Ghafur

Cingolani

, et al. The effects and preventability of 2627 patient safety incidents related to health information technology failures: A retrospective analysis of 10 years of incident reporting in England and Wales. Lancet Digit Health 2019; 1(3): e127–e135.

12.

Macrae

Managing risk and resilience in autonomous and intelligent systems: Exploring safety in the development, deployment, and use of artificial intelligence in healthcare. Risk Anal 2025; 45: 910–927.

13.

Winfield

AFT

Winkle

Webb

Lyngs

Jirotka

Macrae

. Robot accident investigation: A case study in responsible robotics. In: Cavalcanti

Dongol

Hierons

Timmis

Woodcock

(eds.) Software Engineering for Robotics. London: Springer, 2021.

14.

Macrae

Governing the safety of artificial intelligence in healthcare. BMJ Qual Saf 2019; 28: 495–498. DOI: 10.1136/bmjqs-2019-009484

15.

Kim

Gadgil

Lee

SI.

Transparency of medical artificial intelligence systems. Nat Rev Bioeng 2025; 4: 11–29. DOI: 10.1038/s44222-025-00363-w

16.

Macrae

Why safety recommendations don’t (always) work: Understanding the problems and exploring some solutions. J Patient Saf Risk Manag. Epub ahead of print 9 February 2026. DOI: 10.1177/25160435261421366.

17.

Lawton

Morgan

Porter

, et al. Clinicians risk becoming ‘liability sinks’ for artificial intelligence. Futur Healthc J 2024; 11: 100007.

18.

Macrae

From Blade Runners to Tin Kickers: what the governance of artificial intelligence safety needs to learn from air crash investigators. AI Society 2023; 38(5): 1971–1973. DOI: 10.1007/s00146-021-01246-5.

19.

Macrae

Systemic AI safety: challenges and capabilities and assessment. See https://publish.obsidian.md/riskresilience/Capabilities+assessment (last checked 27 February 2026).

20.

Medicines and Healthcare products Regulatory Agency. Yellow Card: Software, apps and Artificial Intelligence (AI) reporting. MHRA 2025. See https://yellowcard.mhra.gov.uk/software-apps-and-artificial-intelligence (last checked 8 December 2025).

21.

Macrae

. Close Calls: Managing Risk and Resilience in Airline Flight Safety. London: Palgrave, 2014.

22.

The Healthcare Improvement Studies (THIS) Institute. Ambient Voice Innovation and Development community (AVID). THIS Institute 2025. See https://ambientvoice.thiscovery.org (2025).

23.

Wiig

Macrae

Introducing national healthcare safety investigation bodies. Br J Surg 2018; 105: 1710–1712.

24.

Macrae

Failing to Learn? The NHS is losing its capacity for system-wide safety investigation. J R Soc Med 2025. DOI:10.1177/01410768251366879.