The possibility of AI-induced medical manslaughter: Unexplainable decisions,epistemic vices,and a new dimension of moral luck

Introduction

The National Health Service (NHS) is under immense financial pressure and clinicians and the most senior management have warned about the risk this presents to patient safety. ¹ At the same time, some legal responses to errors have been accused, by the medical profession, of being uncompromising and lacking in appreciation of context. ² The continuing use of the criminal law is hard to reconcile with current policy, which aims to foster an open and transparent culture of reporting accidents and learning lessons. ³ This tension reached a new level of intensity with the General Medical Council’s decision in 2018 to pursue an appeal to the High Court to strike off Dr Bawa-Garba following her conviction for gross negligence manslaughter (GNM). ⁴

To briefly contextualise the law in the jurisdiction of England and Wales, GNM is a common law offence, which is committed when the prosecution can prove that the defendant owed a duty of care to the victim, that they breached that duty of care by their conduct, and that this breach caused the victim’s death. In addition, the prosecution must prove that the defendant’s conduct was so grossly negligent that it amounts to a criminal offence. ⁵ In the House of Lords case of R v Adomako, ⁶ which is still the leading authority, the court held that the determination of ‘grossness’ was largely a matter for the jury. ⁷ Subsequent cases in the Court of Appeal have tried to capture the threshold in more precise language: perhaps most notably Leveson J, in both R v Sellu ⁸ and R v Rose ⁹ (and now adopted in R v Broughton ¹⁰ ), described ‘gross’ as ‘truly exceptionally bad and was such a departure from the standard [of a reasonably competent doctor] that it consequently amounted to being criminal.’ ¹¹

The storm around the case of Dr Bawa-Garba had been gathering over a number of years, largely as a result of an increase in referrals to the Crown Prosecution Service (CPS), ¹² increased prosecutions, ¹³ an arguably lower threshold for initiating prosecutions, with tougher sentences, ¹⁴ and professional sanctions. ¹⁵ At the same time, the public, police, and sections of the media continue to be guided by the belief that healthcare practitioners (HCPs) can be punished into safer care. ¹⁶ This article makes no claims about the appropriate level of prosecutions, but the flurry of medical manslaughter cases during the 2010s only represents the tip of the iceberg; interviews under caution and referrals to the CPS cause immense stress to HCPs: the very low likelihood of an eventual conviction, or the prospect of a successful appeal may offer little comfort to clinicians. As Quick argues, ‘the process is the punishment.’ ¹⁷ The medical profession has become increasingly worried, and as noted by Prof Don Berwick, ‘fear is toxic’ for clinicians and their patients. ¹⁸ As a result, The Williams Review and Hamilton Review have both called for fairer systems and procedures around the use of the criminal law so that HCPs may operate ‘without fear of retribution’. ¹⁹ It remains unclear whether the fears from a couple of years ago are starting to recede, partially as a result of the perplexing developments within recent case law.

The introduction of artificial intelligence (AI) systems presents a novel perspective on this familiar reform issue: the mens rea of involuntary manslaughter in the medical context. Medical errors have played an oversized role in shaping the law of manslaughter, ²⁰ in historically defining the lower threshold for culpable homicide in England and Wales. A frequent criticism of GNM is that inadvertent conduct can result in the conviction of a very serious offence without recklessness. This fundamental criticism is often aimed at its application in the safety-critical healthcare context, in particular, as this is where errors occur in the course of professional duties attending to critically ill patients. While there should never be peanuts in nut-free products, ²¹ for example, healthcare is more complex, and it is impossible to claim that fatal errors can be completely eliminated.

The first four sections of this article introduce and describe the inherent novel challenges of advisory AI systems and the implications for medical practice, introducing an argument that the dangers of automation bias (AB) and the potential jury approach to AI-induced errors may present a risk to clinicians. The remainder of the article then examines GNM in the jurisdiction of England and Wales and demonstrates that the coming AI healthcare revolution makes a compelling case for a reconsideration of ‘gross negligence’ as the basis for criminal liability, and that the appropriate legal test should be subjective recklessness. A proposed shift to subjective recklessness is already widely advocated for within the legal literature and the introduction of AI systems adds a new dimension and urgency to extant arguments. ²² If, as anticipated, AI systems become more involved in a greater proportion of clinical decisions, the need for reform may be difficult to ignore.

AI in healthcare

AI is not a new technology and has endured a fitful history since the term was first coined in 1956. ²³ Development has ebbed and flowed between modest steps forward and periods of inertia, often referred to as AI winters. ²⁴ During the last decade, substantial progress has been made and AI has benefitted from a long summer, facilitated by the convergence of the ever-expanding availability of big data, ²⁵ the unprecedented speed and reach of cloud computing platforms, and the innovation of increasingly sophisticated machine learning algorithms. There is no universal definition of AI, which is best described as a portfolio of technologies, or a growing family. While many definitions exist within the literature, most now recognise that AI is a novel form of agency with capacity to learn from data; it may be embodied within a physical device, or as software that is instantiated within a system. ²⁶

Investment has flowed into the sector and the UK government has invested over 250 million into AI in healthcare. ²⁷ This enthusiasm among investors and policymakers is built on the hope that these new AI technologies can radically transform patient care, as systems across the globe are struggling with increased costs and decreased outcomes. ²⁸ The mounting pressures of the pandemic are also likely to accelerate the aims to incorporate AI solutions. However, the twin benefits of saving costs and improving outcomes for patients are distinct challenges which can be viewed from separate levels of abstraction. ²⁹

Saving costs is a system goal at institutional and sectoral level. For example, cost savings may be possible by transformative cooperation between machines and doctors ³⁰ with systems to support clinicians, monitor patients, and automate labour-intensive processes. Recent publications like the AOMRC report ³¹ and the Topol review ³² set out many possible efficiency benefits for the workforce in healthcare: research suggests that more than half of the clinical workforce will be routinely using AI predictive analytics, image interpretation, and natural language processing within a decade. ³³

At the individual HCP level of abstraction, AI systems may also reduce medical errors by providing reliable advice on issues such as diagnosis, treatment choice, and treatment or care planning. For example, image recognition algorithms have demonstrated potential in interpretation of head computed tomography scans, ³⁴ as well as in the diagnosis of malignant tumours in breasts, ³⁵ lungs, ³⁶ skin, ³⁷ and brain. ³⁸ Decision-making with support from AI systems has the potential to improve the performance of even experienced radiologists in diagnosing lung cancers. ³⁹ Avoiding common errors may become a ‘core competency’ of AI, as it can putatively avoid inattention, fatigue, and cognitive biases. ⁴⁰ However, claims that AI can reduce individual errors are not guaranteed and there is tension between this utopian vision of error-free healthcare and the economic objective of driving down costs: if responsibility attribution is unjust, then a defensive approach to AI by clinicians may drive costs in the wrong direction. The long-standing claims around the negative impacts of liability are difficult to evidence and quantify and may have developed into a ‘jaded cliché’, ⁴¹ but the reality is complicated because the criminal law is one of a myriad of relevant factors. There are concerns that defensive practice may harm patients through over-testing, over-diagnosis, and wasting of scarce resources. ⁴² On the other hand, there is a perspective increasingly held by the civil courts that legal liability may be standard enhancing which could eventually reduce recriminations and litigation. ⁴³ However, the criminal law is the highest form of moral condemnation within society, and it is unrealistic to expect that there will not be consequential changes to medical practice and underlying clinician behaviour where it is invoked.

AI systems are likely to be introduced when they can consistently outperform HCPs in a given task; ⁴⁴ indeed, when a system surpasses the abilities of human HCPs it may be unethical not to use it. It is likely to become the standard of care. ⁴⁵ In the areas where machines can outperform HCPs, progress is already under way in establishing pathways to clinical use. ⁴⁶ This means that AI systems will be introduced when they make fewer errors: not when they are infallible. If an AI system can be correct 97% of the time and a human HCP has a 94% accuracy rate, then the rate of misdiagnosis should fall, and lives will be saved. However, AI systems will continue to make unexpected errors that may lead to fatalities.

The types of AI examined in this article are considered advisory systems. ⁴⁷ They will not be legally responsible for making decisions. There may be scope in future to introduce closed loops where decisions are made independently by machines, ⁴⁸ but at present an AI system will advise the HCP to take a particular action and then the human will remain responsible for the implementation of the care. Under current EU Law, allowing AI systems full autonomy over healthcare decisions would not be permitted. ⁴⁹ However, this interface between man and machine bears further scrutiny. The argument that the systems are ‘advisory’ does not necessarily reflect the way that they will operate in reality. In the subsequent sections of this article I will argue that AB, the unexplainable nature of AI decisions, and the subsequent epistemic vices construct a legal fiction where HCPs may be unjustly exposed to criminal liability for AI errors.

Unexplainable decisions

At this juncture it is important to consider the first key challenge of many AI systems: they make predictions but do not give explanations. Historically, there have been more primitive uses of rules-based AI in devices such as electrocardiograph machines, or defibrillators. These devices had to be explicitly programmed with prescriptive rules, which limited the potential complexity of any given system to a series of specific commands, sometimes referred to as ‘good old-fashioned AI’. The novel challenges discussed in this article arise through the introduction of more complex machine learning algorithms. Machine learning is not a new AI development; the term was coined by A.L. Samuel in 1959 and defined as the ‘field of study that gives computers the ability to learn without being explicitly programmed.’ ⁵⁰ There are several different methods, but all require high-quality data to perform well. Machine learning systems are already embedded into our social reality: powering smart assistants such as Siri and Alexa, detecting spam emails, and curating social media feeds.

Currently, advanced neural networks such as deep-learning techniques have made the most significant breakthroughs because of this versatility in being able to learn from raw data without the need to encode task-specific knowledge. However, the technology comes at a cost in that the systems are intrinsically opaque and are often referred to as a ‘black boxes’. For example, a diagnostic machine could be trained on millions of scans that show abnormalities and millions that do not, and learn to categorise the scans, examining them pixel by pixel, with exceptional precision where the data are accurate. However, there is a profound challenge in terms of explicability with these systems. The name ‘neural network’ refers to metaphorical way that the processes simulate the neurons of the human brain. The term ‘deep’ refers to the many layers of functions within the system. These mathematical functions cascade through the various layers, adjusting parameters allowing the system to learn from prior outputs and predictions. Explanations of this process become so abstract and mathematically complex that they cannot feasibly be understood in ordinary language, meaning it is not generally possible to understand why a particular result has been achieved. From a criminal liability perspective, this creates a problem for the HCP as the ‘advisory’ nature of the system comes on a ‘take it or leave it’ basis.

The contention that AI is entirely advisory does not necessarily reflect the practical reality of a healthcare setting. It is well established that there are cognitive biases such as AB when decision-support systems are used. The literature shows that AB is prevalent in medical decision-making generally and cannot be reliably removed. Factors such as the complexity of tasks and decision-making under time constraints make AB more likely to occur. Goddard and colleagues describe AB as the process:

by which users tend to over-accept computer output ‘as a heuristic replacement of vigilant information seeking and processing’. AB manifests in errors of commission (following incorrect advice) and omission (failing to act because of not being prompted to do so) when using CDSS [Clinical Decision Support Systems]. ⁵¹

It is important to acknowledge the competing justifications for introducing AI systems and the reason that policy makers are so keen to make large financial commitments. The AI will be deployed to support inexperienced HCPs where more experienced consultants may not be available within healthcare resource constraints. Certain groups of clinicians who perform higher risk work are always more at risk of prosecution ⁵² and the same frontline practitioners may soon be relying on AI systems for decision-support.

To take a hypothetical example, a junior doctor in a highly pressurised Accident and Emergency Department may be treating a recently admitted patient. They follow the correct professional procedure and use an AI system for diagnostic and treatment advice. The system gives a high probability of condition A and treatment recommendations, and low probability of condition B and an even lower probability for a range of other conditions. The doctor is aware that the system has been tested and has a higher diagnostic accuracy than an experienced specialist. They may be uncertain of the diagnosis and may ordinarily have had to double-check with a consultant prior to the implementation of an AI system. The doctor then decides to commence the treatment and move to another patient. However, in this scenario one of the more unlikely scenarios occurs and the patient deteriorates and dies.

In this hypothetical scenario it is legally correct to claim that the decision was made (or validated) by the junior doctor and the substantive facts could make the error appear ‘truly exceptionally bad’. ⁵³ To hold the doctor solely responsible is to maintain a legal fiction because the criminal act will involve the validation of decision that is uninterpretable. The ability to critically examine the methodology of the AI decision is fundamentally impossible with ‘black boxes’. The error may appear serious and obvious in an ex post examination of the error, from a complete informational framework and free from the effects of AB. The juror, with the benefit of hindsight, may consider that uncritically following an erroneous AI recommendation is worthy of punishment.

One possible way to mitigate this risk is that the HCP could exclude all possible conditions listed by performing every perceivable diagnostic test to compensate for the explainability problem; however, this will undermine the system-level objectives of the AI and increase HCP workload. ⁵⁴ Proactively carrying out unnecessary and potentially invasive investigations for conditions that have only a marginal probability will render the value of the diagnostic advice irrelevant.

The lack of visibility into the AI decision-making process has led many to argue that ‘black box systems’ should not be used in high-stakes environments like healthcare. ⁵⁵ In healthcare, it is generally critical to understand the processes behind decisions so that systems can be improved where errors do occur, ⁵⁶ but decisions made by AI systems are incapable of creating epistemic value in this respect. Therefore, some argue that AI in healthcare should be restricted to more ‘interpretable’ models. ⁵⁷ However, ‘interpretability’ is a woolly concept with ‘inconsistently applied terminology’ ⁵⁸ and ‘motives for interpretability and the technical descriptions of interpretable models are diverse and occasionally discordant’. ⁵⁹

It is possible to reduce the complexity of the systems in a way that makes an explanation possible, ⁶⁰ but this will invariably make the system less effective. Demanding explainable AI in healthcare may mean foregoing the benefits of deep-learning techniques altogether, so the opacity problems are likely to remain. It raises the ethical question of: ‘How much are we willing to lose in prediction accuracy to gain any form of interpretability?’ ⁶¹ There is intense academic interest in creating a form of explainable AI. Watson and colleagues note that ‘explanatory breakthroughs have been few and far between’. ⁶² There have been attempts to create AI algorithms that generate explanations by showing the relevant part of a scan that the system has used to make a decision: one AI guesses what another AI is looking at ⁶³ but so far this has not been successful in healthcare. ⁶⁴ The hope of explainable AI may prove to be a fool’s gold or ‘false hope’; however, even if post hoc generative explanations are used, it does not offer a practical solution because it is likely to increase HCP confidence in a less reliable machine that they are still ultimately responsible for.

HCPs working in conditions that leave them exposed to psychological factors could be particularly susceptible to prosecution. Psychological factors can significantly undermine the capacity to avoid harm. Merry and McCall Smith illustrate the issue of ‘mind-set’ ⁶⁵ where a professional will see what they expect to see and do not notice a deviation in repeated tasks. Clinicians are highly likely to see what the AI is advising them to see when they become used to following accurate advice. Gooderham and Toft describe this psychological phenomenon as ‘involuntary automaticity’ and explain that it occurs when a person sees ‘what they expect to see rather than what is actually present’ and describe it as a ‘potent source of medical error’. ⁶⁶ Gooderham and Toft argue that even skilled professionals may become captured by involuntary automaticity and that causes them ‘to act on an unconscious and involuntary basis’. ⁶⁷ This presents a significant danger because judicial commentary has eschewed subjective fault element in favour of the putative reasonable doctor where the conduct may be criminal ‘if you think he did something that no reasonable doctor would have done’. ⁶⁸ If relevant legal tests do not adequately account for the subjective epistemic condition of the clinician, then courts are likely to assert that there is no requirement to consider the defendant’s subjective state of mind, which is likely to overlook the psychological dimensions to AI-induced error. ⁶⁹

Epistemic vices

The next section examines what is likely to be a common problem at the heart of the human–machine interface: the dilemma that is revealed when the problem of clinician–AI disagreement arises. What happens when a doctor is unsure about an AI recommendation and believes that it may be making a medical error?

There are many research articles which compare human performance against the performance of machine learning systems, but they often involve a system classifying images alongside a human who is given relatively little time and the results are judged on a one-off decision. ⁷⁰ This reveals a distinct difference in approach, and machine learning systems may shift clinicians towards more instantaneous decision-making, which the literature has established is a particularly hazardous trait. Both the HCP and the AI system are experts but work in fundamentally different ways. For example, machine learning systems may classify an image by analysing it pixel by pixel and comparing it with thousands of other data sets in a highly mathematical process. Humans would use a more heuristic process involving their experience and judgement. The resulting effect is that both AI and humans will make errors, but they will make different types of errors.

A simple answer to the explainability problem is to accept that the HCP may always follow the system advice because they are entitled to accept that it is accurate: that ‘overruling the advice of the AI system may phenomenologically be experienced as similar to overruling advice given over the phone by a senior colleague’. ⁷¹ A crucial distinction when introducing the criminal law is that individual liability can be transferred to another human agent: the senior colleague may become individually legally responsible for this advice in a way that an AI system cannot. Erroneous advice from a senior college may be exculpatory for gross negligence and the case law supports this. ⁷² However, when AI systems are classed as advisory, this will not be the case.

One solution to this challenge would be for a junior doctor to always seek senior-clinician advice and double-check every time they do not unequivocally agree with the AI; however, as stated, this would fundamentally undermine the system-level goals by increasing costs and raising concerns about defensive medical practice.

A second option would be to accept that, when following the AI recommendation, a clinician could never be criminally liable. This option is not likely to materialise because of the way that machine learning works. There is an abundance of evidence in the literature that machine learning systems can make unexpected errors that may look obvious to a human HCP ⁷³ without the effects of AB. There are many notorious examples of spurious methods of classification, such as an image recognition algorithm that differentiated between wolves and huskies by detecting snow in pictures, ⁷⁴ or a cancer detection algorithm may learn that higher quality scans indicate cancer. ⁷⁵ However effective, the AI is not using the same evaluative process as the human HCP.

To take an example of the kind of error envisioned in this article, a machine learning algorithm was used to predict the probability of death among hospital patients with pneumonia. ⁷⁶ Patients with asthma were systematically classified as lower risk by the algorithm than other patients. This computational determination was fundamentally flawed because asthma patients within the historical data were routinely admitted straight to the intensive care unit where the continuous intensive treatment improved their prognosis, thereby making it appear that they presented less risk. The different clinical pathway skewed the output of the computational model and shows the potential for similar errors: many recurrent error traps may be laid within spurious correlations.

It is the nature of AI mistakes that presents a danger to clinicians, not simply that they occur. It is the fact that AI systems can be remarkable and ridiculous, with aspects that are both superhuman and subhuman. Markus and Ernest describe AI systems as ‘digital idiots savants’; ⁷⁷ if they are right, it presents obvious dangers when introduced into frontline care. Where HCPs validate unexpected errors, it could be ‘truly exceptionally bad’ ⁷⁸ and hindsight may suggest that a clear and obvious risk of death was present. If an asthma patient died shortly after being discharged as low risk, it is highly plausible that a criminal complaint may be made. Where other HCPs pick up a potential system error, it may look damning to the HCP who does not.

HCPs therefore will remain duty-bound to carefully consider that the advice that they are given is valid and appropriate. ⁷⁹ As Pasquale observes ‘there will always be a place for domain experts to assess the accuracy of the AI advice and check with author assess how well it works in the real world.’ ⁸⁰ Since there could be circumstances where an AI recommendation is obviously flawed, it presents a challenge for HCPs: they will be required to act as the ‘common-sense filter’. They will do so under the effects of existing system pressures and the psychological factors caused by AI systems.

HCPs therefore must consider that many of the recommended decisions could be either highly counter-intuitive insights ⁸¹ or the type of brittle errors that occurred with the pneumonia algorithm. ⁸² Any recommendation that challenges the HCP’s own diagnosis creates this epistemic dilemma. HCPs are capable of making omission errors when they fail to follow accurate advice and commission errors when they follow erroneous advice. The epistemic vice is a metaphor that explains the pressure that this dilemma will place upon HCPs, and it is highlighted as a significant practical ethical challenge by Grote and Berens:

Now, in the relevant philosophical debate, there are different theories about what would be reasonable for the clinician to do. According to the ‘Equal Weight View’, learning that an epistemic peer’s proposition differs from your own should diminish the confidence in one’s judgement. Hence, deferring to the algorithm is the most reasonable choice. By contrast, the ‘Steadfast View’ emphasises the epistemically privileged status of one’s own beliefs, which is why it is reasonable for the clinician to stick to her proposition. Therefore, we end up with a stalemate. ⁸³

If the HCP may reasonably take either position, then this leaves them potentially exposed to the legal and moral consequences either way. An AI system could never be considered responsible when the recommendations are not followed, but HCPs may still be held accountable when erroneous AI advice is followed. The current CPS guidance states that ‘the deliberate overriding or ignoring of systems which are designed to be safe and have proven to be safe may be evidence of a serious breach of duty’ ⁸⁴ which unhelpfully turns the vice a little tighter. Whether an AI advisory system could fall into this category will require future clarification.

The new dimension of moral luck

A long-standing criticism of GNM is that moral luck determines who is (and who is not) culpable of manslaughter: death is the engaging threshold for the offence. ⁸⁵ Two HCPs may make the same substantive error and both patients may become critically ill. One patient may survive and the other may not; the stakes are high, and the outcome is binary: prosecution or no prosecution. Once the critical error has occurred the HCP’s agency has ended and where it goes from there is largely a matter of luck.

In JC Smith’s famous example, a father leaves a colourless weedkiller in a bottle of lemonade, accessible to his young child. ⁸⁶ If the child drinks the poison and dies then the father would be prosecuted; if the child does not drink the poison, then the father faces no action. His conduct is no less culpable, irrespective of the outcome. Smith demonstrates that the moral luck extends forward in a temporal dimension where punishment largely depends on factors outside the HCP’s control. ⁸⁷ The introduction of AI systems creates a new dimension of moral luck extending backwards in a temporal dimension. Outcome luck has always permeated the criminal law, ⁸⁸ but AI systems demand an analysis of input luck, which has largely been ignored in medical manslaughter cases to date. ⁸⁹ The putative ‘reasonably competent doctor’ is an abstract construct that exists in a rational metaphysical space free from the vagaries of bad luck, exhaustion, and system pressures beyond their control.

AI development is often referred to as the ‘AI lifecycle’ to describe the many stages that are required to build machine learning systems including: the design of the product; the acquisition of the data; creating and evaluating the model; and then deployment. ⁹⁰ There are many points of potential failure and safety risks exist in data collection, product development, as well as clinical use of AI. ⁹¹ As demonstrated, AI can be brittle, ⁹² but there are other significant safety risks such as: ‘concept drift’ where AI systems become less effective when they leave the training data behind; adversarial attacks that can easily mislead a model; ⁹³ data poisoning where training data are compromised; or unsupervised machine learning systems that may realise objectives by taking dangerous actions that could not be anticipated by their creators. ⁹⁴ Creating reliable, secure, and robust AI systems is a complex sociotechnical endeavour involving many actors and processes. This is a clear example of what the philosophical literature refers to as ‘the problem of many hands’. ⁹⁵

While the nature of the errors may be highly technical and difficult to detect, the errors may materialise in ways that are socially untenable and likely to provoke anger and resentment when discovered. An example garnering much attention is that AI applications are particularly susceptible to bias. ⁹⁶ Bias in training data may occur because the data sources themselves may not reflect the true epidemiology within a given demographic. ⁹⁷ This means that errors may be far more likely to occur in under-represented groups such as ethnic minorities, ⁹⁸ women, ⁹⁹ and those with disabilities. ¹⁰⁰

In the United States, an algorithm used to allocate healthcare resources had been widely discriminating against African Americans; subsequently, they were far less likely to be referred for treatment than white people when equally sick. ¹⁰¹ The proprietary aspect of many algorithms may make this difficult to detect, which may cause further harm by leaving errors undiscovered. Another example is that skin cancer detection algorithms may be less effective on darker skin. ¹⁰² This is an issue that could have severe consequences where machine learning is used in safety-critical scenarios.

If AI systems are not transparent and explainable, then they cannot be reliably detached from other aspects of the sociotechnological system, making it difficult to identify and correct errors. Therefore, structural inequalities in data are distilled into unsafe recommendations before they are validated by the HCP. Errors may also remain in the system because data are inaccurately labelled. There is currently no clear regulator for data quality which may leave HCPs facing the consequences of a poorly designed data set which produces dangerous and discriminatory outcomes. ¹⁰³

Where these systemic failures exist, AI systems may be doomed to fail for particular patients at a particular time. There are already concerns that serious heart problems have been under-diagnosed when women use a primary care AI chatbot. ¹⁰⁴ Those with a rare condition, uncommon co-morbidity, or those susceptible to a rare drug interaction may find the AI system makes an error that was fated to occur in the system. The asthma example will not be the last time an AI makes a dangerous highly counter-intuitive correlation. Mistakes like this may become silent killers within otherwise highly accurate systems and the HCP standing in front of the computer at this juncture may find themselves set up to fail, whereas the next clinician to use the system may find the computer working within the expected parameters on a paradigmatic case. Both clinicians may carefully follow the same processes, but one HCP may find that they are being investigated for a fatal error when they over-rely on the AI. These types of errors are impossible to detect; therefore, where and when they materialise will largely be a matter of luck for the clinician. Returning to Smith’s example, for clinicians using AI, the bottle of lemonade is already poisoned before it is opened. It is served up in good faith and if someone ingests it, they will point a finger towards the doctor that poured the drink.

GNM: the unexplainable law

The introduction of AI systems will present significant challenges to applying the established principles of the criminal law for fatal errors. However, the present GNM legal landscape is already far from clear: AI is not alone in having explicability problems.

At a doctrinal level, the fundamental rationale for prosecuting offences of GNM is set out by Lord Hewitt in R v Bateman that ‘the negligence or incompetence went beyond a mere matter of compensation and showed such disregard for the life and safety of others as to amount to a crime against the state and conduct deserving punishment.’ ¹⁰⁵

The current legal paradigm is derived from a patchwork of poor decisions and perplexing judgements that arguably demonstrate ‘an exhibition of the common law at its worst’. ¹⁰⁶ The confusion and uncertainty arise from five aspects of the development of the law: the historical swing between recklessness (advertence) and gross negligence (inadvertence) as a basis for liability; the Court of Appeal test in Adomako; ¹⁰⁷ the subsequent judgement in the House of Lords, which is still the leading authority; ¹⁰⁸ the human rights challenge on certainty in Misra; ¹⁰⁹ and, finally, the recent case law developments in Rudling ¹¹⁰ and Rose. ¹¹¹ As each judgement tried to make sense of the circular and nebulous nature of ‘gross negligence’, the confusion of each judgement was sewn into the next.

In Bateman the Court of Appeal decided that a doctor could be guilty of manslaughter by gross negligence if they showed ‘such disregard for life and safety of others as to deserve punishment’, ¹¹² which suggested reckless intent as the bar to prosecution: to ‘disregard’ is to have an awareness of risk. This test was broadly reaffirmed in Andrews v DPP, ¹¹³ Lamb, ¹¹⁴ and Stone and Dobinson. ¹¹⁵ In the 1980s’ HL and Privy Council cases of Seymour ¹¹⁶ and Kong Cheuk Kwan, ¹¹⁷ the courts moved to a test of objective recklessness as outlined in the criminal damage case of Caldwell. ¹¹⁸ Ashworth noted that: ‘manslaughter by gross negligence had been replaced by and absorbed into reckless manslaughter’. ¹¹⁹ At that stage the challenging task of trying to define the elusive nature of ‘gross’ appeared to have been abandoned altogether. This detour into objective recklessness was short-lived, confusing, and may have achieved little as it risked simply rebranding conduct that would have otherwise been grossly negligent as reckless. Objective recklessness is still theoretically a putative legal option for GNM ¹²⁰ but could be uniquely unhelpful in the AI-enabled healthcare system where medical decision-making is epistemically more complex.

The foundation of the present legal test begins with the decision of Adomako in the Court of Appeal, which has only had minor adjustments since. ¹²¹ Dr Adomako was a locum anaesthetist who failed to monitor the oxygen supply for over 4 minutes during routine surgery, leading to an unnecessary death, and was charged and convicted with GNM. Dr Adomako was profoundly negligent and oblivious to the risk. The case eventually reached the HL where Lord Mackay LC established the precedent that plotted the subsequent course for GNM stating that:

The ordinary principles of the law of negligence apply to ascertain whether or not the defendant has been in a breach of duty of care towards the victim who has died. If such a breach of duty is established, the next question is whether that breach of duty caused the death of the victim. If so, the jury must go on to consider whether that breach of duty should be categorised as gross negligence and therefore a crime. ¹²²

The judgement is confusing and contradictory in many respects. Lord Mackay confirms that gross negligence rather than recklessness is the basis of liability but offers little further clarification stating that gross is simply a matter of degree and that ‘to specify that degree more closely is I think likely to achieve only a spurious precision’. ¹²³ At times, the use of the terms ‘recklessness’ and ‘gross negligence’ have been used interchangeably in judgements, which has amplified the confusion. Lord Mackay rejects a subjective recklessness standard on one hand but states that it is ‘perfectly open to the trial judge to use the word “reckless” in its ordinary meaning as part of his exposition of the law’. ¹²⁴ It is not clear what the ordinary meaning of reckless is, but Lord Mackay notes ‘a danger of over-elaboration of definition of the word “reckless”’.

The judgement was not generally well received, with criticisms that ‘prosecutors, experts, judges and juries are thus left to grapple with a difficult and circular concept’. ¹²⁵ The juries are ultimately responsible for determining when conduct has crossed this line from civil to criminal liability. However, jurors are likely to over-rely on expert witnesses ¹²⁶ which ‘underplays the risk of jury usurpation by investing too much epistemic authority in the expert’. ¹²⁷ Therefore, understanding when the criminal law should be invoked is difficult because it is ill-defined, potentially broad in scope and difficult to consistently apply.

Judicial analysis has comprehensively failed to adequately capture the meaning of ‘gross’ without cycling through synonyms. Perhaps most famously, Leveson J, in both Sellu ¹²⁸ and Rose ¹²⁹ (and now adopted in Broughton ¹³⁰ ), described gross as ‘truly exceptionally bad’. The problem with linguistic definitions like ‘truly exceptionally bad’ is that they mean little without reference to a juror’s own moral compass and they may fail to appreciate the complexities faced at the clinician–AI interface. After all, nobody assumes that they, personally, would tacitly follow sat nav instructions and drive into a river, yet it continues to happen. ¹³¹

An opportunity to clear up the confusion presented itself in the Court of Appeal in another medical manslaughter case involving a misdiagnosis and untreated infection. R v Misra and Srivastava ¹³² was appealed on the grounds that the lack of legal certainty in determining grossness amounted to a breach of Art 7 of the European Convention on Human Rights, which prohibits ex post facto criminalisation. It was argued this would inevitably lead to inconsistencies in prosecution policy. The court held that the issue ‘is not whether the defendant’s negligence was gross and whether, additionally, it was a crime, but whether his behaviour was grossly negligent and consequently criminal’. ¹³³ As Ashworth commented, ‘it is a distinction without a difference.’ ¹³⁴ The judgement crystallised the confusion in Adomako and infused an untenable level of vagueness into the law. Quick argued that the dismissal is unconvincing. ¹³⁵ Ashworth notes ‘it should not be the end of the matter’ ¹³⁶ and the introduction of AI systems may mean that this point needs to be revisited. As a result of the enduring state of confusion, a steady stream of cases continued to reach the Court of Appeal seeking clarification.

Responsibility of the HCP

This article has demonstrated that there are many circumstances where it is predictable that even a careful, diligent, and reflective HCP may act on erroneous advice. The system-level objectives will demand that AI systems will be used by HCPs who are not at the top of the HCP hierarchy and where the HCPs will be confident that the AI system performs (on average) better than they do at the particular classification task at hand. In such contexts it would be perverse not to accept that HCPs are likely to over-rely on the AI advice, and that they should not be criminally liable in cases where they do over-rely.

It may be tempting to dismiss the likelihood of prosecutions but there is a history of humans in the loop being held responsible for errors involving automated and assisted decision-making. Elish argues that humans in complex systems act like a ‘moral crumple zone, like a car bonnet designed to absorb the force of impact in a crash’ and suffering the ‘moral and legal penalties when the system fails’. ¹⁶³ A relevant example of what may lie ahead for HCPs involves the prosecution of Rafaela Vasquez in the United States for a fatal accident involving a ‘self-driving car’. ¹⁶⁴ The autonomous vehicle, owned by Uber, failed to automatically stop when cyclist Elaine Herzberg was crossing the road and a collision occurred at 39 mph proving to be fatal. The accident occurred at the peak of the hype around autonomous vehicles and created much speculation around potential liability. ¹⁶⁵ The ‘backup driver’ Rafaela Vasquez was blamed and is alleged to have been distracted by her mobile phone, but the National Transportation Safety Board also found that multiple complex factors contributed to the accident including, most notably, Uber’s inability to address ‘automation complacency’. ¹⁶⁶ Rafaela Vasquez is awaiting trial, while Uber has not been prosecuted for any of the failures that contributed to the crash. ¹⁶⁷ This single individual prosecution that seeks to parse individual responsibility from complex interconnected failings bears much similarity to medical manslaughter prosecutions.

There is also likely to be some theoretical weight behind the idea that doctors should be responsible for AI decisions. Hart supported the classification of negligent conduct as criminal provided the individual was of normal capacity. ¹⁶⁸ There are contemporary academics who have been largely supportive of the concept of negligent liability in criminal law. ¹⁶⁹ The basis for this vision of moral responsibility is that where the HCP has a duty and capacity to avoid harm, then they are culpable when that harm occurs. The potential risk for HCPs is that they will always retain the technical capacity to avert the harm because they could theoretically ignore erroneous AI advice and will have a legal duty to do so where it is obvious to them. The HCP will be judged objectively on what amounts to ‘truly exceptionally bad’ ¹⁷⁰ and this might not take account of the effects of AB or the epistemic vice. If the jury believe that the reasonably competent doctor should not have allowed the error to happen, then the HCP may be convicted.

There are principled objections to gross negligence because of the lack of any intent to do harm: there simply is not enough mens rea. ¹⁷¹ Manslaughter has moved away from the foundational principles of the criminal law and the notion of blameworthy conduct. It amounts to ‘a very crude form of retribution, based not on the moral culpability, but on the harm caused’ ¹⁷² amounting to an ‘unprincipled privileging of harm over culpability’. ¹⁷³ The appropriate response to medical error is an ethical question as well as a legal question and it has been noted that manslaughter has disconnected from ethics in the way that it depends on moral luck, ¹⁷⁴ and that it involves a low level of mens rea for such a serious offence. One way for the law to reconnect with ethics is to consider the ‘control’ and the ‘epistemic’ conditions for responsibility ¹⁷⁵ and ask in what sense the HCP has sufficient knowledge of the ‘black box’ system to genuinely avert the error, and what level of control they have under AB and operating in an epistemic vice. Since human professionals will retain legal responsibility for validating the AI system, the AI is a novel form of agency but does not meet the standards required for legal agency. Therefore, the human HCP becomes the ersatz legal and moral agent for AI decisions. Since human HCPs have this instrumentality imposed upon them, the law should reflect the reality of this position. AI-assisted decisions are the result of long chains of sociotechnical complexity, which fundamentally challenges the traditional ‘snapshot’ approach to responsibility attribution.

In Adomako, the court rejected the argument of the defence that involuntary manslaughter should have the characteristics of (1) clarity, (2) certainty, (3) intellectual coherence, and (4) general applicability and that the test should be subjective recklessness. ¹⁷⁶ The submission was rejected but the argument should be rebooted because a test of subjective recklessness presents a much more convincing approach to assess culpable conduct with errors that involve erroneous AI recommendations.

The concept of subjective recklessness is well developed in judicial commentary. Following the legal analysis as set out in the criminal damage case of Cunningham, ¹⁷⁷ the test for subjective recklessness is stated in Re G by Lord Bingham: states that a person acts recklessly when: (1) a circumstance in which he is aware that the risk exists or will exist; (2) a result that when he is aware of the risk it will occur; and (3) it is in the circumstances known to him, unreasonable to take the risk. ¹⁷⁸ The test of recklessness is ‘uncontentious and well-understood’ ¹⁷⁹ and could bring much anticipated relief to HCPs and much-needed clarity to a problematic area of law: rarely do such complex problems have such an obvious solution. The argument that subjective recklessness should replace gross negligence is not revolutionary, or indeed something that would set involuntary manslaughter apart from other criminal offences: quite the opposite is true. Repeated judgements have held that subjective recklessness should apply to non-fatal offences, such as Spraggan, ¹⁸⁰ Morgan, ¹⁸¹ and Parmenter. ¹⁸² It is fair to say that the wider criminal law is shifting towards subjectivism. The reason that the courts are presently reticent to fully embrace subjective recklessness for involuntary manslaughter involves a legacy of unresolved legal and philosophical tensions over whether negligence should ever reach threshold of criminal conduct. As noted by Gardener and Jung: ‘The dispute between “subjectivists” and “objectivists”, meanwhile, is largely about a single issue – the propriety of imposing criminal liability for negligence.’ ¹⁸³ It has previously been suggested that a specific offence could be created for manslaughter in healthcare settings to take account of the unique clinical context, ¹⁸⁴ which could go some way to addressing challenges posed by AI advisory systems; however, with a test of subjective recklessness this requirement would dissipate.

A potential objection to adopting a legal standard of subjective recklessness is that there may be practical difficulties in establishing subjective fault; however, this may be over-stated because the jury remains entitled to find the defendants account unconvincing. Quick argues that ‘any such worry that prosecutors couldn’t prove subjective awareness of risk is an exaggerated one.’ ¹⁸⁵ With AI systems, these objections may be diminished by the nature of AI systems laying bare the frame for the subjective epistemic condition. HCPs will be given AI recommendations and when they are followed in good faith, it should not meet the threshold of criminality.

Another popular objection to a subjective recklessness standard may be that allowing the most catastrophic examples of AI-induced errors to go unpunished could lead to the removal of a potent deterrent. Glanville Williams accepted this utilitarian justification for negligent liability, reasoning that the threat of sanction would encourage improved standards of behaviour. ¹⁸⁶ However, this utilitarian justification for criminal negligence is inextricably linked to the efficacy of its deterrent effect: it must stand or fall on whether it works. AI-induced errors likely to be complex, unexplainable, and involve a psychological dimension. They are not the kind of errors that can be deterred, and it will do nothing to improve safety. Instead, punishing inadvertent AI-induced errors may contribute to a climate of fear and undermine the policy initiatives to create a fairer response to medical error. It is not controversial to state that prosecutions do not help promote a culture of candour and reporting errors; as Merry and McCall Smith argued: ‘blaming the person “holding the smoking gun” may simply leave the scene set for a re-occurrence of the same tragedy.’ ¹⁸⁷ Prosecuting doctors for AI-induced GNM is likely to create three undesirable social consequences: it is likely to be inimical to safety; it could undermine system goals by incentivising defensive practice when using AI; and it may damage trust and fatally undermine the adoption of beneficial technologies. Where the criminal law is likely to have such negative social consequences, there is a strong argument for taking a minimal approach. As Husack argues, it is wrong to create an offence or set of offences where this might cause greater social harm than leaving the conduct outside the criminal law. ¹⁸⁸

Conclusion

The potential sources of AI-induced error are manifold and complex. The nebulous nature of ‘gross negligence’ is impossible to define and it will be profoundly difficult to apply to AI-induced errors. A legal test of subjective recklessness could reduce the risk of an AI-induced error falling within the ambit of the criminal law and, on balance, presents a better option. If the policy aims for the AI healthcare system are realised, then an increasing proportion of medical decisions may soon involve AI systems. Therefore, the inevitability of AI-induced errors in healthcare could have significant implications for the mens rea of culpable homicide more generally. In the absence of clear subjective recklessness, the criminal law should not be invoked; it is likely to damage trust and in extreme circumstances may lead to rejection of beneficial AI solutions.

The AI ethical literature highlights maintaining societal trust as fundamental to realising the potential benefits of AI technologies. ¹⁸⁹ The problem with the criminal law is that it can erode trust in two ways: when it is applied unjustly; and when not used at all and creates a perception that nobody has been held responsible. Therefore, adopting a more minimal application of the criminal law ¹⁹⁰ alone as a solution to these types of errors will remain unconvincing without having a just response to AI-induced medical error. AI systems make a strong case for moving towards more patient-centric responses to errors. The patiency perspective involves two critical aspects that require different legal and ethical frameworks: first, an ex ante need to know how the technology will be used and that it will be safe and well regulated; second, an ex post system that can ensure that, where errors inevitably occur, the appropriate response takes place so that victims can understand what happened and that responsibility is taken for the harm.

The aim of this conclusion is not to introduce new concepts at a late stage, but to highlight that there are extant arguments for more patient-centric approaches to adverse medical events, which should be given renewed examination through the lens of AI-induced errors. For example, no-fault systems may be necessary to address AI errors ¹⁹¹ and non-contentious legal responses could help avoid criminal complaints where the police station is the option of last resort. Engaging with what a victim needs is the appropriate response from a relational framework asking: Who has been affected? and What are their needs? It is also correct to recognise that the human HCPs are also patients of AI systems. Incorporating the policy aims of creating fairer systems in healthcare will require a shift to asking, ‘what is responsible, not who is responsible’. ¹⁹² As the healthcare system starts to incorporate AI systems, it presents an opportunity to bring a more human approach to the aftermath of fatal errors. There are already arguments that restorative justice is an appropriate response to many fatal medical errors ¹⁹³ and the possibility of AI-induced liability highlights the ongoing importance in exploring alternative responses outside of the criminal law and theorising new models of liability. There is no expectation that the pain caused by the death of a loved one is a situation that can be restored; instead, the challenge for healthcare systems is to develop an appropriate and just legal response to manage the aftermath of AI-induced errors and proceed into our AI healthcare future with the trust of patients and the medical profession.