Who is responsible when AI kills?

Thomas Wakley, the founder and first editor of the Lancet, was driven by his concerns about the lack of accountability of the medical profession. In Nick Black’s novel about Wakley, Bare-knuckle Surgeon, Wakley declares that “The public are to be grateful for anything they [doctors] do, even if it kills them.” Richard Smith’s review of the book captures the radical reform that medicine is undergoing in the early 19th century. ¹ It’s a quote that resonates strongly today. The “tech bros” and other zealous advocates of artificial intelligence want us to be grateful for anything and everything that AI does. If it kills us, so be it.

Carl Macrae tackles the question from a system perspective in asking, what happens when AI kills a patient? ² The question is a complex one to answer. With AI being rapidly integrated into clinical decision support systems, it is also an urgent one to address. The imperfections of AI are known but easily forgotten. AI fabricates. It reinforces biases inherent in the scientific record. It has difficulty in differentiating well conducted and poorly done research. It struggles to know what matters and what doesn’t. It doesn’t forget – an underrated virtue. AI algorithms are under the control of the world’s richest corporations who are in a ruthless battle to become the richest of all. Too many of us, professions included, have swallowed the AI kool aid. 

In these circumstances, AI’s coded hand in patient deaths is as inevitable as medical error is a system and a human failure, Yet, who will be held accountable? A standard disclaimer for any decision support technology is that the doctor is ultimately responsible for decision making. However, AI blurs that boundary more than any technology that has gone before it. AI seeks to make decisions in place of doctors and, if not that, it deskills them.

England’s NHS plan is gambling on AI reducing the need to recruit more doctors. It’s a risky bet given that we don’t know if AI can do that but we do know that one of the biggest impediments to delivering care is medical staff shortages. A new research paper tells us how staff sickness absence impacts elective backlogs. ³

Another issue is that corporations don’t want to accept liability, and that they make denial of liability a central part of their agreements with governments desperate for their products. We saw it with covid-19 vaccines and we are seeing it again with AI. This places a great responsibility on governments to ensure that medical data are secure and appropriately analysed, ⁴ and that medical information is considered a public good for all to access. ⁵ Which leaves us with a final dilemma. Many leaders are outwardly unfit for office in health terms, and it is a taboo to question their health. Why should that be the case? A regular independent medical assessment might make democracy stronger and create a healthier world. ⁶