The lamp and the lighthouse: Joseph Weizenbaum,contextualizing the critic

A clear homage to Michelangelo's ‘The Creation of Adam’ dominated the first page of the ‘Perspective’ section of the November 13, 1977 edition of The Baltimore Sun, yet the hands that reached out towards one another did not belong to the naked Adam or to the bushy-bearded God. Instead, Adam had been replaced by a robot, while God had been replaced by a bald-bespectacled man in a lab coat. Beneath this provocative image, Joseph Weizenbaum sought to shine a light, for the Sun's readers, on the hazards of being taken in by this high-tech retelling of ‘let there be light’ (Weizenbaum 1977b).

Between his role in composing the programming language SLIP, creating the influential proto-chatbot ELIZA, and teaching at MIT – Joseph Weizenbaum has secured an undeniable place in the history of computing, and artificial intelligence. Yet what most distinguishes Weizenbaum from other prominent figures in these fields is not his technical achievements, but the way he publicly turned against them (Crevier 1993; Kling 1996; McCorduck 2004). Weizenbaum made his most lasting contribution as a critic, though what mattered to him was not the computer in and of itself but ‘the role of computers in our society’ (Weizenbaum and Wendt 2015, 33). It was precisely his intimate knowledge of the way that computers operated that led him to challenge the excited pronouncements that were made by those he referred to, with a mix of derision and respect, as the ‘Artificial Intelligentsia.’

Weizenbaum is one of AI's most notable discontents. A rock in the field's shoe which remains stubbornly present, he looms in the background of every dystopia-tinged article about AI, whispering, ‘I warned you.’ For those inclined to believe that computers and AI have set humanity on the wrong track, Weizenbaum appears as a hybrid of patron saint and Cassandra; while to those with high-hopes for computers and AI, who see ‘artificial intelligentsia’ as a badge of honour, Weizenbaum appears as a curmudgeonly Chicken Little. Drawing on Weizenbaum's letters to his fellow social critic Lewis Mumford, this article argues that in order to properly understand Weizenbaum's warnings it is necessary to recognize that he was not a lone prophet, howling in the wilderness, but a part of a broader milieu of social-criticism that saw science and technology as important sites for critique.

Lighting the lamp

In an anecdote about ELIZA which he returned to frequently, Weizenbaum recalled his secretary reacting as if he was interrupting a private conversation when he walked in on her using the program – despite the fact that she knew that the program did not understand the things she was saying to it (Weizenbaum and Wendt 2015, 90). Initially, Weizenbaum had expressed the belief that when the workings of a program were made clear ‘its magic crumbles away’ (1966, 36). However, in terms of the reactions to ELIZA, Weizenbaum was struck by how it ‘created the most remarkable illusion of having understood,’ which was ‘most tenaciously clung to among people who knew little or nothing about computers’ (1976, 189).

Named for the character from Pygmalion, ELIZA allowed a person to communicate in natural language with the computer and receive scripted responses. In its most famous version, ‘doctor,’ ELIZA took the form of a Rogerian psychotherapist – taking the statements of the user and echoing them back in the form of a question (Weizenbaum 1966). As far as Weizenbaum was concerned, the program ‘parodied’ a psychotherapist, and it disturbed him ‘that psychiatrists and other people took this parody seriously’ (Baumgartner and Payr 1995, 252).

Prior to ELIZA, Weizenbaum had constructed a program that excelled at playing the game Five-in-a-Row/Go-MOKU – his description of the program was published under the auspicious title ‘How to Make a Computer Appear Intelligent’ (1962). The key for this appearance boiled down to the program's creator ‘setting out to fool’ users about the program's workings (24). Success was measured by how many people had been fooled and for how long (24). This program created ‘a wonderful illusion of spontaneity,’ and based on Weizenbaum's wry admission that he had trouble beating the program himself, the program seems to have played Five-in-a-Row fairly well (24). That a program should be able to fool a user into believing it to be intelligent, for Weizenbaum, was not proof of the program's intelligence but of the intelligence of the program's creator – though he still believed that the user would catch on eventually. This game-playing program helps underscore that what shocked Weizenbaum about ELIZA's reception was not that the program fooled its users, but how willing those users were to be fooled.

Granted, it is one thing to trick someone into thinking their opponent in a game is intelligent. Fooling someone into believing the therapist in whom they place their trust is intelligent is another. Though Weizenbaum's early admission that the point was to trick the user, his technical articles on ELIZA suggest he did not believe users would be consistently or willingly fooled. Sometimes ‘magic crumbles’ but sometimes the enchantment deepens, and ELIZA's powers were so entrancing that this sort of mystified misunderstanding has come to be known as ‘the Eliza effect’ (Broussard 2018; Murray 2017; Natale 2019; Suchman 2009; Turkle 2005; Wardrip-Fruin 2009). Rather than glorying in his success, Weizenbaum seemed chastened by it. ELIZA's only power was ‘to deceive,’ and he emphasized that ‘no humane therapy of any kind ought to be grounded on’ such a power (1977a, 354).

Though ELIZA quickened Weizenbaum's critical impulse, his concerns with the responsibilities of computer scientists predate the public reaction to the program. As a participant in a RAND Corporation Symposium in 1965, Weizenbaum was already giving voice to the concerns that make his reactions to ELIZA intelligible (McCracken 1965a, 1965b). Alluding to the doomsday clock of the Bulletin of the Atomic Scientists, Weizenbaum opined that it would only be a matter of time before ‘computer people will start publishing a bulletin with the clock hands showing five minutes to twelve’ (1965a, 24). This proximity to apocalyptic midnight was linked for Weizenbaum to his sense that computers were ‘potentially dangerous instruments’ though few others were willing to admit it (24). Over the course of the symposium, Weizenbaum echoed the belief, shared by the other speakers, that computers would become more widespread, but for him this was why it was incumbent upon computer professionals to learn to distinguish between ‘what is worthwhile and what is not worthwhile,’ ‘what is right and wrong,’ and to learn to say ‘I will not do this’ (25).

Referring to the doomsday clock, Weizenbaum noted ‘Atomic scientists said after 1945 that they felt they had known sin’ (24). It seems the ELIZA effect filled Weizenbaum with a similar feeling.

The lamp and the lighthouse

The concerns Weizenbaum raised regarding computers and AI transcend the moment in which they were originally posed, but his reference to the doomsday clock is a reminder that his thinking did not occur in a vacuum. The 1960s and 1970s were a period of social tumult, in the US and internationally, and it was a period in which many individuals in the technical and scientific community spoke out, even as a mixture of enthusiasm and mistrust greeted the latest technoscientific developments (Egan 2007; McCray and Kaiser 2016; Moore 2008; Roszak 1995; Schmalzer, Chard, and Botelho 2018; Tierney 2019; Turner 2008; Wisnioski 2016). As a critic from within the technical world, Weizenbaum has been seen as part of an intellectual tradition consisting of figures who have used their scientific status to bolster their criticisms of computers and AI – such as Norbert Wiener, Hubert Dreyfus, and some contemporary critics (Bynum 2001; Fleischman 2015; Frischmann and Selinger 2018; Gill 2019; Nadin 2019; Natale and Treré 2020; Mitcham 1994). While Wiener and Dreyfus certainly influenced him, to fully grapple with Weizenbaum's thinking it is necessary to place him in a broader milieu of twentieth-century social critics who were focused on technological issues.

From the outset of Computer Power and Human Reason (1976), Weizenbaum draws attention to the line of thinkers who have ‘expressed grave concern about the conditions created by the unfettered march of science and technology.’ This list included ‘Mumford, Arendt, Ellul, Roszak, Comfort, and Boulding’ (11), as well as Max Horkheimer and Erich Fromm. What unites this set of disparate thinkers is the attention they directed to the ways in which complex technological systems, and allegiance to technical thinking, resulted in the sacrifice of human needs on the altar of technological expedience. For many of these figures – several of whom, like Weizenbaum, were Jews who had fled from Nazi Germany – the Holocaust and the onset of the nuclear arms race stood as grim testaments to what occurred when the steely logic of mechanized rationality held power. While Weizenbaum clearly pays homage to these influences, his most significant comment about any of these figures is his expression of gratitude to ‘Lewis Mumford, that grand old man’ who Weizenbaum notes ‘read all’ of the manuscript (x). Weizenbaum gives Mumford further credit by noting that it was Mumford who had convinced him that ‘it sometimes matters that a member of the scientific establishment say some things that humanists have been shouting for ages’ (x).

Mumford was certainly one of the ‘humanists’ who had ‘been shouting for ages.’ By the time Weizenbaum set about writing Computer Power, Mumford had reached the twilight of his years as a public intellectual. Though Mumford was still publishing in the 1970s, his books that came out in that decade were largely collections of earlier work and autobiographical reminisces (Mumford 1972, 1975, 1979, 1982). Throughout his oeuvre, few topics escaped Mumford's critical gaze, he wrote about literature, art, architecture, politics, education, utopias, and much else – but he made his greatest impact with his writings about technology and cities (Blake 1990; Hughes and Hughes 1990; Jacoby 1987; Kuhns 1971; Luccarelli 1996; Miller 1989; Mitcham 1994; Wojtowicz 1996). Although he was aware that the approach would not win him many friends, Mumford tended to see himself in the mold of the prophet Jonah, someone ‘who keeps uttering the very words you don't want to hear … warning you that it will get even worse unless you yourself change your mind and alter your behavior’ (Mumford 1979, 528).

Drawing on the thinking of his mentor Patrick Geddes, Mumford's early engagement with technology, saw him speaking of technological eras wherein a society could be defined by its sources of power and technological modes (Mumford 1934). Yet the onset of World War II and the rise of the nuclear era significantly darkened Mumford's worldview (Mumford 1939, 1940, 1946, 1951, 1954, 1956, 1959). Throughout the post-war period, Mumford worried about the power that was accumulating around complex technical systems, and his warnings about ‘authoritarian technics’ gradually turned into a theory of the ‘Megamachine,’ a vast social structure composed of ‘human parts’ that was remaking the world in its own mechanized image (Mumford 1964, 1967, 1970). A key feature of authoritarian technics was the danger that ‘Once one opts for the system no further choice remains’ (Mumford 1964, 6) – a development clearly visible in the growth of ‘computer-dominated society’ (Mumford 1970, 191). Indeed, for Mumford, the computer was the key device central to the functioning of the Megamachine (Loeb 2018). While he was concerned with specific machines, his greater concern was with the ideology that had developed around these machines: the belief that technoscience would solve all human problems. Mumford chafed at this ‘myth of the machine,’ which had become ‘the basic religion of our present culture,’ having ‘so captured the modern mind that no human sacrifice seems too great provided it is offered up to the insolent Marduks and Molochs of science and technology’ (Mumford 1975, 379). As a critic, Mumford confessed to having frequently felt rather lonely, yet in Weizenbaum he found a kindred spirit.

Friendships do not always leave behind many records, and it is not always possible for a historian to reconstruct the details of phone conversations or personal visits, but Weizenbaum's letters to Mumford provide a window into their relationship – despite the fact that Mumford's letters to Weizenbaum have not been preserved. The correspondence features many of the expected contents of a discourse between two friends, but it also includes ruminations on the state of the world that provide insight into Weizenbaum wrestling with his new identity as a social critic. Weizenbaum's letters make it clear that he was inspired by Mumford, praising him as ‘an invaluable example to those others of us who also struggle to become and remain free’ even if that struggle should be painful (Weizenbaum to Mumford, 8/17/1972). 1

In much of their early correspondence, Weizenbaum provided Mumford with running commentary on the progress he was making on Computer Power. Though Weizenbaum had not been hiding his opinions on computers in the years prior to publication, he still expressed nervousness about what was to come, noting ‘I am as frightened of the task ahead of me as layman might be were he called on to do surgery’ (W2M, 11/21/1973). Yet Weizenbaum overcame his trepidation due to his certainty that what he was saying needed to be said, and this sentiment was bolstered by Mumford's belief that this message would have added weight coming from someone within the technical community. Weizenbaum found himself grappling with just how deeply entrenched the ‘technological metaphor’ had become amongst his technical peers; he worried that none believed ‘reports of atrocities’ until it was too late to prevent them, thus his task was to ‘find a way of telling my readers that these particular atrocities are happening to them, that all this is not merely academic foolery’ (W2M, 6/24/1974). The challenge of reaching the public is a leitmotif running throughout the correspondence – yet a definitive answer always eluded both men.

As Weizenbaum went about completing his book, the US was rocked by the resignation of President Nixon. While Weizenbaum was not upset to see Nixon go, his assessment of that situation featured the refusal to take good news at face value that was a hallmark of both correspondents’ thinking. To Weizenbaum the risk of Nixon's resignation was ‘that the people will think that all troubles are finally behind us,’ whereas he saw nothing to celebrate in ‘a system that elevates men like Mr. Nixon to the stewardship of the nation’ (W2M, 8/8/1974). In a playfully apocalyptic comment, Weizenbaum pondered that ‘What appears to be a dawn may be the herald of the coming Gotterdammerung!’ (W2M, 8/8/1974). But in the midst of the world's tumults, with the war in Vietnam and the political fallout surrounding President Nixon looming in the background, working on Computer Power kept Weizenbaum grounded.

By the final months of 1974 Weizenbaum was circulating his manuscript to friends and students, he hoped the message of the book would resonate with readers both in and outside of the technical community as it had been composed with such a varied community of readers in mind. Computer Power and Human Reason is many things: an exploration of how Weizenbaum had come to criticize computing, a basic explanation of how computers work, a cutting commentary on the culture surrounding computing at prestigious universities, and most of all an impassioned plea to computer professionals to take responsibility for the things they were creating. Throughout the book, Weizenbaum upbraids his fellow computer professionals, not for thinking in terms of technical possibilities, but for failing to first ask whether it was appropriate to delegate certain tasks to machines; as he explained ‘since we do not now have any ways of making computers wise, we ought not now to give computers tasks that demand wisdom’ (1976, 227). And ‘wisdom’ was a trait that Weizenbaum doubted a computer could ever possess. A concern with responsibility animates the text, and Weizenbaum feared the ways in which ‘the myth of technological and political and social inevitability is a powerful tranquilizer of the conscience’ that serves ‘to remove responsibility from the shoulders of everyone who truly believes it’ (241). The book concludes with an appeal to readers, particularly those who are intimately involved with computers, to find the ‘simple kind of courage’ that involves learning to say ‘No!’ (276).

When he had started writing it, Weizenbaum had worried about how Computer Power would be received at MIT, as the book painted a less than flattering picture of those close to computers, and by arguing that they must assume responsibility the book suggested that they were currently failing to do just that. Yet, Weizenbaum seemed genuinely relieved and proud when he told Mumford how the graduate students studying AI with whom he had shared the book

have read it and not rejected its message out of hand. To the contrary, they all agree that the things it talks about ought at least to be on the agenda and that the book places them there in a responsible and thoughtful and thought provoking way. (W2M, 11/23/1974)

Granted, it was not so much students’ reactions that Weizenbaum dreaded, but that of his colleagues. Weizenbaum humorously recounted an encounter with John McCarthy, ‘one of the founding cardinals of the artificial intelligence priesthood,’ who expressed a distaste for the book but who could not find any fault in it other than its style (W2M, 3/12/1976). According to Weizenbaum, McCarthy had accused him of learning his ‘writing style from Lewis Mumford,’ adding that ‘John was very puzzled that I seemed so pleased’ at this comment, which was not meant as a compliment (W2M, 3/12/1976). Yet McCarthy's comment on style, and his allusion to Mumford, speaks not so much to a ‘fault’ as a faultline: the growing gulf between Weizenbaum and ‘the artificial intelligence priesthood.’ Whereas the former had become a proud heretic, the latter group was only becoming more fervent in its AI evangelism.

Though Weizenbaum expressed amusement at McCarthy comparing his work to Mumford's, what buoyed him was not this idle comparison to his idol, but a host of other experiences. Weizenbaum noted that since the book's publication he had spoken with many colleagues who had thanked him for his work, telling him that they shared many of his concerns, but ‘have not uttered them publicly’ (W2M, 3/12/1976). It seemed to Weizenbaum, based on these conversations, that his book was for many of these unnamed individuals ‘a kind of legitimation of their own ideas and even of having nonconventional ideas at all’ (W2M, 3/12/1976). Though Weizenbaum expressed to Mumford that he was pleased to take on this role, he also noted how saddened he was to see that so many of his colleagues were unwilling to speak out against the accepted orthodoxy. Their response further demonstrated ‘how urgent it is for everyone who can do so to set examples – and how enormously powerful examples can be’ (W2M, 3/12/1976).

While attending the Rose-Hulman Institute of Technology's conference on ‘Technology at the Turning Point,’ Weizenbaum found himself once more reminded of the reasons why many of his colleagues, even if they agreed with him, refrained from uttering their views publicly. In this regard as well, he recognized that Mumford had paved the path he was now treading. As he described it to Mumford, Weizenbaum delivered a talk attacking the idea of technological inevitability and warning that ‘popular attitudes toward computers have converted them to instruments which tell us what we must hope for … Having been told what we must hope for, we immediately turn computers into instruments which realize our hopes’ (W2M, 4/6/1976). Yet in recounting the event to Mumford, Weizenbaum did not dwell on his own comments, but instead expressed frustration at the talk delivered by the historian Melvin Kranzberg, who had ‘railed against the enemies of technology,’ amongst whom Mumford was identified ‘most emphatically’ (W2M, 4/6/1976). This was certainly not the first time that Kranzberg had criticized Mumford publicly. Several years earlier, Kranzberg had called those who opine on technology's impact on human values ‘intellectual Luddites,’ framing them as ‘concerned, articulate, but ineffectual’ (Kranzberg 1964, 580). That he placed Mumford within this group is made clear in his review of the first volume of Mumford's Myth of the Machine, where Kranzberg fondly recalled Mumford's earlier work – before calling the current Mumford ‘a prophet of doom’ and asking ‘Has something gone wrong with technology? Or just the prophet?’ (Kranzberg 1967, 687).

While Kranzberg's earlier critiques of Mumford had been printed, the conference provided Weizenbaum an opportunity to rise to Mumford's defense, challenge Kranzberg's reading of Mumford's work, as well as his technological optimism. For Weizenbaum, Kranzberg's framing of critics as ‘enemies of technology’ served as an easy way for the celebrants of technology to avoid having to engage with the merits of those critiques. For Weizenbaum the accusation of ‘technological determinist,’ which Kranzberg hurled at Mumford, overlooked Mumford's lifelong commitment to searching for alternative technological modes, while also overlooking how certain technological systems are created in order to remake society in their own image. Commenting on Kranzberg's rhetorical techniques, Weizenbaum described them as a ‘very neat trick’ that frames ‘every good that technology has brought about as a product of the essence of technology, while any evil that might be mentioned was naturally a consequence of the weakness and the fallibility of man’ (W2M, 4/6/1976). Beyond being a tactic to dismiss criticism, Weizenbaum felt that assessments such as Kranzberg's permitted ‘the technological optimist to charge ahead without thought to the consequences his magic may engender’ (W2M, 4/6/1976). The encounter was a discomforting reminder that views such as his and Mumford's were still quite unwelcome within the technoscientific community.

Despite some cheering comments from students, words of support from colleagues, and some positive reviews of the book, encounters of the sort that Weizenbaum had with McCarthy and Kranzberg had a draining impact. Whereas Mumford had ample experience sounding the siren in vain, the failure of Computer Power to overcome the cultural influence of the artificial intelligentsia and their ilk left Weizenbaum deeply depressed. Reflecting on Kranzberg's comments, Weizenbaum told Mumford that he had not been ‘particularly upset’ by them as ‘you and I both know that there is an enormous amount of this kind of nonsense being spoken every day’ (W2M, 4/6/1976). Considering the body of letters Weizenbaum sent to Mumford, it is hard to accept his claim that he was mostly unbothered. While noting that he was doing his best ‘to guard against becoming a “true believer” and a victim of my own skepticism,’ Weizenbaum nevertheless found the ‘superficiality’ with which most people approached computers compelled him to darken his vision (W2M, 10/2/1977). Many of Weizenbaum's public writings feature their fair share of glum assessments, but few equal the mournful heft with which he told Mumford, ‘I have the insuppressible impression that the world is getting exponentially worse while my efforts to save an individual life here and there and a modicum of sanity generally can be increased only linearly’ (W2M, 3/18/1978) – a combinatorial explosion of technoscientific development his critique could not overcome in real time.

In his initial letters to Mumford, Weizenbaum's words were tinged with optimism. Though he had appreciated the enormity of the task before him, and the risks to his reputation it entailed, as Weizenbaum dispatched draft after draft to Mumford he was animated by a faith that speaking out would make a difference. In the aftermath of the book's publication, however, the flames of hope diminished into frail sparks, especially as the years marched on. Weizenbaum continued to derive succour from Mumford's work and their friendship. Yet it seemed that another area in which Weizenbaum came to emulate Mumford was in feeling like the mocked prophet who watches in solitude as the very things they glumly warned of come to pass. Weizenbaum feared that ‘the world is rushing madly and apparently thoughtlessly into a whole series of crises, into all of them with increasing acceleration,’ and though he hoped that he could contribute ‘to the rise of the general level of sanity in our society’ he was increasingly doubtful it would be enough (W2M, 1/19/1984). Looking about the world Weizenbaum asked ‘If one has only ten fingers, which holes in the dam which holds back the tide of barbarism should one attempt to plug?’ (W2M, 1/19/1984).

Where once Weizenbaum had asked Mumford for guidance on how to be a public intellectual, he later sought guidance on how to be at peace with being ignored. Weizenbaum and Mumford both tried to plug the holes created by an overly eager embrace of science and technology, but even their best attempts had not stopped the rising tide from pouring through.

Illuminating metaphors

At the conclusion of ‘The Last Dream,’ an article in which Weizenbaum couched the history of AI in humanity's long obsession with creating new life, he delivered a punchy sentence that neatly summarizes his outlook: ‘Real life is not computable’ (2019, 193). Throughout the article, a copy of which he sent to Mumford, Weizenbaum emphasized that technoscience was not the only way to understand human experience, and, moreover, that the attempt shamefully cheapened that experience. He critiqued prominent members of the Artificial Intelligentsia for overestimating the capabilities of AI, but noted that they were correct in recognizing ‘the readiness of people to believe such things, both then and now’ (181).

Weizenbaum's understanding of the proclivity to put faith in the pronouncements of the Artificial Intelligentsia is one of many reasons his thinking remains important today. Yet what truly makes his perspective on this significant is not his comments on the matter, but the ways in which he explained how this had come to pass. Technological enthusiasm courses through much of American history (Adas 2006; Hughes 1989; Marvin 1988; Segal 2005), and though this optimistic/utopian streak has periodically been challenged by a contrary perspective (Ezrahi, Mendelsohn, and Segal 1994; Kling 1996; Winner 1989) – Weizenbaum's forlorn letters to Mumford capture how the critics felt their warnings could not overcome the widespread enthusiasm for technoscience. In addition to the Mumford-focused confrontation with Kranzberg, Weizenbaum had run-ins with his own ideological opponents as well. In some cases these played out in the pages of popular periodicals (Feigenbaum and McCorduck 1983; Weizenbaum 1983a, 1983b) and soured the relationship in perpetuity (McCorduck 2019;Weizenbaum and Wendt 2015). And in still other cases Weizenbaum's viewpoints seemed to be included just so they could be shot down. For example, his article ‘Once more – a computer revolution’ (1978), reprinted in the volume The Computer Age (Dertouzos and Moses 1979), was followed by two responses that disagreed with him, whereas no other chapter in the book was followed by similar retorts.

Curmudgeonly though Weizenbaum could be, he was no misanthrope. There is a genuine note of disbelief in his comment to Mumford that ‘it's hard to understand sometimes how the world can be so screwed up in spite of the fact that most people are good’ (W2M, 3/25/1980). This matter is doubtlessly one which has concerned many individuals, yet, at least when it came to technology, Weizenbaum and Mumford had an answer. In Mumford's estimation ‘the ultimate religion of our seemingly rational age’ was ‘the myth of the machine,’ wherein an idolatrous belief in the power of science and technology to solve all problems resulted in ‘progress’ becoming a shorthand for technical achievements alone (1970). Over the two volumes of the aptly titled Myth of the Machine, Mumford had gone into detail explaining how this myth was maintained by a new priesthood that had enshrined the computer as its new god (1967, 1970). Explaining how this grim situation could come into being, Mumford described how this new system ‘bribed’ the masses by offering them a share in the technologically produced bounty in exchange for their complicity (1970). Far from rejecting Mumford's analysis, Weizenbaum deepened it, drawing explicitly on the idea of the myth of the machine to argue that ‘technological metaphors … and technique itself … so thoroughly pervade our thought process that we have finally abdicated to technology the very duty to formulate questions’ (1972, 611). The challenge of formulating questions was thus rendered moot as the technological system only permitted questions to which it was itself the answer. As Weizenbaum pithily put it, ‘the computer has almost since its beginning been basically a solution looking for a problem,’ thus it framed every problem as one that only a computer could solve and presented itself as the solution (ben-Aaron 1985, 2). Just as someone with a hammer sees every problem as a nail, to the Artificial Intelligentsia, every problem looks computable.

Weizenbaum stressed the importance of investigating who the victims and victors ‘of our much-advertised technological progress’ would be; how the computer and AI would impact our idea of ourselves; what ‘irreversible forces’ would come into existence thanks to ‘our worship of high technology’; the sort of world these decisions would leave our descendants; and the importance of asking ‘what limits ought’ to be imposed on the application of computation to ‘human affairs’ (1978, 19) – precisely because such questions do not give themselves to easily computable answers. No technological determinist, Weizenbaum emphasized that much was dependent on the choices being made by technologists and he sought to remind his peers that ‘since the beginning of recorded history, decisions having the most evil consequences are often made in the service of some overriding good’ (1976, 273). Weizenbaum, much like Mumford, underscored that the technological situation in which we find ourselves is not the result of inevitability, or the result of irresistible progress, but the result of decisions. And different decisions could have been, and still can be, made.

There are certainly aspects of Weizenbaum's thought that have aged oddly, which is why it is important to see him not as a lone Jeremiah howling in the desert but as a member of a community of critics. Weizenbaum's frequent allusions to the Nazis and the Holocaust can appear anachronistic to those who forget that Weizenbaum fled Nazi Germany in his youth, and his statement ‘we have learned nothing. Civilization is as imperiled today as it was then’ (1976, 256) may appear hyperbolic to those who view the Nazis as an aberration instead of as a product of modernity. Nevertheless, the technological critiques by numerous Jewish critics in the post-war period drew clear links between the experience of the Holocaust and a distrust that new technoscientific advances could be handled responsibly (Anders 2010, 2013; Arendt 1958; Fromm 1968; Jonas 1985). Zygmunt Bauman, for example, drew directly on Weizenbaum to warn of the dehumanizing risks of computers (Bauman 1989, 115–116). While Weizenbaum saw himself as leaning on the work of a range of social critics, his own writing entered into that conversation, and eventually many of those who Weizenbaum had drawn on wound up citing him in return (Ellul 1990; Roszak 1994).

Nevertheless, Weizenbaum's work remains vital today, not because he was an early critic of AI, but because the questions he raised concerning AI and computerized society are still the ones that trouble technological civilization. Computers remain powerful metaphors for describing our world (Chun 2011; Golumbia 2009; Hong 2020; Mosco 2005). Hype around artificial intelligence continues to overstate the capabilities of these systems (Broussard 2018). The myths around AI distract from the humans doing the work (Gillespie 2018; Roberts 2019). The complexity of various systems continues to deepen and exacerbate social divisions (Noble 2018; Pasquale 2015). All the while, computers and AI remain alluring devices onto which societies project their hopes for social transformation (Benjamin 2019; Ames 2019).

Where once Weizenbaum cited past critics in order to place himself in a continuum of technoscientific critique, many contemporary critics cite Weizenbaum to similarly situate themselves (Loeb 2015, 28). Yet what keeps Weizenbaum relevant today is not only his place amongst past critics, but that his critiques still speak directly to the social implications computers and AI. It is an enduring relevance that Sun-ha Hong captures by evoking Weizenbaum's ‘counsel’ that ‘not everything that can be done with technology ought to be done’ (Hong 2020, 186). Contemplating the ‘paradoxical role of computers,’ Weizenbaum observed

on the one hand the computer makes it possible in principle to live in a world of plenty for everyone, on the other hand we are well on the way to using it to create a world of suffering and chaos. (1983c, 10)

For Weizenbaum, much like Mumford, technologies could never be considered in isolation from the societies in which they were found. Thus, it was impossible to assess computers or AI merely by looking at machines. As Weizenbaum put it ‘The computer … is a mirror in which certain aspects and qualities of contemporary America are reflected’ he went on to add that his purpose was not ‘to say anything good or bad about the mirror. I am talking about what it reflects’ (Mulvihill 1986, 135). As for what it reflected? Weizenbaum minced no words, noting that ‘the computer is embedded in our crazy society … and this society is obviously insane’ (Weizenbaum and Wendt 2015, 42).

A light in the dark

In describing the computer metaphor Weizenbaum warned how it had become a ‘lamppost under whose light, and only under whose light, men seek answers to burning questions’ (1976, 158). Considering the darkness threatening to envelop the world, it was understandable that people would flock to the glimmer of that lamppost and satisfy themselves with whatever illumination it could bring. Yet, Weizenbaum was not content to ‘seek answers’ under that glow.

If only certain answers could be cast into the brightness of the lamppost of the computer metaphor, then it was necessary to find other light sources. Thus, in explaining what he hoped to accomplish in his work, Weizenbaum told Mumford that ‘all I have done is to carry a lamp into a dark fortress’ (W2M, 9/2/1974). Noting how Mumford had expressed the belief that merely carrying that lantern is ‘an important thing to do,’ he also credited Mumford with being one of ‘the keepers of its light’ (W2M, 9/2/1974). In expressing gratitude to Mumford for his guidance and assistance, Weizenbaum noted that they had been brought together by a shared set of thoughts and worries. ‘I came to know yours as a lost sailor comes to know a lighthouse on whose wide ranging beam he has come to rely long before he ever [became] its keeper’ (W2M, 11/23/1974). The lamp that Weizenbaum held aloft burnt brightly for a time, and he had sought not only to rally his fellow computer scientists but the broader public as well. Thus, beneath the image of the scientist and the robotic Adam, Weizenbaum had sought to illuminate for the readers of The Baltimore Sun that ‘Perhaps we have even more to fear from people who act as if they were computers than from computers programmed to pretend to be human’ (Weizenbaum 1977b, K2). Despite his efforts to hold the lantern high, the darkness grew as more and more people came to gather around the lamppost of the computer metaphor. And though Weizenbaum strived to be a keeper of the light, he confided that ‘I lean toward the view these days that the lights are going out’ (W2M, 1/19/1984).

Whether one agrees of disagrees Weizenbaum's assessment, his work remains a lantern worth rekindling as we make our way through this dark fortress. This lantern may not tell us the correct path to take, but it will at the very least remind us that we do not have to stay on our present path. That so much of what awaits us remains uncertain and unpredictable only heightens the need for us to carefully evaluate the route we shall take. And thus the question remains, who will be willing to tread this lonely path, littered with accusations of alarmism, in order to hoist this lantern anew?

 Let there be light.

 It is sorely needed.