Abstract
For the public to be prepared to deliberate about the proper role (if any) of artificial superintelligence in society, artificial intelligence literacy needs to address technical, empirical, and normative philosophical topics related to this task. Yet recent reviews of artificial intelligence literacy studies and frameworks suggest that artificial superintelligence receives little to no coverage and that the philosophical dimensions of artificial intelligence literacy are often minimized. Toward supporting the development of artificial superintelligence in literacy approaches, curricula, and policy, we conduct a scoping review of the last 20 years of English-language philosophy articles on artificial superintelligence in the PhilPapers database. We reviewed 65 articles using both bibliometric and thematic forms of analysis. Using qualitative methods, we identify 9 primary themes and 50 subthemes in these articles, as well as the domains of philosophy covered in each. In this set of articles, three topics with an axiological focus—artificial superintelligence value alignment (n = 41), existential risk (n = 33), and machine ethics (n = 31)—were identified as the most common themes. We also identify the foundational texts most frequently cited by articles in the study set. By placing these themes in dialogue with artificial intelligence literacy reviews, we identify three main types of opportunities to support deeper artificial superintelligence literacy: (1) content opportunities, (2) skill opportunities, and (3) pedagogical opportunities. Specifically, we recommend expanding both artificial superintelligence and philosophical content in artificial intelligence literacy, developing argument assessment skills through computational thinking, and extending artificial intelligence literacy pedagogies to support philosophical inquiry into foundational principles of artificial intelligence and artificial superintelligence.
Keywords
Introduction
Artificial superintelligence (ASI) is often defined as a form of artificial intelligence (AI) capable of solving any problem a human can solve at a significantly greater than human speed, quality, or scale (Bostrom, 2014; Russell and Norvig, 2021). ASI is, thus, a more advanced form of artificial general intelligence (AGI), which refers to AI technologies capable of work on par with humans in a wide range of domains. Leading AI researchers argue that for better or for worse, achieving ASI will be the most significant event in human history, redefining health, economics, politics, warfare, and education (Russell, 2019, 2021). If predictions by leading AI experts are correct, then the future lives of today’s students are likely to be impacted by ASI in profound ways, making ASI preparation one of the most important educational issues of our time. The largest survey of leading AI researchers to date found that, “if science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047” (Grace et al., 2024: 1). Although 68.3% of these same AI researchers thought “good outcomes” from ASI are “more likely than bad,” the survey also found that “57.8% considered extremely bad outcomes (e.g. human extinction) a nontrivial possibility (⩾ 5% likely)” (Grace et al., 2024: 12).
Are educational systems preparing the public to deliberate about and shape ASI futures? Contrary to what one might expect, recent literature reviews suggest that current AI literacy initiatives include little to no coverage of the unique existential risks and opportunities raised by ASI nor of the foundational philosophical questions at the root of the design, development, evaluation, and regulation of these technologies (Casal-Otero et al., 2023; Kaspersen et al., 2022; Tanchuk, 2025). To have an informed say in the shape of these systems, including the decision whether they should be built at all, the public needs to be prepared to reflect on principles of design and risk-benefit assessment prior to their arrival (Gabriel, 2020; Tanchuk, 2025). This future-oriented task is not only technical but also involves ontological, axiological, and epistemological questions: What evidence would be sufficient to justify the belief that we could control a system radically more powerful than all of humanity? How should intelligence in a powerful AI system be defined? What limits, if any, should there be on reengineering or “enhancing” human biology to interface with such systems? Who, if anyone, should control such technologies? What values and distributive rules should orient these AI systems? One rigorous treatment of these questions, guiding the first principles of design and risk-benefit assessment of ASI systems, can be found in the philosophical literature. Yet there is a current lack of systematic or scoping reviews of philosophical work on ASI. Without such reviews, it is difficult for educators and policy makers to identify which philosophical themes warrant inclusion in AI literacy curricula. In this study, we report the findings of a scoping review of English-language academic journal articles on ASI in the past 20 years in the PhilPapers database, a “comprehensive index and bibliography of philosophy” (PhilPapers: Online Research in Philosophy, n.d.). We then analyze these findings in relation to existing AI literacy reviews to provide a basis to help teachers, policy makers, and curriculum designers better support learners in identifying central philosophical themes implicit in the design, risk assessment, and development of ASI systems.
AI literacy, philosophical inquiry, and ASI
Since 2020, AI literacy studies and frameworks have proliferated, mirroring advances in machine learning and generative AI globally (Casal-Otero et al., 2023; Ng et al., 2024). In the academic literature, some distinguish AI literacy from AI education more broadly (Long and Magerko, 2020; Ng et al., 2024; Yim and Su, 2025; Zhang et al., 2025). AI education may involve any form of teaching about AI, including highly technical engineering topics. AI literacy, in contrast, has been advanced as a subset of education about AI that is necessary for any member of society to successfully navigate a world with AI (Kaspersen et al., 2022; Long and Magerko, 2020; Ng et al., 2024; Yim and Su, 2025; Zhang et al., 2025). One of the most widely influential definitions of AI literacy is, thus, as “a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” (Long and Magerko, 2020: 2). 1
Here, we review the current state of AI literacy literature. First, we find that democratically evaluating possible ASI futures is largely absent from this AI literacy work. We then identify promising strengths that might be extended from AI literacy to address ASI literacy and limitations that point to the need for deeper engagement with the philosophical literature on ASI. Second, by way of strengths, we note that some AI literacy research and frameworks helpfully embed philosophical reflection on first principles in technical design tasks; this practice is not, however, scaled up to its full potential. By way of limitations, we observe, third, that current AI literacy frameworks typically do not address important relationships between technical and ethical, ontological, and epistemic questions necessary for assessing ASI’s potential social impact, even in the context of narrow AI.
Let us elaborate on these three themes. First, as mentioned above, recent reviews of AI literacy do not include any significant treatment of unique issues related to ASI (Cf. Casal-Otero et al., 2023; Long and Magerko, 2020; Ng et al., 2024; Zhang et al., 2025). Although some studies discuss workforce and societal implications of automation (e.g. Zhang et al., 2023), the aim of achieving ASI and the unique challenges such superintelligent technologies raise receive little to no significant attention in this literature or these frameworks. In Zhang et al.’s (2025) systematic review of reviews on AI literacy, for example, superintelligent AI is not mentioned, despite the fact that achieving ASI is a, if not the, central goal shared by all frontier AI companies. If we are to prepare democratic societies to deliberate in advance of the arrival of these technologies, greater attention to questions of ASI design and risk-benefit assessment is needed. Although imperfect, democratic deliberation and regulation remain one of the most scalable potential checks, regionally, nationally, and internationally, on the increasingly concentrated power and financial incentives of frontier AI companies. Without such democratic steering, efforts to build increasingly powerful forms of AI are arguably less likely to reflect the public interest in general and, in particular, the interests of less powerful groups. As Google DeepMind ethicist Iason Gabriel (2020) argues, since “the methods we use to build artificial agents may influence the values or principles we are able to encode” (p. 415), it is arguably essential that these democratic conversations occur before ASI technologies arrive.
Second, recent reviews of academic literature and curricular frameworks consistently report that AI ethics and societal impact are areas of concern (Akgun and Greenhow, 2022; Casal-Otero et al., 2023; Ng et al., 2024; Touretzky et al., 2019; Zhang et al., 2025). Some researchers and educators have made notable efforts to embed philosophical ethics curricula within technical projects in K–12 computer science education (Ali et al., 2021; Grosz et al., 2019; Williams et al., 2023; Zhang et al., 2023). Often, however, ethical and social impact topics in AI literacy frameworks are presented in ways that limit the role of systematic philosophical inquiry and argumentation into the typically controversial and contested first principles guiding these evaluations.
One way the treatments of philosophical aspects of AI literacy are often limited is in the content covered. As others have observed within AI literacy frameworks and studies, technical topics typically receive greater emphasis than questions of ethics or societal impact (e.g. Bozkurt et al., 2021; Kaspersen et al., 2022). In a recent systematic review of academic literature and curricular frameworks, for example, Casal-Otero et al. (2023) identify four thematic categories within learning experiences focused on understanding AI, only one of which is focused on social impact: (1) learning to recognize artifacts using AI, (2) learning how AI works, (3) learning tools for AI, and (4) learning for life with AI (pp. 6–9).
In addition, the values identified in AI ethics and societal impact questions are often treated as largely reducible either to knowing and applying a handful of predefined concepts, such as bias, fairness, accountability, transparency, and equity or ethics to AI tools (e.g. AI4K12, 2022; Akgun and Greenhow, 2022; Ng et al., 2024; Touretzky et al., 2019) or matters of merely subjective opinion (e.g. Kaspersen et al., 2022; Norouzi et al., 2020; Payne, 2019). By contrast, philosophical approaches tend to highlight the reasoned and contested nature of value frameworks. A leading philosophical analysis of AI ethics policy documents, advanced by Floridi and Cowls (2022), for example, holds that one can reduce the complexity in those documents to five core AI ethics principles: (1) beneficence, (2) nonmaleficence, (3) autonomy, (4) justice, and (5) explicability. The definition and relative weight of the values included on lists such as that of Floridi and Cowls (2022), however, are widely recognized in philosophical AI ethics as admitting of multiple reasonable competing interpretations, requiring arguments in favor of one specification or another (cf. Beauchamp, 2003). Errantly specified values with undesirable or unintended effects can become existential threats in the specific context of ASI, where it may become more difficult, if not impossible, to correct errant designs (Bostrom, 2014; Gabriel, 2020; Russell, 2019; Yampolskiy, 2024). To deepen deliberation about ASI futures, the public needs to be prepared not only to understand philosophical concepts but to evaluate arguments about such high-stakes normative questions.
Third, although the ethical and political topics sometimes included in AI literacy are important, they are not unrelated to questions of ontology and epistemology—about what there is and how we know in relation to AI systems. With rare exceptions, these questions about, for example, the nature of machine intelligence and its similarities and differences with conscious human intelligence receive limited attention in this literature. 2 The neglect of this sort of ontological question and the related epistemic question of how one knows when other minds are present or absent is central to understanding ways AI systems may mislead or support learners in forming healthy and ethical relationships with these systems. Since consciousness is a constituent of central human and educational goods like cognitive and emotional empathy, as well as claims to moral respect, this is arguably a significant oversight.
The philosophical literature on ASI, by contrast with the AI literacy literature, is deeply engaged with the ethical, epistemic, and ontological issues implicit in the design and assessment of these technologies. Despite these strengths, the rigorous investigations of ethical, epistemic, ontological, and social impact questions related to ASI in philosophical literature are overwhelmingly disconnected from broader efforts to educate about AI in K–12 education. To remedy this disconnection and chart a more unified path forward, an analysis of this philosophical literature and a synthesis of it with the AI literacy literature is needed.
Research questions
To address the lack of scoping reviews of philosophical work on ASI, we ask the following two questions:
What philosophical articles and texts about the nature and value of ASI are central to English-language philosophy journal articles in the last 20 years?
What philosophical themes about the nature and value of ASI are most central in English-language philosophy journal articles in the last 20 years?
We then place our findings in dialogue with the existing reviews of AI literacy toward identifying opportunities for adding depth to the latter body of work and extending it toward a more robust consideration of issues related to ASI systems.
Methods
The research method we used to answer our research questions is the scoping review adapted for philosophical research. Scoping review methods were first introduced by Arksey and O’Malley (2005). We based our methods on the approach offered by Arksey and O’Malley because of its flexibility and applicability for philosophical research. Our approach draws on two purposes Arksey and O’Malley (2005) identify for scoping reviews: (1) “To determine the extent, range and nature of research activity” (p. 21) in a domain and (2) “To summarize and disseminate research findings” (p. 21), in our case for the particular purposes of enhancing ASI literacy teaching and research. In the following subsections, we summarize the key stages of the scoping review process: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; and (5) collating, summarizing, and reporting the results (p. 22).
Identifying research questions
The research questions above were developed through an iterative process of discussion within the research team. We aimed to capture the scope of topics that would be most relevant to researchers, curriculum designers, and policy makers seeking to enable learners to deliberate democratically about the future forms ASI might take. Based on these considerations, we decided to focus our attention and resources on (a) academic journal articles discussing ASI in (b) the past 20 years. English-language publications were selected, as this was the only language common to all research team members.
We selected academic journal articles as our primary unit of analysis due to their combination of scholarly rigor and responsiveness to rapidly emerging trends around ASI in the field. In addition, we conjecture that a review of academic journal articles would adequately capture themes gaining uptake in the academic discourse as a whole. We do not include academic books or edited volumes. One measure of the influence of such works, however, is their discussion in journal articles. To minimize redundancy and manage limited resources, we decided to approach themes from other types of academic works indirectly via journal articles.
We hypothesized that a 20-year time horizon would provide sufficient scope to identify the central current themes of inquiry into ASI that would be most relevant to curriculum designers, researchers, and policy makers. The current live issues are likely to be more relevant than those that have receded into the history of the discipline. At the same time, we recognize that there may be important and still widely influential works prior to the 20-year cut line that may be important to compile for future research and AI literacy work.
Our first question, “What philosophical articles and texts about the nature and value of ASI are central to English language philosophy journal articles in the last 20 years?” seeks to identify influential articles and texts within recent debates in philosophy about ASI. By identifying articles and texts cited within this more recent set of articles, we are also able to identify touchstone works and media that are either nonacademic or from prior to the 20-year cut line but still influencing the current debates in academic journals. These resources may be of use to researchers and AI literacy proponents in developing curricular resources, policy, or standards. Our second research question, “What philosophical themes about the nature and value of artificial superintelligence are most central in English language philosophy journal articles in the last 20 years?” aims to identify the topics that philosophical researchers are prioritizing thematically in academic journal articles. This question allows us to create a map of the themes and issues philosophers are engaging with in the space of ASI and to place these themes in dialogue with the AI literacy literature. Both research questions focus attention on articles and texts analyzing the nature and value of ASI, an intentionally broad semantic range that aims to be inclusive of a diverse array of themes and texts. Due to linguistic limitations of our research team, we leave analyses of non-English philosophical works to future researchers to place in dialogue with our work. We note that we also leave analyses to other forms of conceptual and theoretical work (e.g. in fields such as curriculum theory) that may not be indexed in the PhilPapers database for future research. We advance this scoping review as an incremental contribution toward mapping one important domain of scholarly work as it bears on AI literacy but do not claim it is exhaustive.
Identifying relevant studies
We included articles (1) written in English; (2) in peer-reviewed academic journals; (3) published on or after 1 January 2004; (4) focused on issues of axiology, epistemology, or ontology (i.e. were philosophical, not empirical); and (5) focused on ASI. We excluded articles that were (1) not written in English, (2) not published in peer-reviewed academic journals, (3) published before 1 January 2004, (4) were empirical or descriptive studies, and (5) were not focused on ASI.
We identified the PhilPapers database as an appropriate information source to implement the search process for our unit of analysis. PhilPapers is [. . .] a comprehensive index and bibliography of philosophy maintained by the community of philosophers. We monitor all sources of research content in philosophy, including journals, books, and open access archives. We also host the largest open access archive in philosophy. Our index currently contains 2,674,417 entries categorized in 5,982 categories. (Bourget and Chalmers, 2023)
PhilPapers’ comprehensive scope afforded the opportunity to identify the most common themes and most central papers in the philosophical discourse on ASI in the anglophone world.
All searches were performed using PhilPapers search engine’s “extended query” function on works from 2004 until 2024. All searches were conducted between 1 June 2024 and 1 August 2024. Per our research questions, we limited the scope of our search to publications in the past 20 years at the time of the search. The search string we used, following the PhilPapers search guidelines, was “(super + intelligen*) | superintelligen*” and was designed to return any philosophical work from 1 January 2004 to June 2024 mentioning superintelligence or its cognates (e.g. superintelligent). 3 Our initial search delivered 273 results. It is important to note that this search strategy yielded results that include the key term superintelligence, but other papers on topics related to ASI may not have been captured if this term was not present in the searchable fields on PhilPapers.
Study selection
We then conducted a screening review of abstracts and initial coding to ensure inclusion criteria were met. From an initial yield of 273 works, 186 were excluded during abstract screening. Reasons for exclusion included not being written in English (n = 10), not being published in peer-reviewed academic journals (n = 83), being descriptive rather than philosophical (n = 32), not being focused on ASI (n = 44), or being a duplicate (n = 17). During the full manuscript read of the 87 remaining sources, an additional 22 works were excluded. At this stage, 20 works were excluded for not meeting the second criterion. All of these works were either not published as an academic article or published outside of peer-reviewed academic journals. They were, for example, self-published (e.g. to Research Gate or PhilPapers only; n = 14), published to a nonpeer-reviewed preprint server (n = 2), published as conference proceedings (n = 1), or published as an editorial, a master’s thesis, a book chapter, or other nonacademic work (n = 3). One work was excluded based on the third criterion—not being published after 1 January 2004. The research team was unable to access one paper. The result is a study set of 65 papers meeting inclusion and exclusion criteria.
Charting the data
Papers that met the inclusion criteria were organized into a spreadsheet and assigned unique paper IDs. Data charting proceeded in three phases. First, an initial set of data was manually extracted and recorded in a spreadsheet by one member of the research team. The data items included the author(s), article title, abstract, full article, year of publication, publication venue, type of publication (e.g. journal article), number of Google Scholar citations, and areas of philosophy.
The second phase of data charting employed a qualitative coding process. Authors 1 and 3 conducted an initial review of abstracts, developing a set of open codes identifying ASI themes (Corbin and Strauss, 2015: 220). They then organized these open codes through a round of axial coding into an initial codebook of ASI themes and subthemes (Corbin and Strauss, 2015: 239). Next, all three authors conducted a full read of 10% (n = 7) of the 65 papers that met the inclusion criteria to assess the initial codebook. After discussion and refinement of the initial codes based on this sample, Authors 1 and 3 conducted a full read of the remaining 58 papers and iteratively revised the codebook to establish coherence between the themes, subthemes, and emergent open codes across the full set of papers. In any case where a new theme or subtheme was added, prior papers were recoded to reflect the revision to the codebook and maintain consistency. The codebook with definitions of all themes and subthemes is provided in Appendix 1.
In the third phase of data charting, Authors 1 and 4 developed a strategy to extract reference lists for all 65 papers in the set. Reference lists were extracted using Notebook LM, first on an initial sample of 10 papers (15% of all papers) to establish reliability. Using this approach, based on manual verification, there were no wrongly included or excluded items from the reference lists of the initial set of 10 papers. The process of extracting references using Notebook LM was extended to the rest of the set of papers with periodic manual verification by Author 4. This process yielded 3173 citations, including duplicates in different citation styles. Python scripts were used to combine multiple citations in diverse style guides into countable unique instances of each cited paper. Utilizing Python and cosine similarity (90% threshold), we clustered citations, identifying near-duplicates and true identicals. This process yielded exactly 2600 unique references. Finally, Author 4 extracted total Google citation metrics for each paper included in the study as of 19 December 2024.
Collating, summarizing, and reporting results
In order to synthesize the results after data charting, Author 2 used Stata to tabulate code frequencies for the areas of philosophy, as well as all themes and subthemes. In addition, the mean and spread of Google Scholar citations in the set were calculated. Finally, Authors 1, 2, and 4 compiled a bibliographic report and histograms and charts reporting each of these counts.
Results
Here, we present the results of the literature search. We first report on the philosophical articles identified. Then, we present a bibliometric analysis of the set of philosophical ASI papers that we reviewed, including details extracted from PhilPapers as well as citation count data from Google Scholar. We present data on which sources in or beyond the study set are most cited by articles in the PhilPapers study set. We then present a qualitative analysis of the philosophical themes represented in the full set of reviewed articles. A total of 87 papers were obtained through the PhilPapers database search. Of these 87, 65 were determined to meet the inclusion criteria for the study. The 65 included articles are marked by a star in the reference list.
Characteristics of the sources
Figure 1 shows the number of ASI-related philosophical papers published per year over the past 20 years included in our analysis (from 1 January 2004 through June 2024). Of the 65 papers in the set, none were published in 2004, one was published between 2005 and 2009, 10 between 2010 and 2014, 25 between 2015 and 2019, and 29 between 2020 and 2024. The 65 sources in the set were published by 30 different journals, with the highest number (18 papers) appearing in AI & Society: Journal of Knowledge, Culture, and Communication, which is published by Springer (Table 1).
Number of ASI-related philosophical papers between June 2004 and June 2024.
Number of papers per journal.
Google Scholar citation counts ranged from 0 to 468 within the set, with a mean of 35 citations. Sixty-two of the 65 sources had at least one citation; 43 sources had at least 10 citations, and 6 had at least 100 citations. The six most cited papers are listed in Table 2.
Most cited papers on Google Scholar.
We also examined the reference lists contained within our data in order to identify foundational texts (citation hubs) relevant to philosophy of ASI. Analyzing reference lists in our data of 65 papers yielded exactly 2600 unique citations. In the set of 2600 citations, sources that were cited five or more times by other papers within the data were identified, resulting in a list of the top 31 referenced foundational works (see Appendix 2 for the full list). Within this set, Nick Bostrom was the most frequently cited author, contributing to seven of the entries. The most cited work, Bostrom’s (2014) book Superintelligence: Paths, dangers, strategies, was cited by 40 papers in the set. Following Bostrom, Eliezer Yudkowsky appeared as the second most represented author in the top 31 papers (n = 4 papers). The 10 most cited foundational sources, along with their citation counts, are visualized in Figure 2.
Top 10 foundational works most cited by papers in the study data set.
Thematic analysis of the sources
Areas of philosophy
The first thematic analysis we conducted coded for the three main branches of philosophy—axiology, epistemology, and ontology—represented in the papers. Papers could include multiple branches of philosophy. Within the set of 65 papers, 89% (n = 58) addressed questions of axiology. Examples of axiological questions explored included questions about value alignment and control of ASI (e.g. Agar, 2016; Müller and Cannon, 2022; Scheessele, 2022), how we should assess and mitigate potential existential risks posed by ASI (e.g. Armstrong et al., 2012; Carayannis and Draper, 2023; Jebari and Lundborg, 2021), and challenges in conceptualizing robot rights and responsibilities in the context of ASI (e.g. Andreotta, 2021; Gordon, 2022; Harris, 2019; Scheessele, 2022). Ontological questions arose in 60% (n = 39) of the papers. These questions ranged from how to conceive of superhuman AI (e.g. Hutter, 2012; Nagl, 2022) and its relationship to agency or consciousness (e.g. Primiero, 2017; Torrance, 2012) to questions about transhumanism, consciousness uploading, and the metaphysics of personal identity (e.g. Coseru, 2024; Steinhart, 2008; Torrance, 2012). Finally, 42% (n = 27) addressed questions of epistemology. Epistemological questions included how to know when other minds are present (e.g. Harris, 2019; Nyholm, 2019; Shanahan, 2012), justification for predictions about how and when ASI is likely to occur (e.g. Batin et al., 2017; Sotala, 2017), and epistemic justification for claims about existential risk (e.g. Danaher, 2015).
Themes and subthemes
Papers in the sample addressed 9 primary themes, which were further divided into 50 subthemes. The most frequent theme was alignment (n = 41), followed by existential risk (n = 33) and machine ethics (n = 31). Figure 3 presents the number of papers in the set addressing each of the nine primary themes. In what follows, we describe each theme and its subthemes in more detail. (See Table 3 for full details on subtheme frequency across all themes.)
Number of papers addressing each theme.
Number of papers addressing each subtheme.
Note. Papers could be coded into more than one subtheme.
Alignment
We have defined ASI alignment as the problem of aligning ASI systems to the right values or human goals. Since defining these values and goals is the work of many papers in the set, we leave the definitions of the values or goals intentionally open.
Across the 65 papers, alignment appeared with the highest frequency, in 41 papers. Eight alignment subthemes were identified in these 41 papers 73 times. Alignment subthemes include ethical value in alignment (n = 31), implications of instrumental reason in ASI alignment (n = 10), alignment of ASI to political values (n = 7), nonexistential risks of ASI misalignment (n = 6), the orthogonality thesis (which asserts that intelligence and final goals are independent of each other; n = 6), coherent extrapolated volition (a counterfactual ideal of what humans would want if they were more well informed and rational; n = 5), bias in alignment (n = 4), and ASI confinement (n = 4).
In the ethical value in alignment subtheme, many authors address topics related to whether and how ASI might support or undermine human flourishing and other ethical values (e.g. Bostrom, 2012; Corabi, 2017a; Danaher, 2015; Davis, 2015; Dung, 2024; Gallow, 2024; Goertzel, 2015; Gruenwald, 2020; Haselager and Mecacci, 2020; Lorenc, 2015; Nagl, 2022; Peeters et al., 2021; Scheessele, 2022; Sotala and Yampolskiy, 2015; Søvik, 2022; Sparrow, 2024). Others advance arguments about how we should conceive of the values used to assess whether ASI is aligned (e.g. Lorenc, 2015; Moret, 2023; Nagl, 2022; Scheessele, 2022; Søvik, 2022; Sparrow, 2024). Discussions of the role of instrumental reason in ASI alignment frequently focused on assessing or deploying Nick Bostrom’s (2012) “instrumental convergence” thesis (e.g. Davis, 2015; Gallow, 2024; Müller and Cannon, 2022). The instrumental convergence thesis asserts that regardless of an ASI system’s ultimate goals, the system will likely have a set of convergent instrumental values—such as pursuing power or resources—that pose existential risks to humanity (Bostrom, 2012). Contesting this thesis, Davis (2015) and Primiero (2017) argue that there are feasible technical methods to block a superintelligence from instrumentalizing the world to a single goal or to orient an ASI to ethical values. Similarly, Müller and Cannon (2022) argue that a standard argument for existential risk rests on an equivocation that renders it unsound.
Other papers have focused more squarely on political values in ASI alignment (Dodonova et al., 2023; Jecker et al., 2024; Luck, 2025; 4 Peeters et al., 2021; Sparrow, 2024; Totschnig, 2019; Yampolskiy, 2022). Totschnig (2019) argues that human–ASI relationships should be understood not in terms of human control of ASI but as a political relationship between agents. Sparrow (2024) draws on political values to argue that the mere existence of ASI systems, even if they could be made benevolent, would pose a threat to human political freedom because they leave humans in a state of domination. Thus, even if we could build a “human-friendly” ASI, doing so would still be insufficient for us to be politically free. Luck (2025), by contrast, argues contra Sparrow (2024) that a benevolent ASI need not make us less free. Others highlight political risks of censorship, propaganda, manipulation, and a loss of political autonomy under ASI (Dodonova et al., 2023; Peeters et al., 2021; Yampolskiy, 2022). Although much of the alignment literature deals with existential or catastrophic risks that might lead to human extinction or lives not worth living, others point out that ASI may also (or only) pose other significant risks—those that affect less than the whole human population or are at lower levels of harm (Bradley, 2020; Casacuberta and Guersenzvaig, 2019; Çelik, 2023; Jecker et al., 2024; Müller and Cannon, 2022; Sotala and Gloor, 2017). Such risks have been found in, for example, unfair algorithmic biases in policing, employment, or the distribution of benefits of ASI (Bradley, 2020; Casacuberta and Guersenzvaig, 2019; Jecker et al., 2024; Lorenc, 2015; Müller and Cannon, 2022). 5 Similarly, Sotala and Gloor (2017) argue that ASI could pose but also prevent significant “suffering risks” to humans (p. 389) that are significant but nonexistential. Others argue that existential risk discussions risk distracting us from assessing these other serious forms of AI risk and benefit (Jecker et al., 2024).
By contrast with efforts to align ASI with a set of known human ethical or political values, discussions of Yudkowsky’s (2004) concept of coherent extrapolated volition focus on the feasibility of having ASI use its superior intelligence to determine the values that humans would want it to align to if they were more intelligent (Carayannis and Draper, 2023; Lorenc, 2015; Moret, 2023; Søvik, 2022; Yampolskiy, 2022). Moret (2023), for example, argues for an extension of Yudkowsky’s (2004) concept to include what all sentient beings would want. Bostrom’s (2012) orthogonality thesis (which asserts the separability of intelligence from final goals or purposes in ASI) has been both refined by others and contested in the literature (Corabi, 2017a; Dung, 2024; Gallow, 2024; Goertzel, 2015; Müller and Cannon, 2022). Finally, some scholars analyze the possibility of confining or restricting ASI in various ways as a way to keep it aligned to human values and interests (Armstrong et al., 2012; Davis, 2015; Sotala and Yampolskiy, 2015; Yampolskiy, 2012). Yampolskiy (2012), for example, considers ways a confined “oracle AI” that cannot act in the world might be made securely, while Yampolskiy (2022) and others (e.g. Armstrong et al., 2012) argue that even a confined ASI would pose significant risks.
Existential risk
Existential risks posed by ASI are closely related to questions of ASI alignment. If we (plausibly) assume that humans have significant value and should exist, then existential risks are a subtype of ASI misalignment to the right values (Cf. Scheessele, 2022). Some, however, have pointed out that an aligned ASI system could pose an existential risk to humans to realize some higher set of values or aims (e.g. maximizing knowledge in the universe) on, for example, some utilitarian moral theories (e.g. Lorenc, 2015; Müller and Cannon, 2022: 9; Prinz, 2012: 86).
Across the 65 papers, existential risk appeared with the second-highest frequency, in 33 papers. Seven existential risk subthemes were identified in these 33 papers 60 times. Subthemes include assessing existential risk (n = 19), managing existential risk (n = 12), political forms of existential risk (e.g. domination or tyranny; n = 10), the view that ASI is not an existential risk (n = 7), the King Midas problem (whereby unintended consequences pose existential hazards; n = 5), the existential risk of war/terrorism (n = 5), and discussions of the probability of “doom” (n = 2).
Topics in these papers included arguments for and against the thesis that ASI poses an existential risk and how these weigh against existential benefits (e.g. Agar, 2016; Bostrom, 2012; Ćirković, 2022; Corabi, 2017b; Davis, 2015; Dung, 2024; Gallow, 2024; Gordon, 2022; Jecker et al., 2024; Müller and Cannon, 2022; Yampolskiy, 2022) and strategies for modeling, assessing, and managing existential risk (e.g. Armstrong et al., 2012; Boyd and Wilson, 2018; Bradley, 2020; Danaher, 2015; DeCanio, 2018; Gill, 2016; Goertzel, 2015; Jebari and Lundborg, 2021; Sotala and Gloor, 2017; Sotala and Yampolskiy, 2015; Sparrow, 2024; Turchin et al., 2019; Yampolskiy, 2012, 2022). Some specific threats considered in the literature reviewed include: the political threat of global domination, where an adversary uses ASI in warfare (Carayannis and Draper, 2023); the domination of humans by ASI (Luck, 2025; Sparrow, 2024); and variants of the King Midas Problem, whereby well-intentioned aims given to an ASI system yield catastrophic unintended consequences (Bostrom, 2012; Davis, 2015; Dung, 2024; Gallow, 2024; Totschnig, 2019). Finally, some papers address the likelihood of “doom” (sometimes referred to in popular culture as the “p-Doom” or probability of very bad outcomes due to ASI (Danaher, 2015; Prinz, 2012).
Machine ethics
We define machine ethics as the study of ethical behavior by machines that are moral agents or the moral agency of machines. Themes from machine ethics were prominent in 31 papers. Four machine ethics subthemes were identified in these papers 43 times. Subthemes include ethical value in ASI machine agency (n = 16), moral status of ASI (n = 10), moral agency of ASI (n = 9), and the motivation of ASI agents (moral or otherwise; n = 8). A number of papers explore whether ASI agents would necessarily have adequate moral values or whether they would be likely to (or would necessarily) fall short of moral adequacy in some ways (e.g. Bostrom, 2012; Boyles and Joaquin, 2020; Corabi, 2017a; Dung, 2024; Luck, 2025; Nagl, 2022; Søvik, 2022; Szocik et al., 2020). Relatedly, a number of papers consider if and under what conditions AI systems have moral patiency or rights (e.g. Andreotta, 2021; Gordon, 2022; Harris, 2019; Nyholm, 2019; Torrance, 2012). Others explore the nature of machine moral agency and motivation (e.g. Armstrong et al., 2012; Boyles and Joaquin, 2020; Casacuberta and Guersenzvaig, 2019; Corabi, 2017a; Dung, 2024; Nagl, 2022; Nyholm, 2019; Sotala and Yampolskiy, 2015). Jebari and Lundborg (2021), for example, consider whether developing domain-general machine agency spontaneously from narrow forms of AI through self-improvement is possible due to the nature of motivation.
Philosophy of mind
Themes related to the philosophy of mind, those bearing on the nature of minds or our knowledge of minds, appeared in 27 papers. Seven subthemes related to philosophy of mind appeared in these papers a total of 40 times. The most frequent subtheme was intelligence, which appeared in 18 papers. The remaining subthemes appeared 5 or fewer times each: computationalism (n = 5), biological theory of consciousness (n = 5), the hard problem of consciousness (n = 5), explainable AI (XAI; n = 4), and knowledge of other minds (n = 3).
Topics in these papers included a range of considerations related to defining the nature of intelligence in relation to sentience, artificial perception, reasoning, and motivational states (Bostrom, 2012; Brödner, 2018; Corabi, 2017b; Davis, 2015; Dodonova et al., 2023; du Toit, 2019; Dung, 2024; Hoffmann, 2023; Hutter, 2012; Jebari and Lundborg, 2021; Lorenc, 2015; Nagl, 2022; Peeters et al., 2021; Primiero, 2017; Shanahan, 2012; Shymko, 2018; Sotala, 2017; Torrance, 2012). Others consider whether minds are well-understood computationally (Chalmers, 2012; Mandelbaum, 2022; Primiero, 2017; Steinhart, 2008; Torrance, 2012) and whether we have reason to believe that silicon or other nonbiological substrates can support consciousness (Andreotta, 2021; Chalmers, 2012; Coseru, 2024; du Toit, 2019; Mandelbaum, 2022). Whereas the first set of subthemes are primarily about the nature of the mind and the conditions for its existence, other papers address questions related to determining when and why one is justified in believing another entity has a mind (Andreotta, 2021; du Toit, 2019; Harris, 2019; Mandelbaum, 2022; Nyholm, 2019) and whether biological substrates similar to those supporting human consciousness help justify claims to know that consciousness is present in other creatures with similar biology (Andreotta, 2021; Chalmers, 2012; Coseru, 2024; du Toit, 2019). Philosophers in the set also consider the intelligibility and explainability of artificial superintelligent minds by human minds (Peeters et al., 2021; Primiero, 2017; Schradle, 2020; Shanahan, 2012) and the “hard problem of consciousness”—the problem of explaining why and how physical processes give rise to subjective qualitative or phenomenal experiences (Andreotta, 2021; Chalmers, 2012; du Toit, 2019; Primiero, 2017; Torrance, 2012).
Transhumanism
Many of the topics in transhumanism—issues related to transcending the human condition—are logically dependent on answers to questions raised in the philosophy of mind. For example, if minds cannot exist in nonbiological substrates, then the possibility of transcending the human condition by uploading one’s consciousness to a silicon-based computer is a nonstarter. Transhumanism was prominent as a theme in 21 papers. Seven related subthemes appeared 45 times across this set of papers, with uploading consciousness appearing with the highest frequency (n = 14). Additional subthemes related to transhumanism included augmentation (e.g. cyborgs; n = 9), enhancement of humans (n = 6), personal identity (n = 5), prohumanism (n = 4), the simulation hypothesis (n = 4), and immortality (n = 3).
Under the theme of transhumanism, the feasibility and coherence of transcending the human condition by uploading human consciousness to nonbiological systems like silicon-based computers has received significant and sustained attention (Batin et al., 2017; Chalmers, 2012; Ćirković, 2022; Galanos, 2017; Lorenc, 2015; McIntosh, 2010; Mandelbaum, 2022; Moret, 2023; Prinz, 2012; Schradle, 2020; Sotala and Yampolskiy, 2015; Steinhart, 2008; Torrance, 2012; Turchin et al., 2019). Some have directly addressed topics related to augmenting human biology with nonbiological systems toward more “cyborg” life forms (e.g. Batin et al., 2017; Ćirković, 2022; Galanos, 2017; McIntosh, 2010; Schradle, 2020; Steinhart, 2008). Still others examine ways that interactions between humans and ASI systems or agents could directly enhance or create incentives to enhance human biology or intelligence (Batin et al., 2017; Harris, 2018; Isac, 2010; McIntosh, 2010; Schradle, 2020; Sotala and Yampolskiy, 2015). In all such cases, ethical questions arise here, such as how best to understand the moral relationships between unenhanced humans, cyborgs with various forms of artificial enhancement, and conscious robotic systems (Galanos, 2017; Harris, 2018, 2019; Lorenc, 2015).
Many scholars address how best to determine if and when a person survives various changes related to augmenting, uploading, or duplicating consciousness (Coseru, 2024; Mandelbaum, 2022; Torrance, 2012). Some scholars offer analyses of the unique nature and value of humanity as a potential check against transhumanist programs (e.g. du Toit, 2019; Lorenc, 2015; Scheessele, 2022). This set of philosophical papers also includes arguments about the probability that we currently live in a simulation generated by ASI (Chalmers, 2012; Prinz, 2012) and the potential for achieving immortality through uploading consciousness or transforming the current human condition (Batin et al., 2017; Lorenc, 2015; Schradle, 2020).
ASI: How?
The question of how we could or should (if possible) create ASI appeared as a theme in 21 papers. We identified seven subthemes related to this question, which appeared a total of 32 times. These included the idea of a singularity—a process of rapid recursive technological improvement toward ASI (n = 13), that ASI is possible (n = 7), the development of ASI through collective intelligence (n = 4), that ASI is not possible (n = 2), the development of general ASI from narrow ASI as nonspontaneous (n = 2), the use of silicon computer substrates for ASI (n = 2), and whole brain emulation (n = 2). The most common subtheme was whether ASI might be achieved through a singularity moment, an “intelligence explosion” (Chalmers, 2012: 1) of rapid, recursive, AI self-improvement. Of these papers, some analyzed the nature and coherence of such a hard takeoff to ASI (Chalmers, 2012; Goertzel, 2015; Hoffmann, 2023; Hutter, 2012; Prinz, 2012; Shanahan, 2012; Sotala, 2017) as well as its practical risks and benefits (Agar, 2016; Chalmers, 2012; Lorenc, 2015; Müller and Cannon, 2022; Sotala and Yampolskiy, 2015; Turchin et al., 2019; Yampolskiy, 2012). In addition, some scholars offer reasons to affirm the technological possibility (i.e. the possibility with sufficient technological progress) or likelihood of achieving ASI (Chalmers, 2012; Daley, 2021; Goertzel, 2015; Hoffmann, 2023; Hutter, 2012; Isac, 2010; Sotala, 2017) while others consider or defend reasons to deny it (Hoffmann, 2023; Nagl, 2022). In addition, many papers considered properties of how ASI might be achieved, ranging from the use of different computational substrates (e.g. silicon as opposed to biological; Goertzel, 2015; Primiero, 2017) to the modeling of individual human brains through whole-brain computer emulation (Lorenc, 2015; Mandelbaum, 2022) or through distributed forms of networked “collective intelligence” (Batin et al., 2017; Goertzel, 2015; Peeters et al., 2021; Turchin et al., 2019). Finally, some papers argue that ASI will not be achieved spontaneously from non-superintelligent forms of AI or without a progression through a specific set of stages or forms of intelligence (Jebari and Lundborg, 2021; Shanahan, 2012).
ASI: When?
The question of when ASI could or should occur was a theme in 11 papers in the set. We identified four subthemes or positions related to the question of when: that ASI is predicted to emerge within a given time frame (n = 3), that ASI development should proceed (n = 3), that it be delayed (n = 3), and that ASI development should not proceed (n = 2). The authors who advance predictions about when ASI might arrive offered analytic insights into the likelihood of conditions that are necessary or sufficient for creating ASI or for the likelihood of defeating conditions for creating ASI (Batin et al., 2017; Chalmers, 2012; Sotala, 2017). Some authors argue that ASI development should proceed due to ASI’s potential benefits (e.g. medical and scientific breakthroughs), ability to reduce other harms (e.g. terrorism), or the harms likely to be created by ineffective attempts to block development through regulations and enforcement (Agar, 2016; Bradley, 2020; Goertzel, 2015). Other authors present arguments for delaying ASI development until ethical or safety conditions can be satisfied (Haselager and Mecacci, 2020; Turchin et al., 2019; Yampolskiy, 2012). Some go further, offering reasons to stop AI development short of ASI (Sparrow, 2024; Yampolskiy, 2022).
ASI and philosophy of religion
Seven papers within the set addressed philosophy of religion in relation to ASI. Under this theme, five subthemes were found. The first is techno-religion (n = 3), where AI or other technological inventions may be utilized, experienced, or explained in a manner similar to religion. The other four subthemes relate ASI to prominent world religions—Christianity (n = 2), Buddhism (n = 1), Islam (n = 1), and Judaism (n = 1). One common topic in these papers is the notion of human distinctiveness and selfhood in the face of AI or ASI (Çelik, 2023; du Toit, 2019; Gruenwald, 2020). Other papers explored how humans may experience and understand ASI systems that are far “superior” to themselves (Schradle, 2020; Shanahan, 2012) and whether these understandings amount to “magical thinking” (Schradle, 2020). Du Toit (2019) considers whether traditional religion will be replaced by techno-religions. Relatedly, Szocik et al. (2020) ask whether ASI systems will behave akin to deities. Papers in the set also examine the theological implications of transhumanism (du Toit, 2019; Steinhart, 2008).
ASI: Politics and economics
The theme of politics or economics (excluding existential risks) related to ASI appeared in five papers in the set. Four of these papers were found to address both political and economic subthemes (Dodonova et al., 2023; Gill, 2016; McIntosh, 2010; Sotala and Yampolskiy, 2015), with one focused more directly on economic issues related to ASI (Brödner, 2018). Common topics in these papers included the problem of human redundancy in the age of ASI (Dodonova et al., 2023; Gill, 2016; Sotala and Yampolskiy, 2015), whether fully autonomous, “smart” production is plausible (Brödner, 2018), whether transhumanism and ASI render politics obsolete (McIntosh, 2010), and ASI’s implications for political power and human development (Dodonova et al., 2023; McIntosh, 2010).
Discussion
Taken together, our analyses provide a means by which to identify (1) central texts in the recent philosophical literature on ASI and (2) central themes and subthemes discussed in recent debates in philosophical articles on ASI. By identifying leading ASI texts and themes, we lay the groundwork to develop curricular frameworks that augment and extend current AI literacy efforts to better capture the philosophical dimensions of evaluating life with (or without) ASI. Based on these findings, here we identify three types of opportunities to develop deeper, more philosophical ASI literacy approaches: (1) content opportunities, (2) skill opportunities, and (3) pedagogical opportunities. These opportunities might enable, among other things, bottom-up teacher or community-driven curricular and instructional initiatives tailored to a given context; the development of policy or learning outcomes at school, district, or state levels; or K–12 and public philosophy efforts on ASI across the lifespan.
ASI literacy content opportunities
Based on the findings of our scoping review of philosophical ASI literature and analysis of AI literacy reviews, we see two opportunities to enhance ASI literacy curricular content: (a) expanding AI literacy curricula to include more ASI topics and (b) strengthening connections between ethical questions and epistemic and ontological questions in both narrow AI literacy and ASI literacy.
Expanding AI literacy curricula to include ASI
The first curricular content opportunity is related to ASI coverage in AI literacy research and practice. As noted above, currently, AI literacy reviews (e.g. Casal-Otero et al., 2023; Ng et al., 2021, 2024; Zhang et al., 2025) reflect no significant coverage of the unique issues related to the design, development, and impact assessment of ASI. Many pressing questions related to ASI—the ASI alignment problem, potential existential risks and benefits posed by ASI, models of how or when ASI could or should be created, and topics in the philosophy of mind as they relate to ASI systems, as well as questions about transhumanism, about the politics and economics of ASI, or about the religious significance of ASI—are revealed to be largely if not totally absent in leading AI literacy approaches. This absence suggests an opportunity to expand and extend the scope of current AI literacy content to include forward-looking analysis and deliberation about these topics. If the public is to engage in well-informed and inclusive democratic deliberation about ASI futures, more educational infrastructure in AI literacy dedicated to this task is urgently needed. Such awareness is arguably of benefit to students, whether they plan to work for, use the products of, oppose, or develop critical alternatives to the work of leading AI labs toward ASI.
Strengthening philosophical elements of AI literacy toward ASI literacy
The second ASI literacy curricular content opportunity we identify is strengthening the ontological, epistemological, and axiological dimensions of current narrower forms of AI literacy toward ASI literacy. By strengthening these interdisciplinary foci as additional ASI topics are added, educators, researchers, and policy makers could support a more continuous interdisciplinary pathway between current AI literacy topics and ASI literacy. Here, we identify two key areas where AI literacy and the philosophy of ASI converge and diverge, respectively:
A common emphasis on ethical topics;
A greater emphasis in ASI philosophical work on the interrelationship between ethics and epistemic and ontological questions in conceptualizing relationships with ASI.
First, as mentioned in the introduction, in reviews of AI literacy literature (Casal-Otero et al., 2023; Ng et al., 2021, 2024; Zhang et al., 2025), limited attention is paid to philosophical issues related to the conception and evaluation of AI. Where philosophical inquiry is foregrounded in AI literacy work, it is most often in the analysis of ethical issues and societal impact of AI (e.g. Ali et al., 2021; Grosz et al., 2019; Williams et al., 2023; Zhang et al., 2023). Our data of philosophical journal articles about ASI somewhat mirrored AI literacy’s emphasis on questions related to ethics and politics. In the philosophical literature, the majority of papers (89%, n = 58) had some ethical or axiological themes. Moreover, the top three most frequent themes—the problem of ASI value alignment, existential risks, and machine ethics—are all explicitly ethical (and/or political), even though they also have epistemic and ontological dimensions. Axiological concerns also permeate many aspects of the other themes, including those about the nature of intelligence, transhumanism, when and how ASI might be achieved, and its relationship to religion, politics, and economics.
A treatment of common narrow AI literacy topics such as algorithmic fairness, accountability, transparency, and stakeholder impact that considers arguments for and against different principles (and specifications of principles or concepts) could be extended, for example, to consider questions of ASI value alignment, existential risk, and machine ethics. Each topic in this sample sequence depends on a specification of the values, populations of concern, and justifications that define AI working inappropriately or in alignment and the potential risks of losing alignment for different stakeholders against potential benefits. This approach has the affordance of developing a spectrum of issues and debates to define and evaluate, from current AI toward more powerful future forms of AI, such as AGI, culminating in ASI as an endpoint. Such a future-oriented approach enables a conception of AI literacy that is not primarily reacting to current AI and its impacts but is also using current issues to define the sorts of technological futures members of the public want to inhabit or resist after democratic reflection.
Extending from this common axiological emphasis also affords opportunities to deepen connections to other domains of philosophy. In our data, philosophers were often more explicit in connecting an analysis of what exists (ontology) and how we know (epistemology) as central to determining what we should do in relation to ASI. In addition to often discussing questions of ethical or political value, 60% (n = 39) of philosophical papers about ASI also included a thematic focus on questions of ontology, while 42% (n = 27) included themes from epistemology. For example, in determining how we should assess the impacts of powerful AI systems, philosophers were likely to emphasize the dependence of such questions on others about how we should understand the nature of ASI’s perception, reasoning, intelligence, action, or consciousness (or lack thereof) in relation to that of human agents (see, e.g. Bostrom, 2012; Davis, 2015; Dung, 2024; Yampolskiy, 2022). These ontological questions concerning how such properties are to be defined also give rise to epistemological questions regarding how knowledge of those properties may be obtained.
In AI literacy reviews, only one source (a 2005 concept map and curriculum developed by Ellis et al.) is mentioned that explicitly focuses at length on questions of epistemology and ontology, in this case, on the philosophy of mind. Ellis et al.’s (2005) work supports the possibility of bringing together topics in AI robotics with questions about the nature of minds and our ability to know when they are present. These questions about ontology and epistemology raised by Ellis et al. in narrow AI interactions could arguably be extended to address related questions in the ASI philosophical literature. Determining when a mind is present or survives various forms of change, for example, is closely related to ontological and epistemic questions about the possibility of human augmentation, enhancement, or transcendence and what these possibilities entail for the identity or survival of the person who undergoes these changes. Extending from issues of augmentation of human capacities by current technology to such transhumanist questions, which comprise a significant domain of ASI philosophical literature, is another opportunity educators might pursue to deepen ASI literacy (see, e.g. Coseru, 2024; Steinhart, 2008; Torrance, 2012). Finally, reflecting on these questions in the philosophy of mind affords opportunities to extend concerns about current human–AI relationships and attachments to determine how increasingly powerful forms of AI may shape, enable, undermine, or distort human flourishing.
ASI literacy skill opportunities
In addition to differences in content, the philosophical literature on ASI also differed in the process used to address conceptual issues. The ASI philosophical literature foregrounds the giving and receiving of reasons, in deductively valid forms, for a set of logically defined positions. For example, philosophers argue both for (e.g. Bostrom, 2012) and against (e.g. Davis, 2015) the thesis that ASI is unlikely to be aligned due to the instrumental convergence and orthogonality theses. This sort of structured argumentation was less common in the AI literacy reviews of ethical topics, where principles were often treated as given or subjective (e.g. AI4K12, 2022; Akgun and Greenhow, 2022; Kaspersen et al., 2022; Ng et al., 2024; Payne, 2019; Touretzky et al., 2019). Moreover, the most cited texts in our bibliometric analyses of philosophical works about ASI—both within the study set and the foundational set—were nearly entirely uncited in these AI literacy reviews. Notably, many of these texts about ASI are authored by computer scientists (e.g. Good, 1965; Moravec, 1988; Russell, 2019; Yampolskiy, 2012), suggesting that this siloing is not necessary.
Computer science education research and practice often emphasize the value of computational thinking, which draws on logical reasoning, pattern recognition, and abstraction to solve theoretical and practical problems using computers (which may be human or machines) across subject domains (Hsu et al., 2018; Wing, 2008). Philosophers arguably use similar logical reasoning skills and analytic processes to develop and test philosophical claims about ASI risk and opportunity. Insofar as this is true, there is an opportunity to expand the scope of the computational thinking skills already valued in computer science education to more robustly engage with arguments about the ontological, epistemological, and axiological dimensions of ASI (and AI) literacy necessary for assessing the social impact of these and other technologies.
Since philosophers in the study articles disagree on a majority of the most central issues involving ASI, we believe it would distort the epistemic landscape of expertise to require or encourage students and other members of the public to simply defer to any of the controversial positions advocated by experts. A more empowering and appropriate response to these controversies, one that reduces risks of indoctrination (Cf. Hand, 2025; Taylor, 2017), is to leverage existing expert philosophical arguments or simplified versions of them as resources for students to consider and elaborate on from their own perspective to reach their own informed conclusions. Even if the academic philosophical literature does not yield consensus on single final answers to most ASI questions, this more open-ended process of inquiry informed by the philosophical literature may enable the public and philosophers alike to rule out some especially problematic ideas or positions, raising the floor of reasonable consensus even as pluralism in ultimate views persists. Such an approach highlights the value of pedagogies that are common in precollege (e.g. philosophy for children) and public philosophy as means to enable deeper democratic inquiry into the first principles of AI through ASI literacy.
Extending instructional strategies in AI literacy to ASI
Finally, by contrast with the philosophical literature on ASI, reviews of AI literacy research and frameworks offer much more concrete guidance for teaching and learning about the technical, applied, and social dimensions of AI. The Embedded EthiCS approach employed by some AI literacy scholars and educators (e.g. Ali et al., 2021; Grosz et al., 2019), for example, offers a model for linking technical decisions directly to questions of evaluating social and ethical impact. Such an approach may be helpfully extended to better address other first principles topics related to the conception, design, and evaluation of ASI futures. If part of the goal of AI literacy is to develop a public capable of flourishing in relation to both present and future AI (Yim and Su, 2025), then expanding the use of effective pedagogies to examine arguments for and against various possible designs of ASI and their implications is one promising way to enhance students’ ability to shape AI futures.
With that said, a potential challenge for teaching about the future of AI is that many of the technologies we have reason to deliberate about are not yet built. To address this limitation, pedagogies and curricular media that allow for the consideration of fictional but realistic scenarios or simulations may be a useful bridge to considering AI futures. Work in futures literacies, which aims to facilitate learning about “possible, probable, and preferable futures” (Horst and Gladwin, 2022: 52), may afford resources to support instructional designers and policy makers aiming to develop ASI literacy that warrants further exploration. As noted above, another opportunity to build off existing pedagogical strengths is to extrapolate from an analysis of the conceptualization and impacts of narrower forms of AI to their implications for ASI. Since such extrapolations (and analysis of discontinuities) are a key part of the process of developing new technologies and democratic deliberation more generally, such an approach extends engineering and democratic skills that may also have broader independent value.
Limitations and future research
We have presented the first scoping review and analysis of the philosophical literature on ASI in relation to AI literacy reviews, with the aim of facilitating a greater integration of these two domains of scholarship. Here, we identify limitations of the study to guide future research on ASI literacy. As noted above, our study only includes English-language sources and a specific search string. Future studies might build on the work here to include works in other languages and additional search strings using key thematic terms identified in this study, additional databases, and source types to expand the scope of the thematic analysis offered here. Since PhilPapers requires that philosophical work in journals not indexed in philosophy be manually added, some relevant philosophical work may not be reflected in this review. As noted above, future work might also examine conceptual and theoretical work on ASI in other fields in relation to the present review that may not be indexed in PhilPapers.
In addition, we do not survey articles on issues related to the distinct concept of AGI. AGI occupies a spot on a continuum between current AI and ASI, which may add key context to futures that include or resist ASI technologies. 6 We offer the current review as a part of a broader conversation on ASI and make no claims to exhaust the terrain. We hope that others will further extend and improve on this work to fill in the picture of this continuum. Finally, the thematic analysis offered here relies on interpretive methods that admit of reasonable pluralism, so while we aim to provide plausible, reliable interpretations of the data, we do not claim perfect universality across all possible interpreters of these works. Nor do we claim exhaustive depth in our analysis of the themes advanced. Whereas this scoping review provides a high-level overview of thematic topics in the field, more detailed analyses of arguments related to each theme and subtheme presented here could provide beneficial insight for educators, curriculum designers, and policy makers. Such future work might employ a “review of reasons” (Strech and Sofaer, 2012) that analyzes philosophical arguments for or against a well-formed question at a more granular level to guide thought or action. For example, a review of reasons might examine philosophical arguments for and against the claim that ASI’s instrumental reasoning will very likely yield bad outcomes for humans. The results of such studies could provide curricular foundations for ASI literacy that address normative ethical, epistemic, and ontological topics in depth. Similarly, reviews of social scientific literature in psychology, sociology, economics, law, and other related fields could be integrated into the review advanced here to deepen deliberation about ASI futures.
Footnotes
Appendix 1
Theme and subtheme definitions.
| Theme | Subthemes | Definition |
|---|---|---|
| alignment | aligning ASI to the correct values or human goals | |
| bias in alignment | bias in AI (e.g. racial, gender, disability, or other forms) | |
| coherent extrapolated volition | coherent extrapolated volition as introduced by Yudkowsky (2004). | |
| ASI confinement | confining ASI to reduce risk | |
| ethical value in alignment | alignment with ethical values | |
| instrumental reason | role of instrumental reason in AI alignment | |
| nonexistential alignment risk | nonexistential AI alignment risk | |
| orthogonality thesis | Bostrom’s (2012) orthogonality thesis: “intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary” (p. 71). | |
| political value in alignment | alignment with political values | |
| existential risk | risks that threaten human or ecological existence, such as death or a life not worth living. A subtheme of normative misalignment. | |
| assessing existential risk | assessing existential risk of ASI | |
| risk/probability of doom | probable or certain bad outcomes from ASI | |
| King Midas problem | existential risks of losing alignment due to unintended consequences (e.g. paperclip maximizer) | |
| managing existential risk | existential risk management in ASI | |
| not an existential risk | not an existential risk | |
| political existential risk | political existential risks. | |
| existential risks of war/terrorism | existential risks related to war or terrorism | |
| machine ethics | creation of ethical machines; deals with moral agency or patiency of machines | |
| ethical value in an AI agency | ethical value and machine intelligence or agency | |
| moral status/patiency of AI | moral status or moral patiency of AI systems | |
| moral agency in AI | moral agency in AI machines | |
| moral motivation of AI systems | moral motivation of AI systems | |
| transhumanism | issues related to transcending the human condition | |
| augmentation | deals with humanity moving beyond current biology by augmentation, cyborg life forms, BCIs, etc. | |
| enhancement | deals with bioengineering or the enhancement of humans | |
| immortality | immortality in transhumanism | |
| personal identity | personal identity in transhumanism (e.g. surviving mind upload). | |
| prohumanism | favors humanism or rejects transhumanism on humanistic grounds. | |
| simulation hypothesis | hypothesis that we currently live in a computer simulation | |
| uploading consciousness | uploading consciousness to computers | |
| philosophy of mind | metaphysics of mind (at least some machine ethics questions depend on philosophy of mind questions). | |
| computationalism | mind is a computational system or computational theory of mind | |
| biological theory of consciousness | biologically based theory of consciousness | |
| hard problem of consciousness | deals with the hard problem of consciousness, the problem of providing an intuitive explanation of how phenomenal consciousness emerges from the brain | |
| intelligence | deals with the nature of or knowledge of intelligence | |
| other minds | problem of knowledge of other minds | |
| explainable AI (XAI) | deals with explainable AI or ASI/issues of inscrutability | |
| ASI: how? | how (if possible) should or could we create ASI? | |
| AI to ASI: not spontaneous | general ASI is not spontaneously possible from narrow AI or requires a stage progression | |
| collective intelligence | a globally distributed approach to the development of ASI | |
| computer substrate | silicon/traditional computer substrates (as opposed to organic brain). | |
| ASI is not possible | ASI is technologically impossible | |
| ASI is possible | ASI is technologically possible | |
| whole brain emulation | whole brain emulation as a path to ASI | |
| ASI: when? | when should or could ASI occur? | |
| delay | we should delay proceeding toward ASI | |
| never | we should or will never build ASI | |
| prediction | predictions on when ASI will occur. | |
| proceed | we should proceed in trying to build ASI | |
| philosophy of religion | ASI and philosophical aspects of religion | |
| Buddhism | ASI and Buddhism | |
| Christianity | ASI and Christianity | |
| Islam | ASI and Islam | |
| Judaism | ASI and Judaism | |
| techno-religion | ASI and new techno-religion(s) | |
| ASI politics and economics | political or economic impacts or analysis other than existential risk. | |
| politics | political impacts of ASI (other than existential risk) | |
| economics | economic impacts of ASI |
Appendix 2
Foundational sources referenced most frequently within the study dataset.
| Source | Citations within the data |
|---|---|
| Bostrom N (2014) Superintelligence: Paths, Dangers, Strategies. Oxford University Press. | 40 |
| Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk. In: Bostrom N and Ćirković MM (eds) Global Catastrophic Risks. Oxford University Press, pp. 308–345. | 19 |
| Kurzweil R (2005) The Singularity Is Near: When Humans Transcend Biology. Penguin. | 18 |
| Chalmers D (2010) The singularity: A philosophical analysis. Journal of Consciousness Studies 17(9–10): 7–65. | 16 |
| Bostrom N (2002) Existential risks: Analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology 9(1). | 10 |
| Good IJ (1965) Speculations concerning the first ultraintelligent machine. Advances in Computers 6: 31–88. | 10 |
| Moravec H (1988) Mind Children: The Future of Robot and Human Intelligence. Harvard University Press. | 10 |
| Russell S (2019) Human Compatible: AI and the Problem of Control. Allen Lane. | 10 |
| Vinge V (1993) The coming technological singularity. In: Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace (NASA Conference Publication No. 10129). NASA Lewis Research Center, pp. 11–22. | 10 |
| Omohundro SM (2008) The basic AI drives. Proceedings of the First AGI Conference 171: 483–492. | 9 |
| Bostrom N and Yudkowsky E (2014) The ethics of artificial intelligence. In: Frankish K and Ramsey WM (eds) The Cambridge Handbook of Artificial Intelligence. Cambridge University Press, pp. 316–334. | 8 |
| Tegmark M (2017) Life 3.0: Being Human in the Age of Artificial Intelligence. Alfred A. Knopf. | 8 |
| Yudkowsky E (2004) Coherent extrapolated volition. Machine Intelligence Research Institute. Available at: https://intelligence.org/files/CEV.pdf | 8 |
| Kurzweil R (1999) The Age of Spiritual Machines: When Computers Exceed Human Intelligence. Viking. | 7 |
| Searle, J (1984) Minds, Brains and Science. Harvard University Press. | 7 |
| Sotala K and Yampolskiy RV (2015) Responses to catastrophic AGI risk: A survey. Physica Scripta 90(1): 018001. | 7 |
| Turing AM (1950) Computing machinery and intelligence. Mind 59(236): 433–460. | 7 |
| Yudkowsky E (2001) Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Machine Intelligence Research Institute. | 7 |
| Bostrom N (2003) Ethical issues in advanced artificial intelligence. In: Spencer G (ed.) Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence (vol. 2). International Institute of Advanced Studies in Systems Research and Cybernetics, pp. 12–17. | 6 |
| Bostrom N and Ćirković MM (eds) (2008) Global Catastrophic Risks. Oxford University Press. | 6 |
| Goertzel B and Pitt J (2012) Nine ways to bias open-source AGI toward friendliness. Journal of Evolution and Technology 22(1): 116–131. | 6 |
| Gunkel DJ (2012) The Machine Question: Critical Perspectives on AI, Robots, and Ethics. MIT Press. | 6 |
| Hanson R (2016) The Age of Em: Work, Love, and Life When Robots Rule the Earth. Oxford University Press. | 6 |
| Sandberg A and Bostrom N (2008) Whole Brain Emulation: A Technical Report. Report no. 2008–3. Future of Humanity Institute, University of Oxford. | 6 |
| Chalmers DJ (1996) The Conscious Mind: In Search of a Fundamental Theory. Oxford University Press. | 5 |
| de Garis H (2005) The Artilect War: Cosmists Vs. Terrans. ETC Publications. | 5 |
| Eden A, Søraker J, Moor JH, et al. (eds) (2012) Singularity Hypotheses: A Scientific and Philosophical Assessment. Springer. | 5 |
| Legg S and Hutter M (2007) A collection of definitions of intelligence. In: Goertzel B and Wang P (eds) Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (Frontiers in Artificial Intelligence and Applications, vol. 157). IOS Press, pp. 17–24. | 5 |
| Moravec H (1999) Robot: Mere Machine to Transcendent Mind. Oxford University Press. | 5 |
| Ord T (2020) The Precipice: Existential Risk and the Future of Humanity. Hachette Book Group. | 5 |
| Yampolskiy RV (2012) Leakproofing the singularity: Artificial intelligence confinement problem. Journal of Consciousness Studies 19(1–2): 194–214. | 5 |
Acknowledgements
The authors would like to thank two anonymous reviewers for their helpful comments and feedback on this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: University of Illinois Campus Research Board Research Support Award (RB24136).
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.