Artificial intelligence and medical education: A global mixed-methods study of medical students’ perspectives

Design

We employed a mixed-methods approach for this study by using an online questionnaire and in-person focus groups. The questionnaire provided quantitative data, which was used to ascertain students’ level of knowledge and understanding of AI and the key aspects of AI-related content that should be taught at medical schools. The focus groups provided an opportunity to take advantage of the diverse background and experiences of medical students from different countries and to generate ideas toward the most effective and engaging way to teach AI-related content. Free text data from the questionnaires along with the verbatim focus group data was used to add depth and new perspectives to the quantitative aspects of the questionnaire. ²⁵

Participants were students of the national member organisations (NMOs) of the International Federation of Medical Students’ Associations (IFMSA). The study received ethical approval under the Low-Risk designation from the King's College London Research Ethics Committee, Review Reference: MRSU-19/20-17484.

Data collection

The online questionnaire was sent to 140 NMOs through the IFMSA email server a month prior to the 69th IFMSA General Assembly (GA) March 2020 in Kigali, Rwanda. To be eligible for recruitment, individuals had to be a medical student and a member of one of the IFMSA NMOs. They did not need to be a delegate to the March GA. A total of 128 individuals filled out the online questionnaire. Of these, 90 responses were gathered from delegates at the GA recruited through convenience sampling and the remaining 38 were gathered before or after the GA via email.

The questions were adapted from a questionnaire used in a previous study. ²¹ It was then reviewed by a team of medical students (HM, HE), qualitative research experts (BW, AG), an AI expert (TV) and a clinician-scientist with expertise in AI research (JS) to ensure internal validity. A copy of the online questionnaire is attached as Supplemental Digital Appendix 1. The questionnaire consisted of the following four sections:

(1) Demographics

Of the 128 participants who filled the questionnaire, 90 attended the in-person focus groups conducted during the GA. Participants were selected for the focus groups through convenience sampling. Those who hadn't already filled out the online questionnaire did so before joining the focus group. The average size of each focus group was 10–15 participants and each was diverse in terms of ethnicity, country of origin, gender, and country of study. Each focus group was led by a trained IFMSA facilitator who obtained written consent from all participants before starting the session. At the beginning of the focus group discussions, we defined the term ‘artificial intelligence’ and we provided examples of uses of AI in fields outside of medicine in order to establish a rudimentary baseline of understanding among participants. The sessions lasted around 90 min each and were divided into discussions around the following two questions:

What are the most important aspects of AI to learn about?: Individuals from different countries were paired and given the instruction: ‘Speaking to the person in your pair, come up with five AI-related topics that you consider most important and relevant for medical students to learn’. Once this discussion was completed, pairs joined to form groups of four. These groups discussed their individual lists to come up with a combined set of five topics and ranked them in order of importance. After discussion, the whole group came together to present ideas and discuss salient findings. Data collectors noted the top five topics put forward by each group of four.

Data analysis

Following data collection, data was downloaded from Google Forms to an Excel spreadsheet. Quantitative data from the questionnaire was imported and analysed in R. ²⁷ The countries of study of the participants were grouped by income status using the World Bank list into LMICs or HICs. ²⁸ Data visualisation was performed using GraphPad Prism (GraphPad Software, San Diego, California).

All free-text data was copied verbatim into the cells of a spreadsheet. This included free-text data from each section of the questionnaire and data from the open focus group discussion. This free text data was coded by three researchers independently using an adapted version of the framework method. ²⁹ Each coder went through the data set to come up with their individual set of codes. All coders then discussed the codes and agreed on a set of consensus codes and definitions that were applied to the dataset. Once coding was completed, similar codes were organised under sub-themes and themes. Two iterations through the data were performed: the first was to come up with codes individually and the second was to apply a set of consensus codes.

What do you know about AI in medicine?

The initial questions from the survey focused on assessing the familiarity and understanding of AI among participants. 93 (73%) survey respondents had routine access to electronic patient records (EPR), while 86 (67%) agreed that they were confident in using EPR – indicating a baseline access to and understanding of core healthcare information technologies among the majority. A small minority of students (15.6%) reported coding experience. 93 (73%) respondents had a moderate or strong understanding of the concept of AI, however, only a single student (0.7%) reported significant teaching on AI in medical school. Students believed that the areas of medicine most affected by the introduction of AI would be EPR, radiology and medical administration (Figure 1).

OPEN IN VIEWER

Figure 1.

A bar plot of frequencies of students' answers to the question: ‘Which five of the following areas of medicine do you think could be most affected by the introduction of artificial intelligence?’. ‘Health systems management’ explicitly included hospital organisation and scheduling while the label ‘Administration’ included note-taking and secretaries.

The remaining questions of the first section of the survey drew on participants’ opinions and expectations of AI. 120 (94%) respondents agreed with the idea that AI will improve medicine in general, and most students (89.8%) believed that they would be using AI in their work within 10 years (Figure 2). A large proportion (44%) of students felt that AI definitely has a role to play in healthcare in LMICs; 13 (10%) believed that AI definitely does not have a place. Subgroup analysis of the results showed that respondents from HICs were two-and-a-half times more likely to believe that AI is applicable in LMICs compared to respondents from LMICs, indicating a gap in understanding and expectations from students not directly embedded in LMIC healthcare systems. However, 55 (43%) participants reported feeling worried about the impact of AI, and 40 (31%) respondents thought AI will replace physicians and non-interventional doctors in the future.

OPEN IN VIEWER

Figure 2.

A bar plot of frequencies of students' answers to the question: ‘How likely is it that you will be using AI in your clinical practice in the following time frames?’

Qualitative analysis of the free-text responses from the survey and the focus-groups outlined three major sub-themes: applications of AI in clinical medicine, scepticism around AI and applications in LMICs. Applications in clinical medicine include knowledge of the use of AI in general medicine, surgery and diagnosis among others. Student opinion was generally in favour of the integration of AI in clinical medicine, as mentioned by a student from Canada, as it could ‘allow for improvements in health services and their efficiency, freeing up both human and physical resources’. Moreover, students were keen to learn more about AI in their routine medical teaching, particularly its limitations and benefits.

Several students raised ethical concerns and exercised cautious optimism, including a student from Germany who said AI should be ‘mostly used as help … rather than as an executor’ and a student from the UK said that it should be ‘applied to complement and streamline the work of healthcare professionals, but not to replace them’. Students mentioned the potential of AI to improve access to care in underserved communities in LMICs and to improve the quality of care in remote and rural communities. According to a student from Pakistan, ‘it will provide the doctor with the efficient assistance and will save time and money’ in underserved settings. However, a student from Morocco was more sceptical, and suggested that ‘low- and middle-income countries should have access to [foundational] technology first … Some regions still don't have access to the internet’.

What do you wish to learn about AI in medicine?

Based on the survey responses, the majority of participants (95%) were interested in learning the basic principles of AI, while 61% said they wanted to learn to code. Going further, 103 (80%) agreed that there should be opportunities to get involved in algorithm development and many (91%) felt that doctors should be involved in the integration of AI in clinical practice.

Thematic analysis revealed that students wanted to learn about general concepts and computer science-related theory as part of AI teaching. Salient quotes grouped by sub-themes are presented in Table 1. General concepts incorporate basic principles of how AI works, ethics around the integration of AI in medicine and the skills needed to critically appraise the use of AI. A student from Australia highlighted the need to ‘approach AI as we approach other interventions like drugs or radiology’ and to learn ‘enough about basic principles that we can begin to evaluate for ourselves whether that particular intervention or AI program, is the right choice for our patient’. Another student from Australia raised the concern of AI causing ‘discrimination or unintentional worsening of health outcomes for vulnerable populations’ due to the impact of data bias in AI. They stressed that ‘we should be taught a healthy scepticism for AI … All AI is not created equal, and garbage in = garbage out … we need to be given the knowledge and the tools to make this assessment’. There was a tone of cautious optimism exercised by many students, who expressed that they wanted to learn about the ‘limits and opportunities AI provides to doctors and how they can use AI to do good, and not harm, as a tool’.

OPEN IN VIEWER

Table 1.

Sub-themes and exemplar quotes generated from analysing qualitative responses to focus group questions. We asked the following: ‘How will AI be applied to medicine in the future? & What would you like to learn about AI that might prepare you for a future in which AI plays an important part in medicine?’ We gathered data from 128 students through online surveys and focus groups carried out at the 69th IFMSA General Assembly in Rwanda, March 2020.

Sub-themes	Exemplar quotes
General Concepts	‘We should be taught a healthy skepticism for AI … All AI is not created equal, and garbage in = garbage out … we need to be given the knowledge and the tools to make this assessment.’ – Student from Australia ‘The basis of how AI operates, risks and sources of errors in AI … what can AI be used for, what are the limits? What hardware is needed?’ – Student from Germany ‘What limits and opportunities AI gives us … How can we use AI to do good, and not harm, as a tool.’ – Student from Norway
Applications in clinical medicine	‘monitoring ICU patient stats … predicting patient outcomes … use of machine learning in order to easily diagnose a patient through imaging … surgical planning and intra-operative guidance.’ – Student from Hong Kong ‘within Primary health care… triaging who needs what level of care, follow up on NCDs and other chronic illnesses … online consultations’ – Student from Sweden ‘detection of diseases at stages where they were previously undetected … to predict … arthritis in the next five years by looking at … MRIs’ – Student from Pakistan it has the potential to recruit massive populations for studies … shown in Apple Heart Study’ – Student from Ireland ‘how it can be used with precision’ – Student from Ethiopia
AI in LMIC	‘low- and middle-income countries should have access to technology first … Some regions still don't have access to internet’ – Student from Morocco ‘reach out to the majority of the community’ – Student from Ethiopia ‘I also think that AI will be used to increase healthcare access/reach in underserved populations …’ – Student from Australia
Patient Safety	‘reduce medical error … identifying … skin lesions etc.’ – Student from Ireland
Computer science	‘how to code … build AI health related algorithms’ – Student from Burkina Faso ‘how the algorithms are developed and trained to understand … the results, and also how to mitigate … bias.’ – Student from the UK ‘the methods used to develop AI algorithms … [to] critically think about the outputs.’ – Student from the United States ‘how the system integrates and works in the medical system’ – Student from India
Scepticism	‘It could be used for general diagnosis … but a health care worker (physician) will still always be required to assess the border line cases ….’ – Student from Iceland ‘to ensure that they do not unintentionally cause discrimination or worsening of health outcomes for vulnerable populations … There is great potential for good but also great potential for harm.’ – Student from Australia
Big Data	‘How to use technology to stock information, to organize data and to analyse it.’ – Student from Morocco

Many students wished to learn the basic computer science theory needed to understand AI and some students requested coding classes. A student from Burkina Faso stated they would like to learn ‘how to code …[and] build AI health related algorithms’ while a student from Latvia felt that ‘coding classes should be implemented in the curriculum’. A student from Bangladesh also supported ‘an introduction to coding and a holistic approach to studying AI’ as part of their core medical curricula. A link was drawn by many students between the need to have a basic computer science understanding and applying this to appraise algorithms for clinical translation. Further, a number of participants stressed the need to learn about AI through use in clinical practice, including a student from Japan who felt that students would be more likely to be interested in AI if ‘hospitals introduce AI tech and show its benefit’.

Data from the focus groups showed a ranked order of topics of interest. The most popular topics, in order, were: (1) Applications of AI in clinical medicine, (2) AI 101 including principles and basic coding, (3) AI ethics, (4) AI and the impact on careers in medicine and (5) Algorithm development for clinicians.

How do you wish to learn about AI in medicine?

The majority (92%) of students who participated in the study stressed that AI-related teaching needs to be incorporated into the core medical curriculum. When asked how they would like AI-related teaching to be incorporated into the curriculum, many students (86%) were interested in exploring interdisciplinary learning with fields including computer science and biomedical engineering.

Qualitative analysis revealed two pertinent themes: core curriculum and teaching delivery. Salient quotes grouped by their sub-themes are presented in Table 2. Most participants agreed on the need to incorporate AI-related teaching into the core curriculum. However, opinions regarding which part of the core curriculum the teaching should be added varied between students. A student from Singapore suggested ‘including more applications of AI during clinical training’ as an integrated course, whereas a student from Russia suggested the need for a ‘specific class on technology and AI in medicine’ during pre-clinical studies. Students from the United Kingdom drew on their experiences with module-based teaching and suggested that ‘each module should have a section about how AI is being and could be used in this field. How to evaluate the effectiveness of AI in a clinical setting and monitor its results’.

OPEN IN VIEWER

Table 2.

Sub-themes and exemplar quotes generated from analysing qualitative responses to the question: ‘How can the medical curriculum in your country be changed to address the learning needs you have described in this questionnaire?’ We gathered data from 128 students through online surveys and focus groups carried out at the 69th IFMSA General Assembly in Rwanda, March 2020.

Sub-themes	Exemplar quotes
Core Curriculum	‘basic computer science class … [in] preclinical studies … a specific class on technology and AI in medicine.’ – Student from Russia ‘how AI is being and could be used in this field’ – Student from United Kingdom ‘Providing an optional curriculum about AI that supports any sort of information around the subject.’ – Student from United Kingdom
Teaching Delivery	‘implementing workshops in the university.’ – Student from Belgium ‘incorporate actual computer scientists in a multidisciplinary team’ – Student from United Kingdom ‘collaboration between medical students, biomedical engineers, computer scientists and data scientists (similar to hackathon-like projects)’. – Student from Hong Kong
Learning through use	‘it will never change unless most of hospital[s] introduce AI tech and show its benefit …’ – Student from Japan ‘Include more technology in teaching such as lectures and practical sessions so that students can familiarize themselves with the available technology. Hospitals and faculties should adopt AI and technologies as well.’ – Student from Malaysia

When speaking about teaching delivery, participants stressed on the need for multi-disciplinary learning and hackathon-style workshops. A student from Hong Kong wanted ‘collaboration between medical students, biomedical engineers, computer scientists and data scientists (similar to hackathon-like projects)’ while a student from the United Kingdom supported this idea with the suggestion to ‘incorporate computer scientists in a multidisciplinary team’. A number of students, including one from Belgium, wanted the content to be taught by ‘implementing [AI] workshops in University’.

Why do students need to know about AI

In the near future, AI is likely to disrupt key areas of medical practice, including psychiatry and diagnostic specialties such as radiology and pathology. ³³ The Topol review in the UK recommended that clinicians should be trained specifically to work with AI in order to ensure improved clinical care. ³⁴ Previous studies have shown positive results in local and national surveys regarding the knowledge and perspectives of medical students and trainee doctors for an AI-driven future.^21,22 These studies indicated that, in high-income settings, students were keen to learn more about AI but received minimal specific training on this as part of their medical curriculum. Our study supports the findings of these earlier works: medical students view AI with cautious optimism; they are generally excited for the change that it will bring and they wish to learn more about it. Moreover, our study offers further unique insight into the global distribution of AI-related teaching, experience and interest. Students in HICs and LMICs displayed a broad knowledge of and interest in AI, including its applications in clinical practice as well as the knowledge required to develop algorithms. This is remarkable considering the limited examples of AI-based technologies that have been successfully implemented in clinical practice along with extensive validation and with evidence of improved health outcomes for patients. These findings provide a guide to medical educators willing to address the growing digital renaissance in medicine. Educational institutions can be catalysts for a digital future that is patient-centred if the AI-learning needs of medical students are met.

AI for mitigating health inequalities

AI offers a unique opportunity to mitigate health inequalities around the globe.^34-37 AI-based systems have shown promising potential in mitigating the impact of challenges that are specific to LMICs, including high neonatal and maternal morbidity and the spread of Dengue fever.^38–41 More needs to be done to expand on this success in LMICs specifically. We found extensive interest in AI among students in LMICs but a severe lack of teaching on the topic, and a lack of access to the foundational technology that AI requires. Our data also showed a large gap in expectation between students from HICs and those from LMICs regarding the applicability of AI-based tools to health systems in LMICs given the current infrastructure. Until now, the implementation of AI has been largely focused on high-income settings and socioeconomic groups with high digital literacy.^36,37 If this trend continues there is the potential for AI to compound the already prevalent issue of health inequality. Further, where AI-based tools are developed for low-resource settings they may not be applicable or appropriate given a HIC-centred approach to development and implementation.^15,38 Our data supports the notion that teaching around AI needs to be provided to those who are embedded within the systems that will use it, thereby ensuring the local applicability of these tools. According to our data, a good place to start is for healthcare providers in LMICs to be provided the foundational technology to utilise AI in the future. Next, they should be offered the education and resources to bolster AI literacy and to support pioneering advances in the implementation of AI-based tools in their healthcare systems.

Delivering a new medical curriculum

The need for a multidisciplinary approach towards teaching AI-related content was emphasised by most respondents. Medical schools and other teaching institutions should work together to facilitate these changes to their curricula. A number of institutions are already offering teaching on AI to medical students. Carle Illinois College of Medicine offers a combined medicine and mathematics/data science course with the intention of fostering clinical innovators. ³⁹ In our study, participants suggested that a major area for change is in content delivery, where AI-related teaching could be given through interprofessional learning between medical and computer science students. Some centres have adopted this approach and one is offering a combined computer science and medicine course in a collaboration between a large hospital and two academic institutions. ²⁴ Many students suggested hackathon-style or workshop projects, where students collaborate to solve clinical problems, similar to those offered at Harvard Medical School. ⁴⁰ This approach would promote creativity as well as improve healthcare and computer science literacy. In terms of organisation, participants suggested that the AI curriculum could be delivered as a separate module, or integrated into current modules with teaching on AI as applied to the relevant specialty. For students who might wish to delve deeper into the topic, undergraduate medical courses could offer an optional, one-year intercalated degree in AI-based healthcare. In postgraduate medical systems, schools should incentivise applicants with a data and computer science background to enter medicine. All of these approaches would address the pressing need for interdisciplinary collaboration and teaching within medical schools.

Strengths and limitations

The study was designed around data collected at the IFMSA General Assembly due to the unique access to a diverse range of medical students in terms of nationality, socioeconomic status, ethnicity, career stage and clinical and research experience. We used a mixed-methods design for the study as the survey data provided valuable quantitative information on students’ current knowledge and understanding of AI while the focus groups offered an opportunity for students to generate practical ideas about changes to the medical curriculum. Our relatively unbiased sampling is a notable strength of the study – the conference where data was collected was unrelated to AI and technology, and the convenience sampling approach that we took ensured that our participant selection was independent of students’ interest in AI. Offering the survey prior to the focus group provided participants the opportunity to collect their thoughts before the focus group discussions and to maximise the value of these discussions.

We note important limitations with this study that may affect the generalisability of our findings. A key principle in our study was gaining a broad sample of medical students in order to guide educators around the world. In doing so, our sample included a small number of participants from each nation, thereby reducing our ability to reliably associate our findings with particular regions. The geographic distribution of the knowledge, perspectives and wishes of medical students regarding AI in medicine could be an interesting point for future study in order to cater educational interventions to particular regions. We also note the limitations of self-reported data due to its subjective nature. For instance, when ascertaining the understanding of AI among participants, the level of knowledge that one individual defined as a 'poor understanding’ may have been an excellent understanding of AI for another. It would be interesting to explore a more objective understanding among students in the future using questions that test knowledge around AI. The entire study was conducted in the English language in line with the language of the conference. This may have biased study data as participants with a poorer command of English may have been less likely to share their perspectives in group discussion and/or may have articulated these to a lesser extent when answering the questionnaire. While this limitation is difficult to overcome with such a diverse cohort, it could have been mitigated through an offering of (live) multi-lingual translations. Another important point to consider was our impact, as focus group facilitators, on the group discussion. We were mindful of this and hence refrained from expressing our own thoughts on the topic. Instead, our contributions were limited to asking questions that guided and moved the discussion when needed.