Evaluating Artificial Intelligence-Generated Multiple-Choice Questions in Wilderness Medicine: Quality,Feasibility,and Time Savings for Educators

Abstract

Introduction

Wilderness medicine (WM) multiple-choice questions (MCQs) are a tool used for education and assessment. This study evaluates MCQs generated by a general large language model artificial intelligence (AI), a grounded AI (based on WM resources), and human experts while also assessing potential time savings for educators.

Methods

A randomized 45-question test was created from a general Gemini AI, a grounded Gemini AI, and human written questions (with 15 questions from each source). The grounded Gemini AI was a version grounded on Auerbach's Wilderness Medicine and Wilderness Medical Society guidelines. The human written questions were obtained from prior Medical Wilderness Adventure Races because these questions were written and peer reviewed by medical educators. All questions pertained to 5 core WM topics: hypothermia, drowning, heat illness, altitude medicine, and dive/marine medicine. This test was distributed to members of WM committees with a focus on education to both take the test and rate the questions. Correct responses and ratings for accuracy, clarity, complexity, and relevance were recorded on a 5-point Likert scale. Generalized estimating equations and the 2 one-sided tests framework were used for analysis.

Results

Most respondents were attending-level emergency medicine physicians with at least 5 y of WM experience. Respondents were significantly more likely to correctly answer questions written by both general AI and grounded AI compared with human-written questions. General AI and human-written questions differed significantly with regard to accuracy, clarity, and complexity but not relevance. No statistically significant differences were found between grounded AI and human-written questions on any of the 5 metrics.

Conclusions

AI-generated WM MCQs do not significantly differ from human-written questions across all metrics, including relevance, accuracy, clarity, and complexity. Using AI-generated MCQs edited by a human expert may offer educators high-caliber MCQs with potential time savings, provided that the difficulty level of the questions is sufficiently high.

Keywords

education artificial intelligence multiple-choice questions large language model

Introduction

Learning can be assessed and encouraged by testing. One common tool to test a student's knowledge of material is the multiple-choice question (MCQ). MCQs are common in education because they allow for testing in an objective manner. Trends and patterns also can be noted among test takers.¹ Writing MCQs that test data interpretation, synthesis, decision making, and problem solving is challenging and may require multiple iterations.¹

The recent growth and advancement of artificial intelligence (AI) have introduced new possibilities for medical education, particularly after large language models (LLMs) performed at or near the passing threshold for all 3 United States Medical Licensing Examination tests without any specialized training.² Consequently, AI is increasingly being explored as a tool to help educators create MCQs.^3-5 A study evaluating LLMs for generating questions concluded that the models could create items of comparable standard to those of physicians.³ Further research has supported these findings, showing that by using AI, educators can save significant time, although all studies have concluded that the questions still need expert review.^4,5 Expert review is needed because of the risk of “hallucinations” (AI generating plausible but false information) and the novel use of this technology. Grounded AI systems connect a preexisting AI model to external, verifiable sources of information at the user's request with the goal of increasing reliability and decreasing hallucinations. This investigation focused on content and expert appraisal and learner performance and complements, rather than replicates, these prior studies focused on the psychometric characteristics of MCQs.

Unlike many medical specialties, wilderness medicine (WM) does not have formal board exams or other standardized testing that would prompt the creation of validated question banks. Experts in professional organizations have noted the need for such a resource for elective students, residents, and WM fellows for more than a decade. However, the complexity and time required for writing questions, coupled with the vast amount of wilderness content, have thwarted efforts by at least 2 professional organizations to create such a resource. This study aimed to determine if the use of LLMs to create WM MCQs is a feasible solution to this long-standing problem. We hypothesized that AI-generated MCQs would be comparable in complexity, relevance, accuracy and clarity to those written by human experts and that their use would result in significant time savings for educators. We further expected grounded AI to improve results compared with ungrounded AI. Although this study was not designed to prove equivalence, we do produce minimum equivalence margins that may help inform a follow-up study designed for this purpose. This paper outlines the methodology and results of a study comparing MCQs from a general LLM AI, a grounded LLM AI, and human WM experts.

Methods

Study Design

This study employed a cross-sectional, comparative design to evaluate MCQs generated by 3 different sources: a general LLM AI, a grounded LLM AI, and human experts. A 45-question test, comprising 15 questions from each source, was created and distributed to members of WM committees with an education focus for evaluation. Exact text from the survey evaluating each question and response pair and overall assessment can be found in the online Supplementary Material. The primary outcomes were the mean ratings for question relevance, accuracy, clarity, and complexity and the percentage of correct answers for each question source. Secondary outcomes were the participants’ perception of whether a question was AI written (referred to as “AI-ness”) and the estimated time savings from using AI to assist in writing questions vs writing them from scratch. The study was determined to be exempt by the University of Massachusetts Institutional Review Board (ID No. 00002821).

Participants/Sample

The study population consisted of a convenience sample of WM committee members who self-identified as being involved in WM education. Participants were recruited via committee listservs from the American College of Emergency Physicians and the Wilderness Medical Society. Inclusion criteria required participants to self-report as actively involved in WM education. There were no specific exclusion criteria.

Procedures/Data Collection

Fifteen MCQs were generated from each of the 3 sources: a general LLM AI, a grounded LLM AI, and human experts. To ensure consistency across the question sets, the topics were restricted to 5 core domains in WM: hypothermia, heat illness, altitude medicine, drowning, and dive and marine medicine.

The general LLM AI questions were created using the standard Gemini 2.0 (February 2025) AI model with the following prompt:

Generate 15 wilderness medicine multiple-choice exam questions aimed at physicians. Create five questions for hypothermia, five questions for drowning, five questions for heat illness, five questions for altitude medicine, and five questions for dive and marine medicine. At least half the questions should be scenario or case based, and the question stems should be a couple of sentences long. For all questions, provide a detailed explanation for each answer.

For the grounded LLM AI group, the Gemini AI model was “grounded” by uploading the relevant chapters and guidelines from Auerbach's Wilderness Medicine (7th Edition) textbook and the Wilderness Medical Society's Clinical Practice Guidelines that corresponded to the 5 domains into a secure, private Google Cloud environment. This process ensures the AI responses were based on the provided reference sources. The same prompt was used to generate questions from this grounded model.

For the human-written questions, researchers first identified a pool of questions from a large preexisting question bank used in prior Medical Wilderness Adventure Races that covered the 5 specified domains. An online random number generator was used to assign a number to each question, and the first 15 questions were selected sequentially, ensuring that at least 3 questions from each of the 5 domains were included.

The final set of 45 questions was randomized for the survey using an online random number generator. Each question was assigned a random number, and the questions then were ordered sequentially for presentation in a Qualtrics survey (available by email request to the corresponding author). For each item, participants first answered the question as if they were a test taker. Immediately after submitting their answer, they were asked to rank the question relevancy. Next, the complete question was shown again, this time with the correct answer highlighted and a detailed explanation of the rationale. Participants then were asked to rate a series of statements on a 5-point Likert scale ranging from 1 for “Strongly disagree” to 5 for “Strongly agree.” These statements assessed whether the question, answer, and explanation were accurate, clearly stated, and demonstrated appropriate complexity. A final statement asked participants to rate their perception as to whether AI wrote the question. The ratings of clarity, complexity, accuracy, and relevance were chosen by medical educators to be exploratory in nature and were not validated prompts. At the end of the survey, participants were asked to estimate the time it would take them to write 45 high-quality MCQs from scratch and the time it would take to edit the 45-MCQ test to meet their standards.

Statistical Analysis

To determine whether the general AI questions and the grounded AI questions each differed from the questions written by humans, we made use of generalized estimating equations (GEEs) and an exchangeable working correlation matrix. This approach can account for intra-observation correlation and is a good fit for our data given that each participant responded to all 3 sets of questions. All GEEs further controlled for respondent training, Fellowship in the Academy of Wilderness Medicine completion status, Graduate Medical Education fellowship status, and total years of experience. For each metric of interest, we then performed 2 statistical tests: 1) We investigated whether the estimated beta coefficient (used here to quantify the differences between groups) for each AI group vs the human group significantly differed from zero, and 2) we employed the 2 one-sided tests framework to determine the minimum margin of equivalence, or the minimum “window” for which the 2 sets of questions (general AI vs human or grounded AI vs human) may be considered equivalent. The estimated time savings between the time to write a similar test and time to edit the AI-generated test was calculated using median and median absolute deviation (MAD). Given that time data are often skewed, the median and MAD were used as primary measures of central tendency and variability because they are robust to outliers and skewed distributions. All statistical analyses were performed using Microsoft Excel (MAD) and R 4.5.1 (GEEs) and were subject to a statistical significance threshold of α=0.05, where appropriate. Because this was a pilot study intended to inform larger, more targeted investigations, and because no information exists on a feasible margin of equivalence for these data, an a priori power analysis was not performed.

Results

A total of 36 respondents completed the evaluation. The demographic characteristics of the 36 respondents are summarized in Table 1. Most of the respondents were attending-level physicians (81%) specializing in emergency medicine (67%). A quarter had 5 to 10 y of experience in WM (25%), with about three fifths (58%) having ≥5 y of experience. Just over half of the respondents (53%) had completed the Fellowship in the Academy of Wilderness Medicine.

Table 1.

Demographic and Professional Characteristics of Study Participants (N=36).

Characteristic	Count	Percent
Training
Medical student	1	3%
Resident MD/DO	1	3%
Attending MD/DO	29	81%
APP (PA/NP)	5	14%
Primary specialty
Emergency medicine	24	67%
Family medicine	8	22%
Internal medicine	1	3%
Trauma or surgery	1	3%
Other	2	6%
Completed FAWM?
Yes	19	53%
No	17	47%
GME fellowship?
Yes	11	31%
No	25	69%
Years of experience
≤1 y	2	6%
1–2 y	8	22%
2–3 y	0	0%
3–4 y	3	8%
4–5 y	2	6%
5–10 y	9	25%
10–15 y	5	14%
15–20 y	0	0%
>20 y	7	19%

Note. APP, Advanced Practice Provider; FAWM, Fellowship in the Academy of Wilderness Medicine; GME, Graduate Medical Examination.

Across all sets, the mean respondent correctly answered questions about three quarters of the time (76.9%). Grounded AI questions were answered correctly the most often (80.6%), followed by general AI questions (77.8%) and human-written questions (72.2%), as shown in Figure 1. Mean question metrics (ie, accuracy, clarity, complexity, and relevance) across all sets generally ranged between 3.5 and 4.5 on a 5-point Likert scale, with higher ratings noted for relevance and lower ratings noted for complexity (Figure 2). Across all 3 question sets, the mean AI-ness was ∼3.0. A full listing of these summary statistics can be found in Table 2.

Figure 1.

Mean percentage of questions answered correctly by the educators.

Figure 2.

Comparison of educator ratings of question quality by source.

Table 2.

Question Set Metric Summary Statistics.

Statistic	Overall	Human	General AI	Grounded AI
Correctness	76.85% (14.43%)	72.22% (18.78%)	77.78% (9.29%)	80.56% (8.93%)
Accuracy	3.96 (0.48)	4.06 (0.42)	3.82 (0.55)	4.00 (0.43)
Clarity	3.84 (0.51)	3.89 (0.45)	3.69 (0.56)	3.96 (0.48)
Complexity	3.67 (0.65)	3.77 (0.60)	3.53 (0.69)	3.73 (0.64)
Relevance	4.39 (0.45)	4.38 (0.45)	4.34 (0.51)	4.45 (0.41)
AI-ness	3.02 (0.43)	2.93 (0.45)	3.08 (0.47)	3.04 (0.35)

Note. All results are presented as mean (SD). Correctness is percentage correct from 0% (minimum) to 100% (maximum). All other metrics are on a 5-point Likert scale from 1 (maximum disagreement) to 5 (maximum agreement). Because respondents answered questions in all 3 sets, N=36 throughout.

The GEEs demonstrated that, on average, respondents correctly answered a significantly greater proportion of questions written by both general AI ($\hat{\beta }$=0.115, P=0.001) and grounded AI ($\hat{\beta }$=0.143, P=0.001) compared with humans. According to the estimated beta coefficients (the $\hat{β}$ values), the difference was 11.5 percentage points for general AI and 14.3 percentage points for grounded AI. To claim equivalence to human-written questions in terms of correctness under the 2 one-sided tests framework, equivalence margins of at least Δ=±17.0 and Δ=±19.0 percentage points would have needed to have been set a priori for general AI and grounded AI, respectively. Beyond this, there were no statistically significant differences between the questions written by grounded AI and those written by humans with respect to relevance, accuracy, clarity, complexity, and AI-ness, with minimum equivalence margins across these 5 domains ranging from 0.135 to 0.220 on a 5-point Likert scale. In contrast, the questions written by general AI and those written by humans differed significantly on accuracy, clarity, complexity, and AI-ness, with the general AI questions judged to be less accurate ( $\hat{β}$ =−0.239, P<0.001), less clear ( $\hat{β}$ =−0.204, P<0.001), less complex ( $\hat{β}$ =−0.237, P<0.001), and more AI-ish ( $\hat{β}$ =0.156, P=0.021) than the human-written questions. There was no statistically significant difference between the relevance of these 2 sets of questions, however. For the general AI, across all question domains save correctness, minimum equivalence margins ranged from 0.093 to 0.331. A full listing of results from these models can be found in Table 3.

Table 3.

Generalized Estimating Equation Model Results.

Factor	Estimate of $\hat{β}$ [95% CI]	Results of H₀: $\hat{β}$ =0 vs H_A: $\hat{β}$ ≠0	Minimum Margin of Equivalence (Δ_min)
Correctness
General AI vs human	0.115 [0.049, 0.181]	Rejected H₀ P=0.001	0.170
Grounded AI vs human	0.143 [0.086, 0.199]	Rejected H₀ P<0.001	0.190
Relevance
General AI vs human	−0.035 [−0.105, 0.034]	Failed to reject H₀ P=0.320	0.093
Grounded AI vs human	0.072 [−0.007, 0.152]	Failed to reject H₀ P=0.075	0.139
Accuracy
General AI vs human	−0.239 [−0.338, −0.139]	Rejected H₀ P<0.001	0.322
Grounded AI vs human	−0.061 [−0.163, 0.040]	Failed to reject H₀ P=0.238	0.146
Clarity
General AI vs human	−0.204 [−0.304, −0.103]	Rejected H₀ P<0.001	0.288
Grounded AI vs human	0.067 [−0.034, 0.167]	Failed to reject H₀ P=0.194	0.151
Complexity
General AI vs human	−0.237 [−0.349, −0.125]	Rejected H₀ P<0.001	0.331
Grounded AI vs human	−0.041 [−0.154, 0.072]	Failed to reject H₀ P=0.479	0.135
AI-ness
General AI vs human	0.156 [0.024, 0.288]	Rejected H₀ P=0.021	0.266
Grounded AI vs human	0.119 [−0.003, 0.240]	Failed to reject H₀ P=0.056	0.220

Note. All models control for respondent training, specialty, Fellowship in the Academy of Wilderness Medicine status, fellowship status, and years of experience. Correctness is parameterized as the proportion of questions answered correctly and ranges from 0 to 1. All other metrics are parameterized using a 5-point Likert scale from 1 (“Strongly disagree”) to 5 (“Strongly agree”). Minimum margin of equivalence is the minimum value for which both H₀: $\hat{β}$ ≤–Δ_min and H₀: $\hat{β}$ ≥Δ_min are rejected, thereby suggesting equivalence under the 2 one-sided tests framework.

In terms of time burdens, respondents estimated that it would take a median of 220 (MAD=100) minutes to write 45 MCQs from scratch compared to a median of 60 (MAD=42.5) minutes to review and edit 45 AI-generated questions. This represents an estimated perceived time savings of 160 min, or ∼3.5 min per question. Please note that these are self-reported estimates and that actual times to write or revise questions were not tested in this study.

Discussion

The goal of this pilot study is to generate preliminary insights to encourage further consideration and more robust confirmatory research. This study was not designed as a superiority trial; rather, it was an exploratory investigation into the adequacy of AI-generated questions as a time-saving tool for medical educators. The primary finding of this preliminary study is that AI-generated MCQs, from a grounded LLM, revealed no statistically significant differences in the accuracy, clarity, or complexity compared with human-written questions. General AI was ranked significantly lower for accuracy, clarity, and complexity compared with human-written questions. Notably, though, questions written by both grounded AI and general AI were significantly more likely to be answered correctly than questions written by humans, with the computed minimum equivalence margins being quite wide (see Table 3). Possible explanations are that 1) AI MCQs may be of lower difficulty than questions written by humans, 2) AI (and more so grounded AI) questions were easier to understand, or 3) the AI may draw from more commonly known information. However, we cannot draw any definitive conclusions from this study.

Provided that question difficulty is not a key concern, these findings suggest that grounded AI may serve as a reliable tool for generating assessment materials in a highly specialized medical field. These findings align with previous research in other specialties, which found that LLMs can produce many questions not markedly different from those created by physicians, although expert oversight remains essential.^3,6

A contribution of this study is the quantification of the potential efficiency gains for educators. Respondents estimated a perceived time savings of 160 min for creating a 45-question test when using AI-generated questions as a starting point for editing compared with writing them from scratch. Although the MAD values indicate a high degree of variability in the time estimates, particularly for the more demanding task of writing from scratch (MAD=100), the medians remain distinct, reinforcing a clear central tendency toward greater efficiency with AI assistance. These statistics suggest that the perception of time savings holds true despite the variation in individual estimates. This finding reinforces the idea that LLMs are effective tools for generating quick drafts that require expert validation, thereby streamlining the content-creation process.^4,5 Notably, these are all estimates, and the time to write questions was not measured. However, it is congruent with what is reported in the literature of 211 min to write 50 questions, with an average of 4.2 min per question.⁶ Notably, though, there is significant variance in the literature regarding time to write MCQs.⁷

The lack of a significant difference in complexity between grounded AI and human-written questions is consistent with a study on grounded AI questions for obstetrics and gynecology residents and attendings.⁸ A study regarding doctors preparing for emergency medicine exams found that general AI questions lacked complexity compared with human-written questions,⁶ consistent with our findings. This study found a significant time savings of 119 min between ChatGPT questions and human-written questions, without sacrificing quality.⁶

This study has several limitations. First, the sample size of 36 self-identified WM educators is relatively small and may not be representative of all WM educators. We did not fully characterize the credentials of our respondents as educators or their expertise with MCQ writing or evaluation. It is possible the respondents’ lack of question-writing experience may have skewed estimates. Second, the human-written questions were sourced from a preexisting bank and may not have been created with the same rigorous, standardized process as questions for a board examination. Third, the estimation of time saved was based on self-reporting and may be subject to recall bias. Timing the writing and editing of MCQs would provide more definitive data. Instead, we report an estimation in perceived times to complete each task. We did not obtain qualitative data on perceptions regarding questions or perform a full psychometric analysis. Finally, these questions were generated by Gemini 2.0 from the first quarter of 2025.

Future research should explore the impact of different fine-tuning methodologies on question creation, determine the role of prompt engineering, investigate the performance of AI-generated questions with novice learners, and analyze the potential influence of educator characteristics on question perception. Additionally, different AI LLMs could be compared, and follow-up studies using the minimum equivalence margins computed here could be performed with the express purpose of demonstrating equivalence.

Conclusions

In conclusion, this pilot study provides initial evidence that grounded AI is a feasible and efficient tool for generating MCQs in WM. AI-generated questions did not differ significantly from human-written questions across several metrics and may offer a potential time-saving advantage for educators. Further work is needed to fully validate this work, but integration of AI into the curriculum development and assessment process represents a promising avenue for addressing the long-standing need for a comprehensive question bank in WM.

Supplemental Material

sj-docx-1-wem-10.1177_10806032261460022 - Supplemental material for Evaluating Artificial Intelligence-Generated Multiple-Choice Questions in Wilderness Medicine: Quality, Feasibility, and Time Savings for Educators

Supplemental material, sj-docx-1-wem-10.1177_10806032261460022 for Evaluating Artificial Intelligence-Generated Multiple-Choice Questions in Wilderness Medicine: Quality, Feasibility, and Time Savings for Educators by Carver Haines, Susanne J. Spano, Hillary R. Irons, Ben Brewer and Stephanie Lareau in Wilderness & Environmental Medicine

Footnotes

ORCID iDs

Carver Haines

Susanne J. Spano

Ben Brewer

Stephanie Lareau

Author Contribution(s)

Carver Haines: Conceptualization; Data curation; Formal analysis; Writing – original draft; Writing – review & editing.

Susanne J. Spano: Conceptualization; Data curation; Formal analysis; Writing – original draft; Writing – review & editing.

Hillary R. Irons: Conceptualization; Data curation; Formal analysis; Writing – original draft; Writing – review & editing.

Ben Brewer: Data curation; Writing – review & editing.

Stephanie Lareau: Conceptualization; Data curation; Formal analysis; Writing – original draft; Writing – review & editing.

Financial/Material Support

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Previous Presentation

Presented as a research abstract presentation at the 2025 Wilderness Medicine Society Summer Conference, Lake Geneva, Wisconsin, July 23, 2025.

Supplemental Material

Supplementary material associated with this article can be found in the online version at https://doi.org/10.1177/10806032261460022.

References

Coughlin

Featherstone

. How to write a high quality multiple choice question (MCQ): a guide for clinicians. Eur J Vasc Endovasc Surg. 2017;54(5):654-658.

Kung

Cheatham

Medenilla

, Sillos C, De Leon L, Elepano C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198.

Laupichler

Rother

Kadow

, Grunwald Kado IC, Ahmadi S, Raupach T. Large language models in medical education: comparing ChatGPT- to human-generated exam questions. Acad Med. 2024;99(5):508-512.

Kaya

Sonmez

Halici

Yildirim

Coskun

. Comparison of AI-generated and clinician-designed multiple-choice questions in emergency medicine exam: a psychometric analysis. BMC Med Educ. 2024;24(1):949.

Law

AKK

Lui

, Choi YF, Cheung KH, Hung KKC, et al. AI Versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination. BMC Med Educ. 2024;24(1):208.

Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS et al. ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLoS One. 2023;18(8):e0290691.

Kurtz

Holman

Monrad

. Training medical students to create and collaboratively review multiple-choice questions: a comprehensive workshop. MedEdPORTAL. 2020;16:10986.

Jackson

Keller

Diaz-Garelli

, Kouba W, Bracero LA, Blitz MJ. Large language model clinical vignettes and multiple-choice questions for postgraduate medical education. Acad Med. 2025;100(10):1163–1166.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.94 MB