Effectiveness of simulation-based training for obstetric internal medicine: Impact of cognitive load and emotions on knowledge acquisition and retention

Abstract

Background

Simulation-based training’s impact on learning outcomes may be related to cognitive load or emotions during training. We evaluated the association of validated measures of cognitive load and emotion with learning outcomes in simulation-based obstetric internal medicine cases.

Methods

All internal medicine learners (n = 15) who completed the knowledge test pre-training, post-training (knowledge acquisition), and at 3–6 months (knowledge retention) for all three simulation cases were included.

Results

Mean knowledge scores differed over time in all three cases (p < 0.0001 for all). Knowledge retention scores were significantly higher only for cases 1 and 3. Cognitive load associated with frustration was positively associated with knowledge acquisition for case 2 (beta = 5.18, P = 0.007), while excitement was negatively associated with knowledge retention in case 1 (beta = –33.07, p = 0.04).

Conclusion

Simulation-based education for obstetric internal medicine can be effective in select cases. Attention to cognitive load and emotion may optimize learning outcomes.

Keywords

Simulation training patient simulation graduate medical education cognitive load

Background

According to the Royal College of Physicians and Surgeons of Canada, the management of medically complex pregnant women is considered a core competency for both internal medicine and general internal medicine trainees.^1,2 However, a survey of graduates from General Internal Medicine training programs across Canada demonstrated that trainees felt unprepared in the management of medical disorders in pregnancy.³ This is not surprising given the complexity of obstetric medicine cases, the risk of severe maternal and fetal morbidity and mortality, and limited exposure to obstetric medicine cases throughout training. Simulation may be an effective educational tool to provide trainees with exposure to medically complex obstetric cases not routinely encountered during training to assist in developing competency in obstetric medicine.

Simulation education is widely used for training healthcare professionals and has shown a moderate positive educational effect size when compared to other forms of instruction in healthcare.^4–6 It is an effective method to teach skills in stressful scenarios such as advanced cardiac life support (ACLS) and resuscitation.^6,7 However, the evidence for using simulation in obstetric medicine is limited. We previously conducted a randomized control trial evaluating the use of scenario-based simulation in an obstetric medicine case.⁸ Our study did not demonstrate improved knowledge acquisition and retention compared to conventional case-based instruction and even demonstrated a non-statistically significant decrement in knowledge acquisition and retention.⁸ We hypothesized that the absence of a positive learning outcome may have been due to high anxiety and cognitive overload experienced during the obstetric simulation. Indeed, simulation-based training has been shown to increase stress levels compared to conventional tutorial-based training,⁹ and the impact of stress on attention, decision making, and memory may affect learning and performance in unpredictable ways.¹⁰ The simulated environment requires the learner to use their knowledge and skills in real time, respond to abrupt changes in the status of the standardized patient/mannequin, communicate with other health care professionals, and utilize crisis resource management skills, all of which may serve to impair learning. However, existing data examining emotion and cognitive load on knowledge and performance in simulation is conflicting. High cognitive load in simulation has been shown to be a potential barrier to learning and performance in some studies.^11–14 In contrast, an observational study examining trainees participating in an ACLS simulation found that added anxiety correlated with enhanced performance,¹⁵ whereas other studies have shown poor correlation between self-reported stress and performance.^16,17 In another study on medical students, stressors improved clinical performance.¹⁸

To examine the relationship between cognitive load, emotion, and learning, our study seeks to evaluate the effectiveness of simulation-based training on learning outcomes across three different simulated obstetric medicine case presentations and to analyze the effect of cognitive load and emotion on knowledge acquisition and retention. Our study utilized the NASA Task Load Index (NASA-TLX) to measure cognitive load.¹⁹ This estimate of mental workload has been used in a variety of medical and medical education settings, including surgical procedures and learning,^20,21 resuscitation,²² endoscopic procedural training,²³ and point-of-care ultrasound performance.²⁴ Higher workload has previously been demonstrated to be associated with poorer performance.¹³ For learner emotions, we measured the dimensions of affect proposed by Feldman Barrett and Russel,²⁵ whereby more negative emotions had previously been found to be associated with worse learning outcomes.²⁶

Methods

Participants

All postgraduate year (PGY)-2 and -3 internal medicine residents from the University of Calgary were invited to participate in this study between 2018 and 2020. This study was approved by the University of Calgary Conjoint Health Research Ethics Board.

Simulation scenarios

Between September 2018 and September 2019, each participant completed three simulation-based training sessions, in groups of 2–3 learners per group. Only participants who provided written informed consent were included in this study. Prior to the session, participants were provided with a five-minute orientation to the simulated environment and features of the computer-based mannequin (Laerdal SimMan® 3 G). Each simulation session was led by three simulation-trained instructors: one to operate the mannequin, one to act as the patient’s voice and consulting specialists, and one to act as the nurse confederate. Participants were instructed to evaluate the simulated patient as a team and each participant was encouraged to perform as a team leader for at least one simulation case. Each interactive case took 15–20 min. This is followed by a 15-min debrief,^27,28 whereby the first 5 min, facilitated by a simulation-trained instructor, were devoted to addressing learner reactions to the simulation case and/or environment. In the remaining 10 min, a didactic lecture was provided by an obstetric general internist, with discussion of learner performances and a summary of key teaching points (Supplementary Table 1). Content of the lecture covered key obstetric medicine learning objectives pertaining to the cases, based on the Royal College of Physicians and Surgeons of Canada Objectives of Training for the specialty of internal medicine.¹

The obstetric internal medicine simulation cases were created based on chart reviews from patients. Case 1 was on severe hypertension in pregnancy, case 2 was on thromboembolic disease in pregnancy, and case 3 involved cardiac arrest in pregnancy (Supplementary Table 2a–c). These cases were piloted in 2017 on PGY-4 and PGY-5 (n = 6) general internal medicine trainees to ensure optimal content and flow prior to starting the study. All learners were presented with these three cases in the same order.

Outcome assessment

Knowledge was assessed using a multiple-choice test administered before and after the simulation case and didactic session. Number of items on the multiple-choice test ranged from 8 to 12, depending on the topic assessed. The same multiple-choice test was then administered 3–6 months after completing the simulation case. Knowledge assessment scores are presented as the number of correct responses divided by the total number of questions, presented as a percent. Knowledge acquisition was defined as the difference in test scores between post- and pre-simulation. Knowledge retention was defined as the difference in test scores between 3 and 6 months post-simulation and pre-simulation. The multiple-choice tests were created by an expert panel consisting of two obstetric medicine specialists and one specialist with expertise in general internal medicine and medical education. The tests were constructed based on a blueprint to assess multiple cognitive domains including knowledge, comprehension, application, analysis, and synthesis of information pertaining to the obstetric medicine simulation case.

Self-reported cognitive load post-simulation was assessed using the NASA-TLX, a validated subjective assessment tool that rates perceived workload of a task.¹⁹ This tool measures cognitive load on six domains (mental demand, physical demand, temporal demand, performance, effort, and frustration) using a 100-point scale where 0 = very low and 100 = very high. A score of 60 or higher would be considered high.²⁹ Self-reported emotion post-simulation was measured using a validated tool that requires participants to rate eight different emotions on a five-point Likert scale to describe their emotional state.^25,26 Change in self-reported comfort and confidence in the evaluation and management of medically unwell pregnant women was assessed utilizing a three-item questionnaire conducted before and after each simulation case. Comfort and confidence were rated using a five-point Likert scale,⁸ where 1= strongly disagree and 5 = strongly agree with the statements regarding being comfortable or confident.

Statistical analysis

Differences in scores over the three time points were evaluated using repeated-measures ANOVA, with post hoc Tukey analyses corrected for multiple comparisons. Comparisons of measures pre- and post-session were performed using paired t-tests, while comparisons of measures across cases were performed using ANOVA testing. The association of baseline variables, cognitive load, and emotions with knowledge acquisition and knowledge retention was assessed using linear regression analyses. Variables that were significant at a univariate level (P < 0.05) were entered into a multivariate model whereby all domains of that variable (cognitive load or emotions) were included, in order to evaluate for the independent association of that variable. All analyses were performed using SAS version 9.2 (SAS Institute Inc., Cary, NC).

Results

Twenty-one trainees consented to participate and 15 (71%) completed the study protocol. Of the 15 trainees, 60% were PGY-2 (n = 9), 40% were PGY-3 (n = 6), 53% were male (n = 8), and 47% were female (n = 7). None of the participants had previously completed a rotation in obstetric internal medicine.

Mean knowledge assessment scores over the three time points were significantly different in all three cases (pre-test 44.0 ± 19.2, post-test 78.0 ± 10.1, 3–6 months 59.3 ± 14.9, P < 0.001 for case 1; pre-test 54.0 ± 19.6, post-test 84.2 ± 13.9, 3–6 months 64.9 ± 15.1, P < 0.0001 for case 2; pre-test 51.9 ± 19.9, post-test 87.7 ± 12.4, 3–6 months 76.9 ± 9.3, P < 0.0001 for case 3, Figure 1). Post hoc analysis suggests that retention scores were significantly higher only for case 1 and case 3, but not for case 2. In case 2, only the post-test score was significantly higher than the pre-test score (P < 0.0001); retention score was not different from pre-score (P = 0.06).

Cognitive load scores and self-reported emotion across the three cases are presented in Table 1. Cognitive load was significantly higher in case 3 than cases 1 and 2 for five of the six domains and emotions were more negative for case 3 in four out of eight emotions. Overall mean cognitive load was below 60 in all instances except mental demand, temporal demand, and effort in case 3.

Table 1.

NASA-TLX cognitive load scores and self-reported emotion post-simulation cases, presented as mean (standard deviation).

	Case 1	Case 2	Case 3	P-value
Cognitive load
Mental demand	55.0 (18.3)	56.0 (11.1)	79.5 (12.6)	<0.0001
Physical demand	14.6 (21.6)	21.2 (19.4)	58.5 (31.7)	<0.0001
Temporal demand	27.7 (17.8)	30.0 (21.3)	65.5 (18.7)	<0.0001
Performance	50.2 (20.3)	45.3 (20.9)	52.7 (13.4)	0.66
Effort	50.8 (16.5)	50.3 (18.5)	72.3 (17.3)	0.0004
Frustration	39.6 (19.8)	22.8 (14.7)	56.5 (23.0)	0.0002
Emotion
Tense/calm (1–5)	3.5 (0.9)	3.7 (0.9)	2.4 (1.1)	0.002
Nervous/relaxed	3.2 (1.0)	3.7 (0.9)	2.6 (1.2)	0.05
Stressed/serene	3.3 (0.9)	3.5 (0.7)	2.6 (1.1)	0.04
Upset/content	4.2 (0.7)	4.2 (0.7)	3.5 (0.7)	0.02
Sad/happy	3.9 (0.8)	4.1 (0.7)	3.6 (1.1)	0.32
Depressed/elated	3.8 (0.6)	3.8 (0.6)	3.4 (0.6)	0.049
Lethargic/excited	3.7 (0.6)	3.6 (0.6)	3.3 (1.1)	0.36
Bored/alert	4.2 (0.6)	4.1 (0.7)	4.3 (0.7)	0.54

On univariate analyses, knowledge acquisition was not associated with baseline variables (gender, training level), individual cognitive load domains or emotions (all P > 0.05) except for frustration (beta = 3.68, SE 1.43, P = 0.02) in case 2 (Supplementary Table 3). Adjusted for all cognitive load domains, frustration remained positively associated with knowledge acquisition (beta = 5.18, SE 1.44, P = 0.007, Supplementary Table 4).

Knowledge retention was not associated with gender, training level, or individual cognitive load domains, except for mental demand for case 3 (beta = 5.28, SE 1.73, P = 0.009, Supplementary Table 5), which after adjusting for the effects of other cognitive load domains, was no longer significant (beta = 6.51, SE 2.95, P = 0.06; Supplementary Table 6). Knowledge retention was not associated with individual emotion except for lethargy-excited for case 1 (beta = –17.97, SE 8.26, P = 0.048, Supplementary Table 5). After adjusting for the effects of the other emotions, lethargy-excited remained independently associated with knowledge retention in case 1 (beta = –33.07, SE 12.51, P = 0.04, Supplementary Table 6).

Mean self-reported comfort in participating and confidence in the evaluation and management of all three cases were statistically higher post-simulation compared with pre-simulation with the exception of comfort in participating in the care of medically complicated pregnant women after the cardiac arrest in pregnancy case (Table 2).

Table 2.

Mean self-reported comfort measures (± standard deviation) pre- and post-simulation based training, where 1 = strongly disagree and 5 = strongly agree.

	Pre	Post	P-value
I am comfortable participating in the care of medically complicated pregnant women
Case 1	1.9 (±0.8)	2.9 (±1.1)	0.0004
Case 2	2.3 (±0.5)	3.5 (±0.6)	<0.0001
Case 3	3.2 (±0.9)	3.7 (±0.5)	0.07
I am confident in my ability to appropriately evaluate a pregnant woman with medical complaints
Case 1	2.9 (±0.9)	4.1 (±0.3)	0.0003
Case 2	3.1 (±1.0)	4.0 (±0)	0.002
Case 3	2.9 (±1.1)	3.9 (±0.5)	0.004
I am confident in my ability to manage a pregnant woman with medical complaints
Case 1	2.5 (±0.9)	3.7 (±0.6)	0.0009
Case 2	2.5 (±0.6)	4.1 (±0.3)	<0.0001
Case 3	2.3 (±0.9)	3.5 (±0.8)	0.002

Conclusion

In this study, we designed three simulation-based training cases covering obstetric internal medicine topics. While knowledge acquisition was demonstrated immediately post-teaching in all three cases, knowledge retention was only significant in two out of three cases. Further, in all three cases, scores at 3–6 months were lower than scores immediately post-training, suggesting some degree of knowledge decrement over time post-training. We were unable to consistently demonstrate the impact of cognitive load and emotion on knowledge acquisition and knowledge retention, although a high cognitive load imposed by frustration appeared associated with a higher knowledge acquisition in case 2, while learners who reported higher excitement for case 1 had surprisingly lower knowledge retention. Lastly, learners reported a significant increase in comfort in participating in the care of medically complicated pregnant women after two out of three cases. They also reported an increase in their confidence in evaluating and managing these women after all three cases. Altogether, our results suggest the following: with careful attention to cognitive load and learner emotions during the design and execution of simulation-based training, significant knowledge retention can sometimes be demonstrated, but by no means a guaranteed learning outcome. While it is likely that cognitive load and emotion plays a role in knowledge acquisition and retention, the exact individual contribution by these domains remains unknown. No specific pattern of association can be identified, but our study was limited both in sample size (n = 15) and in scope (number of cases explored).

Our results demonstrating the association of higher frustration with increased learning is not novel. For example, in a study of math teaching, students experiencing greater frustration demonstrated higher learning gains.³⁰ However, the relationship between frustration and learning is likely not straightforward. For example, others have found that while brief episodes of frustration were associated with improved learning, lengthy periods impaired learning.³¹ In addition, our results in case 1 showed that those who reported higher excitement had lower knowledge retention. While these results seem to be contradictory to the principles of active learning,³² they may nevertheless be a reflection of the complexity of the learning process.

In addition to the aforementioned sample size and scope limitations, our study has further limitations that must be considered in the interpretation of the results. First, because this study was conducted in a single center, generalizability may be limited. Our cases were solely obstetric medicine cases, to which the medical learners at our center have limited exposure. Moreover, our cases were designed by educational experts and executed by those with expertise in simulation, and cases were delivered in the same order each time to each learner (starting with severe hypertension and ending with the cardiac arrest case). Educational outcomes may differ if the educational context significantly differs from ours. Second, despite careful attention to issues pertaining to case design, the cognitive load in case 3 did exceed our intended maximum of 60 on half of the domains. Nonetheless, knowledge was significantly retained for this case, suggesting that learning may not have been significantly impaired as a result. Third, our cases were executed with small groups of learners rather than individual learners. The presence of other learners may alter the cognitive load and/or emotion perceived by individual learners in ways that are not always predictable. However, given the time and resource-intensiveness of simulation-based education, having two or three educators for every individual learner was simply not feasible in our educational setting. Fourth, our self-reported comfort and confidence measures were on the general evaluation and management of medically unwell pregnant women, rather than case-specific measures. Fifth, our learning outcomes involved only knowledge acquisition and retention. Logistical issues prevented the evaluation of higher-level outcomes such as learner performance. It is conceivable that cognitive load and emotion may affect learner performance differently than knowledge outcomes. Lastly, although our initial study intended to assess knowledge retention at three months, scheduling challenges resulted in delays such that knowledge in some cases was assessed as late as six months, thus modifying our learning outcome to the range of between three and six months.

Despite these weaknesses, our study has a number of strengths. First, our learning outcomes included the evaluation of knowledge retention rather than simply knowledge acquisition. Evaluation of knowledge acquisition only would have led to an over-estimation of the benefits of simulation-based training. Second, our cases represented a range of content difficulty as well as acuity in patient presentation. This allows us to have a broader look at the impact of cognitive load and emotion on learning outcomes in more than one setting. Our results are consistent with our suspicion that while cognitive load and emotions may play a role in learning, how they do so is likely context-specific. Thus, it remains prudent for educators to take these contributors into account when designing and executing training cases. Specifically, from a practical standpoint, educators should adhere to design principles and strategies that minimize cognitive overload for the learner.³³ For example, the order of case introduction should begin with the simpler cases to those of increasing complexity.³³ Second, cognitive load and emotions should be monitored and measured. In our cardiac arrest case, the mean cognitive load exceeded 60 in three of the six domains. In such instances, at a minimum, educators should ensure that learning outcomes are not negatively impacted. In the event that learning is impaired, case modification should occur or training environment be modified with cognitive load and learning outcomes re-measured. Alternatively, educators can prophylactically modify the training to ensure that the cognitive load is not high and compare learning outcomes before and after modification to clarify the impact of the high cognitive load.

In conclusion, our results suggest that simulation-based education for obstetric internal medicine can be effective in select cases. In the design and execution of simulation-based training, careful attention to cognitive load and emotion may help optimize learning outcomes.

Supplemental Material

sj-pdf-1-obm-10.1177_1753495X211011915 - Supplemental material for Effectiveness of simulation-based training for obstetric internal medicine: Impact of cognitive load and emotions on knowledge acquisition and retention

Supplemental material, sj-pdf-1-obm-10.1177_1753495X211011915 for Effectiveness of simulation-based training for obstetric internal medicine: Impact of cognitive load and emotions on knowledge acquisition and retention by Kelle D Hurd, Meghan EO Vlasschaert, T Lee Ann Hawkins, Jolene Haws, Julia N Kupis and Irene WY Ma in Obstetric Medicine

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: IWYM is the Chair Holder of the John A. Buchanan Chair in General Internal Medicine, University of Calgary.

Funding

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: This study was funded by the 2018 University of Calgary Teaching and Learning Grant, Taylor Institute for Teaching and Learning, University of Calgary.

Ethical approval

The University of Calgary Conjoint Health Research Ethics Board approved this study (REB14-0004).

Informed consent

Written consent was obtained from the participants for their anonymized information to be published in this article.

Guarantor

KDH.

Contributorship

KDH, MEOV, TLAH, IWYM researched literature and conceived the study. All were involved in protocol development and ethical approval. All were involved in participant recruitment. JNK and IWYM were involved in data analysis. KDH and IWYM wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Data availability

The data that supports the findings of this study are available from the corresponding author (IWYM), upon reasonable request.

Acknowledgements

We would like to thank all the learners who participated in this study.

ORCID iDs

Kelle D Hurd

Irene WY Ma

Supplemental material

Supplemental material for this article is available online.

References

Objectives of Training in the Specialty of Internal Medicine. Royal College of Physicians and Surgeons of Canada, www.royalcollege.ca/rcsite/documents/ibd/internal_medicine_otr_e.pdf (2011, accessed 28 January 2021).

Objectives of Training in the Subspecialty of General Internal Medicine. The Royal College of Physicians and Surgeons of Canada, www.royalcollege.ca/rcsite/documents/ibd/general_internal_medicine_otr_e.pdf (2012, accessed 28 January 2021).

Card

Snell

O’Brien

Are Canadian General Internal Medicine training program graduates well prepared for their future careers?

BMC Med Educ 2006; 6: 56.

Cook

Hatala

Brydges

, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011; 306: 978–988.

Cook

Brydges

Hamstra

, et al. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc 2012; 7: 308–320.

Mundell

Kennedy

Szostek

, et al. Simulation technology for resuscitation training: a systematic review and meta-analysis. Resuscitation 2013; 84: 1174–1183.

Huang

Tang

, et al. Educational efficacy of high-fidelity simulation in neonatal resuscitation training: a systematic review and meta-analysis. BMC Medical Educ 2019; 19: 323.

Kerr

Hawkins

TL-A

Herman

, et al. Feasibility of scenario-based simulation training versus traditional workshops in continuing medical education: a randomized controlled trial. Med Educ Online 2013; 18: 21312.

Bong

Lightdale

Fredette

, et al. Effects of simulation versus traditional tutorial-based training on physiologic stress levels among clinicians: a pilot study. Simul Healthc 2010; 5: 272–278.

10.

LeBlanc

VR.

The effects of acute stress on performance: implications for health professions education. Acad Med 2009; 84: S25–S33.

11.

Fraser

Teteris

, et al. Emotion, cognitive load and learning outcomes during simulation training. Med Educ 2012; 46: 1055–1062.

12.

Schlairet

Sauls

, et al. Cognitive load, emotion, and performance in high-fidelity simulation among beginning nursing students: a pilot study. J Nurs Educ 2015; 54: S5–S11.

13.

Yurko

Scerbo

Prabhu

, et al. Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc 2010; 5: 267–271.

14.

LeBlanc

MacDonald

McArthur

, et al. Paramedic performance in calculating drug dosages following stressful scenarios in a human patient simulator. Prehosp Emerg Care 2005; 9: 439–444.

15.

Demaria

Jr Bryson

Mooney

, et al. Adding emotional stressors to training in simulated cardiopulmonary arrest enhances participant performance. Med Educ 2010; 44: 1006–1015.

16.

Clarke

Horeczko

Cotton

, et al. Heart rate, anxiety and performance of residents during a simulated critical clinical encounter: a pilot study. BMC Med Educ 2014; 14: 153.

17.

Dhaif

Paparoidamis

Sideris

, et al. The role of anxiety in simulation-based dexterity and overall performance: Does it really matter? J Invest Surg. 2019; 32: 164–169.

18.

Pottier

Hardouin

Dejoie

, et al. Effect of extrinsic and intrinsic stressors on clinical skills performance in third-year medical students. J Gen Intern Med 2015; 30: 1259–1269.

19.

NASA TLX: Task Load Index. National Aeronautics and Space Administration, http://humansystems.arc.nasa.gov/groups/TLX/ (accessed 28 January 2021).

20.

Dias

Ngo-Howard

Boskovski

, et al. Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg 2018; 105: 491–501.

21.

Ruiz-Rabelo

Navarro-Rodriguez

Di-Stasi

, et al. Validation of the NASA-TLX score in ongoing assessment of mental workload during a laparoscopic learning curve in bariatric surgery. Obes Surg 2015; 25: 2451–2456.

22.

Zehnder

Law

BHY

Schmölzer

GM.

Assessment of healthcare provider workload in neonatal resuscitation. Front Pediatr 2020; 8: 598475.

23.

Mohamed

Raman

Anderson

, et al. Validation of the National Aeronautics and Space Administration Task Load Index as a tool to evaluate the learning curve for endoscopy training. Can J Gastroenterol Hepatol 2014; 28: 155–159.

24.

Aldekhyl

Cavalcanti

Naismith

LM.

Cognitive load predicts point-of-care ultrasound simulator performance. Perspect Med Educ 2018; 7: 23–32.

25.

Feldman Barrett

Russell

JA.

Independence and bipolarity in the structure of current affect. J Pers Soc Psychol 1998; 74: 967.

26.

Fraser

Huffman

, et al. The emotional and cognitive impact of unexpected simulated patient death: a randomized controlled trial. Chest 2014; 145: 958–963.

27.

Sawyer

Eppich

Brett-Fleegler

, et al. More than one way to debrief: a critical review of healthcare simulation debriefing methods. Simul Healthc 2016; 11: 209–217.

28.

Rudolph

Simon

Dufresne

, et al. There’s no such thing as “nonjudgmental” debriefing: a theory and method for debriefing with good judgment. Simul Healthc 2006; 1: 49–55.

29.

Grier

RA.

How high is high? A meta-analysis of NASA-TLX global workload scores. Proc Hum Factors Ergon Soc Annu Meet 2015; 59: 1727–1731.

30.

Richey

Andres-Bray

JML

Mogessie

, et al. More confusion and frustration, better learning: the impact of erroneous examples. Comput Educ 2019; 139: 173–190.

31.

Liu

Pataranutaporn

Ocumpaugh

, et al. Sequences of frustration and confusion, and learning. Proceeding 6th International Conference, Educ Data Mining 2013; 114–120.

32.

Bonwell

Eison

JA.

Active learning: creating excitement in the classroom. 1991 ASHE-ERIC higher education reports. ERIC clearinghouse on higher education, The George Washington University, Washington, DC, 1991.

33.

Van Merriënboer

JJG

Sweller

Cognitive load theory in health professional education: design principles and strategies. Med Educ 2010; 44: 85–93.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB