Deliberate Practice Improves Pediatric Residents’ Skills and Team Behaviors During Simulated Neonatal Resuscitation

Abstract

Objective. To evaluate the skills and team behavior of pediatric residents during resuscitation with a high-fidelity mannequin before and after a deliberate practice intervention. Methods. Each month residents participate in two 90-minute videorecorded sessions (2-3 weeks apart) in an “off-site” delivery room during their neonatal ICU rotation. Teams responded to a scenario that required 5 skills (positive pressure ventilation, chest compressions, endotracheal intubation, umbilical vein catheterization, and epinephrine administration). Skills were scored for technique and timeliness and team behaviors for communication, management, and leadership. A 2-hour focused intervention was given between sessions. Results. In all, 33 residents (11 teams) completed the sessions. Gaps in procedural skills noted during the first session were corrected. Timeliness for completion of skills remained below expectations. Improvements in team behaviors were noted. Conclusions. Deliberate practice improved procedural skills and team performance. Lack of improvement in timeliness suggests that a different educational paradigm is required.

Keywords

deliberate practice neonatal resuscitation simulation

Introduction

Pediatric residents completed the neonatal resuscitation program (NRP) at the beginning of their residency and every 2 years thereafter. As part of this certification, they demonstrate competency in NR.^1-3 Throughout the years, gaps in residents’ readiness to perform NR have been noted during cognitive skills testing,³ via self-efficacy surveys,^4,5 by observation of real-life interactions,^6,7 or during simulated scenarios using low- and high-fidelity (HF) mannequins.^8,9

Simulation technology is a powerful tool for the education of physicians and other health care professionals because it complements real patient-care experiences.¹⁰ Furthermore, advanced technology (eg, HF mannequins and video recordings) allows placement of trainees in a situation that mimics the real-life environment without compromising patient safety.⁹ Recently, the American Academy of Pediatrics and the American Heart Association modernized NRP courses and included simulation in the methods used to educate and assess technical aspects of NR.^11,12 We recently evaluated procedural skills and team performance of pediatric residents with NR using a HF mannequin during their neonatal ICU (NICU) delivery room (DR) rotation.¹³ Recognizing important gaps in skills performance, particularly in timeliness, we speculated that a different educational experience such as deliberate practice may be effective.^10,14

Objective

Our aim was to ascertain if a deliberate practice intervention improves procedural skills and team performance of pediatric residents with simulated NR during their monthly NICU-DR rotation using a HF mannequin.

Methods

Each month, 2 teams participated in two 90-minute sessions (2 to 3 weeks apart) in an off-site simulated delivery room. Each team consisted of 2 residents and 1 intern (general pediatrics and/or medicine-pediatrics). Each session included an introduction to simulation, a brief skills practice, and review of the video “Just a routine operation” http://vimeo.com/970665, which highlights gaps in team performance. This was followed by a 15-minute scenario (modified from SimNewB NRP 2010), which involved a NR of a 37-week GA infant HF mannequin SimNewB (www.laerdal.com) born by cesarean delivery to a preeclamptic woman with placental abruption. APGAR scores were 1 at 1 and 5 minutes. NR required proficiency in 5 skills: positive pressure ventilation (PPV), chest compressions (CC), endotracheal intubation (INT), umbilical vein catheterization (UVC), and intratracheal (ETepi) and intravenous (IVepi) administration of epinephrine. PPV was provided via a flow-inflating bag connected to a pressure gauge and a manometer. Pulse oximetry was available. By design, there was no heart rate response ≥60 bpm to ventilation and ETepi until the correct dosage of IVepi had been given through the UVC. Teams were expected to perform without instructors’ guidance or redirection. Video recording was used for a 20- to 30-minute debriefing and for independent scoring by a neonatologist, a neonatal fellow, and a neonatal nurse practitioner. Each of the 5 skills included 4 technical elements, given 1 point each if performed correctly (Table 1). A total score ≥3 (≥75% of 4 points) for each individual skill and a cumulative score ≥15 (≥75% of 20 points) were considered acceptable.

Table 1.

Procedural Skills Were Scored on 4 Items, Each Valued 1 if Correctly Done.

Positive Pressure Ventilation	Chest Compressions	Endotracheal Intubation	Epinephrine Administration	Umbilical Vein Catheterization
Dry/Stimulation	Technique	Technique	Correct dose	Preparation
Suction	Count	HR/RR evaluation	Intratracheal	Umbilical tape
HR/RR evaluation	Bagging synchrony	Reposition ETT	Intravenous	Technique
Bag/Mask ventilation	HR/RR evaluation	Secure ETT	HR/RR evaluation	Secure line
Total = 4	Total = 4	Total = 4	Total = 4	Total = 4

Abbreviations: HR, heart rate; RR, respiratory rate; ETT, endotracheal tube.

Each procedure was expected to start or be completed within a predetermined time. Initiation of PPV (≤60 s) and CC (≤120 s if preceded by bag/mask ventilation or ≤180 s if preceded by INT) were derived from NRP guidelines.¹¹ The maximum time for INT (as documented by end-tidal PCO₂ color change) was 120 s, or 180 s if preceded by CC. This limit was selected from the data of successful INTs in real life or in simulated NR.^7,8 ETepi administration was expected by 300 s, UVC placement by 420 s, and IVepi by 480 s. These time expectations were in line with those observed in real-life NR.¹⁵ Each individual skill was scored for timeliness (0 if not done, 1 if done late, and 2 if done on time); a total cumulative score for timeliness ≥9 (≥75% of 12 points) was considered acceptable.

Team behaviors were evaluated by a method modified from Thomas et al^16,17 that consisted of 3 domains: communication (information sharing and inquiry), management (workload and vigilance), and leadership. Each behavior within the first 2 domains as well as the third domain (leadership) was scored (0-4). A total score ≥3 (≥75% of 4 points) for each individual team behavior and a cumulative score ≥15 (≥75% of 20 points) were considered acceptable.

Between sessions, all residents participated at the NICU in a single deliberate practice (1.5 to 2 hours) intervention using a low-fidelity mannequin (www.laerdal.com). This session focused on procedural skills used during a delivery room NR and required individual repetitive practice with immediate feedback by an NRP instructor (RH). The institutional review board exempted this project from review because it was an educational intervention study. Verbal consent from each participating resident was obtained.

Statistical Analysis

Procedural skills and timeliness and behavioral scores were expressed as medians and compared with a Wilcoxon signed rank test. Acceptable performance scores were compared using McNemar’s χ² test with correction for continuity. Interrater reliability measured by Krippendorff’s κ coefficients¹⁸ for skills were as follows: PPV, κ = 0.26; INT, κ = 0.78; CC, κ = 0.48; ETepi and IVepi, κ = 0.65; UVC, κ = 0.63. For team behaviors, they were as follows: communication, κ = 0.37; management, κ = 0.40; and leadership, κ = 0.57.

Results

In all, 11 teams (10 Pediatric Level [PL]-1, 19 PL-2, and 4 ≥PL-3) completed the 2 sessions and deliberate practice. Simulated NR was always led by senior residents. During the 2- to 3-week interval between sessions, residents increased their clinical experience by attending approximately 50 low-risk and 40 high-risk deliveries. In our institution, all “real-life” NR are led by the neonatologist or neonatology fellow.

Procedural Skills: Technical Aspects

Median PPV scores were 3 during the first and 3 (P = .42) during the second session. Drying/stimulation, suction, and evaluation of heart rate and respiratory rate were consistently done by all teams. Bag/mask ventilation, however, was incorrectly performed by some teams because of their inability to maintain bag inflation by not controlling the continuous positive airway pressure valve or the oxygen flow. None of these performance gaps reoccurred during the second session.

Median INT scores were 2 during the first and 4 (P = .01) during the second session. Good technique for INT was demonstrated by most teams; however, half of them needed more than 2 attempts. During the first session, securing the tube with a NeoBar (Neotech Products Inc, Valencia, CA) was not commonly done; thus, operators kept their hands on the tube, limiting their participation in other tasks. This error did not reoccur during the second session.

Median CC scores were 3 during the first and 4 (P = .05) during the second session. Although technique was adequate, bagging was often asynchronous with the CC count. Team leaders seldom recognized fatigue and, thus, failed to reassign tasks. During the second session, these problems did not reoccur.

Median ETepi and IVepi scores were 3 during the first and 4 (P = .03) during the second session. During the first session, only half of the teams knew the correct doses. Most residents did not know how to draw epinephrine, operate the 3-way connectors, or flush epinephrine with saline. These errors did not reoccur during the second session.

Median UVC placement scores were 2 during the first and 3 (P < .01) during the second session. Only 4 of the teams placed umbilical tape around the cord to control bleeding, and only 6 teams withdrew “blood” from the UVC to evaluate placement. In all, 7 of the teams placed a UVC without saline, thus injecting air into the mannequin, and only 1 of the 11 teams secured the UVC. These performance gaps did not reoccur during the second session. Median combined scores for all procedural skills improved between sessions (14 and 18, P < .01).

Procedural Skills: Timeliness

As reflected by their median scores (2 and 2), PPV was started in a timely manner by most teams during the first and second sessions. Conversely, the other tasks were started or completed outside expectations (1 and 1 for CC, 1 and 1 for INT, 1 and 1 for ETepi, 1 and 1 for UVC, and 0 and 1 for IVepi). Median combined scores for all procedures did not change from the first to the second session (8 and 8). None of the differences in individual or combined scores was statistically significant.

Team Behavior Scores

Median communication and management scores improved from the first to the second session (2 to 4 and 3 to 4, P < .01, respectively). During the first session, distribution of tasks without consideration to the capabilities of each resident was common. Vigilance was often compromised because leaders allowed team members to focus only on their assignments while ignoring their potential contributions to other tasks. These performance gaps occurred less frequently during the second session. Most team leaders still did not seem to be aware that difficulties with procedural skills significantly affected timeliness for completion of tasks. Median leadership scores (3 and 4, P < .01) and median combined team scores (8 and 10, P < .01) improved from the first to the second session.

Acceptable Performance Scores

The number of teams with acceptable performance (individual and combined) for technical skills, timeliness, and team behaviors are presented in Table 2. There were definite improvements in the scores for procedural skills as well as for team behaviors; however, no improvement in the timeliness for initiation or completion of the procedural skills was observed.

Table 2.

Team Acceptable Performances.

	First Session	Second Session
Technical Skills	No. (%)	No. (%)	P Value^a
Positive pressure ventilation	7 (64)	10 (91)	.25
Chest compressions	6 (55)	10 (91)	.13
Intubation	4 (36)	10 (91)	.04
Epinephrine administration	7 (64)	11 (100)	.13
Umbilical vein catheterization	4 (36)	10 (91)	.04
Combined	4 (36)	10 (91)	.04
Timeliness
Positive pressure ventilation	11 (100)	9 (82)	.48
Chest compressions	7 (64)	6 (55)	1.00
Intubation	4 (36)	6 (55)	.72
Intratracheal epinephrine	4 (36)	4 (36)	1.00
Umbilical vein catheterization	1 (9)	1 (9)	1.00
Intravenous epinephrine	1 (9)	1 (9)	1.00
Combined	5 (45)	5 (45)	1.00
Team behaviors
Communication	5 (45)	11 (100)	.04
Management	8 (73)	11 (100)	.25
Leadership	6 (55)	11 (100)	.07
Combined	3 (27)	11 (100)	.01

McNemar’s χ² test with correction for continuity.

Discussion

Data from the present investigation showed that the addition of deliberate practice to our simulation training decreased performance gaps. Our intervention incorporated elements of deliberate practice, such as motivated learners, well-defined objectives, difficulty commensurate with residents’ level of training, and focused repetitive practice with immediate feedback provided by experienced trainers.¹⁰ Similar improvements in NR procedural skills of pediatric and internal medicine residents following multiple deliberate practice sessions given over extended periods have been previously reported.^4,19,20

Germaine to our study, a DR-based deliberate practice given twice to pediatric residents during their NICU rotation also resulted in improved performance.¹⁴ NR training done during the NICU-DR rotation assumes motivated learners and minimizes the time subtracted from their busy schedules. The benefit of NR simulation-based medical education with deliberate practice has now been accepted.¹⁰ Whether these gains in performance can be sustained over time or be translated consistently to practice is unclear.^10,21

Mastery of procedural skills during NR, albeit ideal, may be clinically meaningless unless the tasks in question are performed in a timely manner; thus, our assessment of proficiency included an evaluation of their timeliness. Using a low-fidelity mannequin for NR training, White et al⁸ reported time to completion of some skills, but no conclusions were drawn because of lack of standards for comparison. Later, Nadel et al⁴ provided a group of third-year pediatric residents with 3 mock resuscitations with immediate feedback over an extended period and noted that the learners performed many of the critical elements of cardiopulmonary resuscitation more quickly.

A trend toward quicker task performances during simulated NR using video debriefing as compared with oral debriefing has also been reported.⁹ Their data, however, could not be used as a measure of proficiency because during the performance of the skill, the instructors redirected their trainees several times.⁹ Significant delays in starting CCs and defibrillation by pediatric residents using a HF model of cardiopulmonary resuscitation highlighted the importance of timeliness.²²

Sawyer et al,²⁰ who also studied NR residents’ performance following 3 simulations with deliberate practice, given over a 9-month period, reported that intervals for most procedures did not change except for a decline in the time needed to secure vascular access and to start IV medications. Our data showed that although most PPV was started in a timely manner, the significant delays in task initiation (CC) or task completion (INT, UVC placement, and IVepi) observed during the first session continued in spite of the additional deliberate practice intervention. Conceivably, too much focus on the correct performance of procedural tasks may have the unintended consequence of insufficient attention to timeliness. More likely, improvement in timeliness may require repeated sessions over a prolonged period.^4,19,20 This option may not be easily exercised considering the difficulties of assembling teams of residents for multiple simulation sessions during a prolonged and intense period of clinical activities.²³ In addition, the residents’ opportunities to lead or to assume major responsibilities during “real-life” NR are limited by the restrictive duty hours as well as by the shifting of responsibilities among NR teams toward more experienced providers (ie, neonatologists, neonatal fellows, and NNPs).^24-26

Team behaviors can be assessed during simulated NR scenarios with the methodology described above.^16,17 Video recording facilitated debriefing and evaluation of individual as well as team behaviors and technical skills.⁶ In our previous study, we observed that 1 simulation session and 2 to 3 weeks of NICU-DR clinical experience resulted in an improvement in team communication only.¹³ The addition of a single deliberate practice session to the current investigation led to definite improvements, not only in communication but also in management and leadership performances. It is possible that after the residents (especially the leader) mastered the procedural skills, they were able to focus more on team dynamics than on individual tasks.

The importance of team training in preparation for NR has been recognized as critical to success.^6,16,27,28 Videotaped recordings of “real-life” NR highlighted opportunities for improving team and leader performances and outcomes.^6,29 Simulation-based learning provides an ideal opportunity for training health care teams for critical interventions (eg, NR) without compromising patients’ safety.^22,30 Recognizing the need for team training, NRP leaders have updated current NR guidelines to address the importance of teamwork and communication through simulation-based learning and debriefing.¹² Our institution, like others, has incorporated mandatory crew resource management courses in the training and credentialing of all faculty and house staff.³¹

One limitation of our study was that it did not allow for long-term evaluation of retained knowledge or skills. Also, a post hoc analysis showed that the small sample size had sufficient power to detect only large effects, and this would be further diminished if adjustments for multiple outcomes were made. Finally, it is possible that residents’ performance during a simulated NR may not reflect their reactions during real-life situations, where other providers with different degrees of expertise participate. Despite these limitations, the strength of this study is that even a single deliberate practice intervention during the NICU-DR rotation can correct procedural performance gaps while improving team behaviors, at least for the short term. Also, this study highlights the need for incorporating timeliness of procedures and team behaviors in the assessment of readiness and competency in NR.

Conclusion

HF simulation with limited deliberate practice intervention results in better procedural skills and team performance; however, lack of improvement in timeliness of tasks may require more sessions for correction. The restricted time available for such interventions during clinical rotations limits the opportunities for further NR training and assessment.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Residency Education in Pediatrics. Chicago, IL: Accreditation Council for Graduate Medical Education; 2007.

Bismilla

Finan

McNamara

LeBlanc

Jefferies

Whyte

. Failure of pediatric and neonatal trainees to meet Canadian neonatal resuscitation program standards for neonatal intubation. J Perinatol. 2010;30:182-187.

Cusack

Fawke

. Neonatal resuscitation: are your trainees performing as you think they are? A retrospective review of a structured resuscitation assessment for neonatal medical trainees over an 8-year period. Arch Dis Child Fetal Neonatal Ed. 2012;97:F246-F248.

Nadel

LaVelle

Fein

Giardino

Decker

Durbin

. Assessing pediatric senior residents’ training in resuscitation: fund of knowledge, technical skills, and perception of confidence. Pediatr Emerg Care. 2000;16:73-76.

Wood

Jones

Jr Wood

Pan

Parker

. Neonatal resuscitation skills among pediatricians and family physicians: Is residency training preparing for post residency practice? J Grad Med Educ. 2011;3:475-480.

Carbine

Finer

Knodel

Rich

. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics. 2000;106:654-658.

O’Donnell

CPF

Kamlin COF Davis

Morley

. Endotracheal intubation attempts during neonatal resuscitation: success rates, duration, and adverse effects. Pediatrics. 2006;117:e17-e21.

White

JRM

Shugerman

Brownlee

Quan

. Performance of advanced resuscitation skills by pediatric housestaff. Arch Pediatr Adolesc Med. 1998;152:1232-1235.

Campbell

Barozzino

Farrugia

Sgro

. High-fidelity simulation in neonatal resuscitation. Paediatr Child Health. 2009;14:19-23.

10.

McGaghie

Issenberg

Cohen

Barsuk

Wayne

. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86:706-711.

11.

Perlman

Wyllie

Kattwinkel

. Part 11: Neonatal Resuscitation: 2010 International Consensus on cardiopulmonary resuscitation and emergency cardiovascular care science with treatment recommendations. Circulation. 2010;122:S516-S538.

12.

Zaichkin

Weiner

. Neonatal resuscitation program (NRP) 2011: new science, new strategies. Neonatal Netw. 2011;30:5-10.

13.

Cordero

Hart

Hardin

Mahan

Giannone

Nankervis

. Pediatric residents’ preparedness for neonatal resuscitation assessed using high fidelity simulation. J Grad Med Educ. In Press.

14.

Barry

Gibbs

Rosenberg

. A delivery room-focused education and deliberate practice can improve pediatric resident resuscitation training. J Perinatol. 2012;32:920-926.

15.

Sparks

Jaleel

Christie

Brion

Wyckoff

. Impact of changes in the 2006 neonatal resuscitation program (NRP) recommendations for delivery room epinephrine use. www.abstracts2view.com/pasall/view.php?nu=PAS11L1_2502

16.

Thomas

Sexton

Lasky

Helmreich

Crandell

Tyson

. Teamwork and quality during neonatal care in the delivery room. J Perinatol. 2006;26:163.

17.

Thomas

Taggart

Crandell

. Teaching teamwork during the neonatal resuscitation program: a randomized trial. J Perinatol. 2007;27:409-414.

18.

Freelon

. ReCal: intercoder reliability calculation as a web service. Int J Internet Sci. 2010;5:20-33.

19.

Wayne

Butter

Siddall

. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21:251-256.

20.

Sawyer

Sierocka-Castaneda

Chan

Berg

Lustik

Thompson

. Deliberate practice using simulation improves neonatal resuscitation performance. Simul Healthc. 2011;6:327-336.

21.

Finan

Bismilla

Whyte

LeBlanc

McNamara

. High-fidelity simulator technology may not be superior to traditional low-fidelity equipment for neonatal resuscitation training. J Perinatol. 2012;32:287-292.

22.

Hunt

Vera

Diener-West

. Delays and errors in cardiopulmonary resuscitation and defibrillation by pediatric residents during simulated cardiopulmonary arrests. Resuscitation. 2009;80:819-825.

23.

Antiel

Thompson

Hafferty

. Duty hour recommendations and implications for meeting the ACGME core competencies: views of residency directors. Mayo Clin Proc. 2011;86:185-191.

24.

Lee

Chitkara

Halamek

Hintz

. A national survey of pediatric residents and delivery room training experience. J Pediatr. 2010;157:158-161.

25.

Gozzo

Cummings

Chapman

Bizzarro

Mercurio

. Who is performing medical procedures in the neonatal intensive care unit? J Perinatol. 2011;31:206-211.

26.

Downes

Narendran

Meinzen-Derr

McClanahan

Akinbi

. The lost art of intubation: assessing opportunities for residents to perform neonatal intubation. J Perinatol. 2012;32:927-932.

27.

Joint Commission. Sentinel event alert. Issue 30: preventing infant death and injury during delivery. http://www.jointcommission.org/sentinel_event_alert_issue_30_preventing_infant_death_and_injury_during_delivery/. Accessed April 20, 2013.

28.

Hunt

Patel

Vera

Shaffner

Pronovost

. Survey of pediatric resident experiences with resuscitation training and attendance at actual cardiopulmonary arrests. Pediatr Crit Care Med. 2009;10:6-105.

29.

Finer

Rich

. Neonatal resuscitation for the preterm infant: evidence versus practice. J Perinatol. 2010;30:S57-S66.

30.

Hunt

Shilkofski

Stavroudis

Nelson

. Simulation: translation to improved team performance. Anesthesiol Clin. 2007;25:301-319.

31.

Haerkens

MHTM

Jenkins

Van der Hoeven

. Crew resource management in the ICU: the need for culture change. Ann Intensive Care. 2012;2:39.