Abstract
Simulation is a widely used teaching approach that is increasingly being used in graduate-level health science programs (Bethea et al., 2014). Although various forms of simulation exist, all of them fit the following definition: “Simulation is a technique, not a technology, to replace or amplify real experiences with guided experiences, often immersive in nature, that evoke or replicate substantial aspects of the real world in a fully interactive fashion” (Gaba, 2004, p. i2). Fidelity in simulation is defined as the degree to which the simulator replicates reality (Beaubien & Baker, 2004). Simulation techniques range from low-fidelity or less realistic instruction methods such as role play with lab partners or video case studies to high-fidelity or real-life situations that include the use of standardized patients (an individual trained to act like a patient) and high-technology patient simulator mannequins that mimic basic physiological functions (Beaubien & Baker, 2004). High-fidelity simulation instruction has been found to be valuable to students because of its realism and believability (Giles et al., 2014)
The literature provides evidence that increasing the formal inclusion of evidence-based simulation methods into occupational therapy education may improve the clinical readiness of entry-level occupational therapy practitioners as it has done in nursing and medicine (McGaghie et al., 2010; Rutherford-Hemming & Jennrich, 2013). Available evidence suggests that simulation instruction is widely used in occupational therapy programs, and it supports the use of simulation in preparation for fieldwork (Giles et al., 2014). In a survey of 245 occupational therapy programs, 71% reported use of simulation as an instruction method, and the types of simulation being used were role play with a lab partner, computer games, video simulators, and mannequin simulators (Bethea et al., 2014). Five recent qualitative studies found that occupational therapy students recognized the value of participating in simulation experiences (Cahill, 2015; Giles et al., 2014; Herge et al., 2013; Knecht-Sabres et al., 2015; Velde et al., 2009).
The frequency of simulation instruction in health care education, including in occupational therapy programs, seems to be on the rise, although little research has been done to assess the impact of simulation instruction on the performance of health care practitioners (Bethea et al., 2014; Motola et al., 2013). Evidence is needed to support the use of simulation with manual and observable skills such as transfers and body mechanics, as well as with skills that are not directly observable, such as the application of clinical reasoning skills in challenging situations. Gathering information about the perceived impact of educational simulation experiences on clinical practice may provide positive evidence to support its continued use in educational programs.
To our knowledge, no existing instrument measures the perceived impact of simulation instruction on health care practitioners’ subsequent clinical performance. The aims of our study were to develop a survey using appropriate methodology and to use survey results to describe the perceived impact of high-fidelity, high-technology simulation experiences during occupational therapy education on occupational therapists’ clinical performance. Our research questions were as follows:
Does participation in simulation experiences during education influence practicing occupational therapists’ perception of their clinical performance skills?
Does the length of time since graduation affect the perceived impact of these simulation experiences?
On the basis of literature demonstrating that students recognize the value of simulation experiences, we hypothesized that clinical performance scores would be higher among more recent graduates.
Method
Study Design and Participants
This study included two design elements: (1) development of a survey and (2) a cross-sectional descriptive study using the survey. Participants were a convenience sample of currently practicing occupational therapists who participated in two high-fidelity, high-technology simulation experiences used to teach transfer skills while they were enrolled in the University of Pittsburgh Master of Occupational Therapy program. All study activities were approved by the University of Pittsburgh’s institutional review board.
Simulation Experiences
Participants responded to survey questions related to two high-fidelity simulation experiences provided during their prelicensure education. Information about these experiences has previously been published (Baird, Raina, Rogers, O’Donnell, & Holm, 2015).
In brief, the first simulation provided a formative assessment experience for students to practice intervention skills in a simulated acute care environment. The simulations were based on medically complex conditions that required students to practice the following essential skills: communicate with clients, use safe body mechanics, transfer clients, assess vital signs, and apply clinical reasoning. The second simulation experience provided a summative assessment of competency level in these essential skills. Both experiences used high-technology patient simulator mannequins and contained prebriefing and debriefing components. Although the focus of this study was occupational therapists’ experiences with these high-fidelity, high-technology mannequins, other forms of simulation such as role play and video-based scenarios were embedded in the program curriculum.
Instrument Development
We developed an instrument to measure the perceived impact of the simulation experiences on occupational therapists’ performance. In the development process, we considered the participants’ specific simulation experiences. Although the curriculum also exposed participants to low-technology simulations (role play, video-based clients), only the high-technology, high-fidelity simulations provided an authentic environment, were specifically labeled as simulation experiences, and were developed to address specific skills (Baird, Raina, Rogers, O’Donnell, Terhorst, & Holm, 2015).
The survey measured performance in four domains—confidence, knowledge, clinical skills, and patient safety skills—using an Essential Skills subscale (Figure 1). These four performance domains are those most widely cited in the occupational therapy simulation literature (Bethea et al., 2014, Cahill, 2015; Giles et al., 2014; Herge et al., 2013; Knecht-Sabres et al., 2015; Velde et al., 2009). The subscale consists of five items—(1) communicating with clients, (2) using safe body mechanics, (3) transferring clients, (4) assessing vital signs, and (5) applying clinical reasoning—that reflect the skills cited as essential in the occupational therapy simulation literature (Bethea et al., 2014) and that were emphasized in the participants’ simulation experiences. For each Essential Skills item, respondents were asked to rate the perceived impact of participation in the simulation experiences on their current performance using a 5-point scale: very low (1), low (2), somewhat (3), high (4), and very high (5). Eight demographic items were also included.
Survey instrument sample question.
Before the survey was distributed, a panel of four content experts reviewed it to establish the tool’s content validity. The experts were faculty from the University of Pittsburgh’s Department of Occupational Therapy who had expertise in teaching essential skills in the classroom and laboratory and in the simulation center. Experts were asked to review each item and rate the item yes (content relevant) or no (not content relevant). The experts obtained 100% agreement on all survey items, establishing evidence of content validity (Polit & Beck, 2006). The experts’ qualitative feedback on survey directions, content, and format were used to revise the survey.
Survey Distribution
Once the final draft was completed, we used the Qualtrics Insight platform (Qualtrics, Provo, UT) to build an online survey. In March 2017, a survey link was emailed to 200 alumni of the University of Pittsburgh’s Master of Occupational Therapy program who graduated between 2013 and 2016 and participated in the two high-fidelity, high-technology simulation experiences.
Statistical Analysis
All data were analyzed using IBM SPSS Statistics (Version 24.0; IBM Corporation, Armonk, NY) using a significance level of α = .05.
Item-level and subscale characteristics were examined to support the use of the survey as a measure of perceived impact. We examined the frequency of responses to determine whether the items had any ceiling effects (i.e., majority of participants responding with the highest rating) or floor effects (i.e., majority of participants responding with the lowest rating). Cronbach’s α was calculated to determine the internal consistency reliability of the survey instrument items. According to Nunnally and Bernstein (1994), Cronbach’s α values of .80 or higher indicate strong internal consistency reliability and show that items are measuring the same construct. We calculated item-level characteristics, such as the corrected item–total correlation, to examine the relationship of individual survey items and overall survey scores. Corrected item–total correlation values of .30 and higher show that items are related to the overall score (Nunnally & Bernstein, 1994). Thus, any items with a corrected item–total correlation lower than .30 were flagged for potential revision.
Next, we examined survey responses to describe the perceived impact of educational simulation experiences on occupational therapists’ current performance. Demographic information was examined using descriptive statistics. Mean scores for performance domains by graduation year were computed by averaging across scores on all essential skills in each domain. Then we performed a one-way between-groups analysis of variance (ANOVA) with Bonferroni-adjusted post hoc testing to determine whether there were differences in performance domains on the basis of year of graduation. Mean scores were also computed for each essential skill by averaging item scores within each domain. For example, a mean score for communicating with clients was calculated by averaging participant responses to that item in each of the four domains.
Results
Survey Characteristics
We found no floor or ceiling effects when we examined the frequencies for individual survey items. Strong internal consistency reliability was found when Cronbach’s α was calculated for the each of the four performance domains: confidence, α = .89; knowledge, α = .85; clinical skills, α = .89; and patient safety skills, α = .87 (Table 1). All internal consistency reliabilities exceeded the recommended threshold, indicating that items in each domain were consistently measured with the Essential Skills subscale. An examination of the corrected item–total correlation scores revealed that the five essential skills were related to the overall score and were over the recommended r = .30 level.
Internal Consistency Estimates and Descriptive Statistics for the Performance Domains
Note. M = mean; SD = standard deviation.
Survey Results
Sixty-seven responses were received and used for data analysis (33.5% response rate); 63 (94.0%) respondents were female, and 65 (97%) were White. There were no missing data. The majority of the survey responses received were from 2013 graduates (n = 19; 28.4%) and 2016 graduates (n = 19; 28.4%). When asked to report all settings in which they had practiced as a licensed occupational therapist, respondents reported that they had practiced in hospitals (non–mental health; n = 39; 58.2%), long-term care or skilled nursing facilities (n = 22; 32.8%), and freestanding outpatient centers (n = 22; 32.8%). Seven survey respondents reported that they had not completed transfers as part of their work requirements. A sensitivity analysis, or a comparison of results with and without these 7 respondents’ data, did not influence mean or internal consistency scores; therefore, their data were not excluded from the final analysis.
Impact on Performance Domain Subscales by Graduation Year.
Data were analyzed by year of graduation across all four performance domains (Table 2). The 2013 graduates reported the highest perceived impact of simulation lab instruction on performance domains. Conversely, more recent graduates from 2015 reported the lowest perceived impact. A one-way ANOVA indicated a statistically significant difference between 2013 graduates and 2015 graduates in mean confidence (p = .003) and mean patient safety skills (p = .028).
Descriptive Statistics for Performance Domains by Year of Graduation (N = 67)
Note. M = mean; SD = standard deviation.
Impact of Simulation Experiences on Performance Domain Subscale Items.
In the confidence and knowledge performance domains, respondents reported that greatest perceived impact was on the essential skill of safe body mechanics (Table 3). In the clinical skills and patient safety skills domains, survey respondents reported that greatest perceived impact was on the skill of transferring clients. Across all performance domains, participation in simulation experiences had the least impact on communicating with clients (see Table 3).
Descriptive Statistics for Essential Skills in Each Performance Domain (N = 67)
Note. M = mean; SD = standard deviation.
Discussion
Our survey was intended to measure practicing, licensed occupational therapists’ perceptions of the impact of high-fidelity, high-technology simulation experiences during occupational therapy education. Specifically, it focused on use of simulation in teaching the essential skills of communicating with clients, using safe body mechanics, transferring clients, assessing vital signs, and applying clinical reasoning in four performance domains (confidence, knowledge, clinical skills, and patient safety skills). Survey scores were examined to assess the perceived level of impact on performance domains by graduation year and for each essential skill.
Scores on the essential skill items in the four performance domains were found to have strong internal consistency reliability, indicating that the items consistently measured the perceived impact of the simulation experiences on these domains. Content validity evidence verified that the survey was a relevant measure of the perceived impact of simulation experiences. The establishment of preliminary psychometric properties for the newly developed survey provides promising evidence for its use in future studies.
Previous occupational therapy research has examined the perceived impact of simulation instruction on students and has indicated that students value the experience (Giles et al., 2014). On the basis of those findings, we hypothesized that more recent graduates would report a higher level of perceived impact. Our findings did not support this hypothesis because the more recent graduates (2015 and 2016) had the lowest mean scores. It may be that, as a result of repeated exposure to the clinical skills learned during simulation, more experienced occupational therapists reflected more on the instructional methods that contributed most to their learning. They may also be mentoring other therapists in the clinical setting.
Respondents perceived that participation in high-fidelity, high-technology simulation experiences had the greatest impact on the performance domains as they related to the ability to use safe body mechanics and to transfer clients (essential skills). This finding is likely because these two essential skills were a focus of the simulation experiences and are manual, observable essential skills that are easier to self-assess during performance in a clinical setting. Respondents consistently perceived assessing vital signs and applying clinical reasoning as the essential skills next strongly influenced by the simulation experiences.
The application of the clinical reasoning essential skill, however, is abstract and difficult to self-assess. During the simulation experience using high-technology patient simulator mannequins, vital signs were recorded on paper but not assessed on a client as they would be in a clinical setting. Therefore, this experience may not have duplicated the realism of assessing vital signs. Another finding was that the participants rated their simulation experiences as least influential on the essential skill of communicating with clients across all four performance domains. Prior research has shown that high-fidelity simulation is valuable; thus, it is possible that the participants did not find the communication component of the simulation experiences believable because they were speaking to a mannequin rather than to a live person (Lewis et al., 2012).
Limitations
This study was not without limitations. The newly developed survey focused on simulation experiences without comparing them with other educational methods. In addition, we did not investigate the direction of perceived impact (positive or negative). Thus, we can describe the level of impact, but its direction is unknown.
The survey used to measure perceived impact required participants to use retrospective recall, which may bias results. A future prospective study should consider evaluating occupational therapists’ actual performance to decrease possible sources of bias that are inherent in data from self-report surveys.
Overall, the study aimed to be descriptive in nature; however, the restriction to a single academic institution, the homogeneity of the sample, and the small sample size are all reasons for caution when drawing conclusions.
Implications for Occupational Therapy Education
Simulation education in occupational therapy is growing, although its impact on the performance of practicing occupational therapists is largely unknown. In this study, respondents perceived that participation in two high-fidelity mannequin-based simulation experiences had an impact on their performance as licensed practitioners. The results of this study had the following implications for occupational therapy education:
It is important to use an appropriate simulation method to effectively teach the skills intended. High-fidelity simulation instruction using high-technology mannequins, although still an emerging method in occupational therapy education, has been effective in nursing and medicine for manual skills training and eliminates the risk of client harm (Johnston et al., 2016; McGaghie et al., 2010). In this study, the results were inconsistent, showing a greater impact on physical or hands-on skills than on communication or the application of clinical reasoning. Further research is needed to address the impact of additional simulation techniques on the development of manual skills.
Simulation education should be accompanied by other pedagogic methods. In this study, the perceived impact of simulation experiences was not reported as high, indicating that a combination of educational approaches is necessary to properly prepare students for clinical practice. Indeed, best practice in simulation education acknowledges that learning objectives, not instructional methods, should always drive the use of simulation (Johnston et al., 2016).
Additional investigation is needed to determine the effectiveness of simulation education in preparing students to be clinically competent entry-level practitioners. The survey instrument developed in this study could be used to measure the impact of simulation experiences on the confidence, knowledge, clinical skills, and patient safety skills of graduates who are practicing, licensed occupational therapists. Emerging research in medicine has shown that simulation education based on proficiency skills training has an impact on clinical competency (Johnston et al., 2016).
Significant resources are needed to implement high-fidelity, high-technology simulation education. Occupational therapy educators should continue to investigate how simulation influences clinical competency and to compare the effectiveness of this teaching method with that of traditional teaching approaches. The value of simulation in education should be evaluated in multiple ways, including not just the benefits to the learner and practitioner but also those to the client, such as improved safety, quality of care, and clinical experience (Johnston et al., 2016).
Conclusion
To our knowledge, this study investigating the perceived impact of simulation experiences on the performance of licensed occupational therapy practitioners was the first of its kind. This study found that high-fidelity, high-technology simulation may affect clinical practice performance of occupational therapists. Further research is needed to investigate the effectiveness of specific simulation techniques compared with traditional teaching approaches, and simulation should continue to be used in conjunction with other pedagogic methods.