Abstract
A core tenet of occupational therapy is that practitioners should use evidence in their practice. The American Occupational Therapy Association (AOTA; 2017) has defined evidence-based practice (EBP) as an integration of critically appraised research and clinical expertise that incorporates client’s preferences, beliefs, and values. AOTA (2020) promotes EBP by providing resources (https://www.aota.org/Practice/Researchers.aspx) and supporting efforts of researchers who contribute to evidence for the field.
Despite AOTA’s support and advocacy for EBP, many occupational therapy practitioners believe they are limited in their EBP knowledge and skills (Graham et al., 2013). However, recent changes to the Accreditation Council for Occupational Therapy Education® Standards (2018) that support moving to a clinical doctorate are designed to enhance practitioners’ EBP knowledge. Although multiple studies have focused on advancing practitioners’ EBP knowledge, the majority have examined enhancing EBP in occupational therapy practitioners, not students (see review in Myers & Lotz, 2017). Scholarly work regarding students’ development of EBP skills primarily focuses on descriptions of teaching methods and courses (Angelis et al., 2013; Cohn et al., 2014) and has little emphasis on assessing EBP knowledge (Crabtree et al., 2012; Evenson, 2013).
According to Huber and Nichols (2015), greater EBP knowledge is associated with student reports of increased confidence for conducting EBP. Therefore, promotion of EBP knowledge in occupational therapy students is one possible solution for combating deficient knowledge and lack of confidence among practitioners. In addition, development of expertise in any given content area such as EPB occurs over time, as explained in Alexander’s Model of Domain Learning (MDL; Alexander, 2003). According to the MDL, the learner’s knowledge of and interest in the content area and how he or she interacts with the content (i.e., processing strategies) function as catalysts to promote content expertise. As learners shift from surface-level strategies, such as reviewing content, to deep-processing strategies, such as forming their own ideas, greater knowledge develops.
In the MDL, learners move through three stages: (1) acclimation (orientation to the content), (2) competence (gaining greater knowledge and personal interest) and, finally, (3) proficiency or expertise (applying knowledge to solve new problems). Therefore, if occupational therapy students acquire strong EBP knowledge in school, they may feel more competent and confident using EBP in future practice. However, before longitudinally evaluating how EBP knowledge from school transfers to competent practice, we must determine whether EBP courses can effectively increase EBP knowledge.
The purpose of this descriptive, mixed-methods pilot study was to assess change in students’ EBP knowledge using the Adapted Fresno Test of Competence in Evidence-Based Practice (McCluskey & Bishop, 2009), hereinafter referred to as the Adapted Fresno Test. We hypothesized that second-year occupational therapy students would demonstrate a substantial improvement in EBP knowledge after completing a course titled Occupational Therapy 611: Reflective and Evidence-Based Practice. In addition, we explored whether students completing a master’s thesis would have significantly higher EBP baseline scores than students who elected not to complete a thesis. Finally, we posed a qualitative research question: “At the end of the course, what are students’ perspectives of their EBP knowledge?”
Method
Participants and Course
Fifty-one occupational therapy students in their first semester of the second year of a 2-yr program were enrolled in one of three total sections of Occupational Therapy 611: Reflective and Evidence-Based Practice. All three sections were conducted similarly. Seven students in the course self-selected to complete a master’s thesis (i.e., thesis students), and the remainder did not (i.e., nonthesis students).
This course is the third in a series of EBP courses designed to integrate reflective and EBP skills after students have completed Level II fieldwork. The course description states,
The focus of OT 611 is on the development of your professional identity as an occupational therapist. As tools of this development, the course emphasizes professional thinking, which requires the integration of reflective and evidence-based practice, as well as ongoing scholarly and personal explorations of the development of professional identities.
Major course outcomes include using and integrating reflection, leveraging evidence and best practice to justify practice decisions, and articulating well-supported arguments for the development of future practices.
Materials
We used the Adapted Fresno Test to assess baseline and follow-up performance. This test has two versions—Version 1 (i.e., pretest) and Version 2 (i.e., posttest)—and includes an example clinical scenario and seven test questions about the scenario with a detailed scoring rubric. Example student and practitioner responses to the seven test questions are provided in Table 1 to illustrate the scoring of the test. The highest possible score on this test is 156. Interrater reliability for the total score is excellent—Version 1 intraclass correlation coefficient [ICC] = .96, Version 2 ICC = .91—and internal consistency is acceptable, Cronbach’s α = .74.
Adapted Fresno Test Example Clinical Scenario, Test Questions, Example Student and Practitioner Responses, Example Scores, and Rationales for Scores
Note. MeSH = Medical Subject Headings; OT = occupational therapy; TBI = traumatic brain injury.
Represent responses from student participants to protect anonymity.
Practitioner responses from McCluskey and Bishop (2009).
Procedure
During the first class, Version 1 of the Adapted Fresno Test was administered by the two authors (Atler and Stephens), who were course instructors. Students completed the test independently within 30 min. After the instructors scored Version 1, students reviewed their test scores and the instructors explained the rubric. Students did not keep copies of their tests or the rubric, although viewing the rubric could have influenced their performance on Version 2.
Student performance on Version 1 was used formatively. On the basis of their performance, learning activities were added or modified to address students’ weaker EBP skills (e.g., critical appraisal, interpretation of quantitative results). For example, students repeatedly critiqued research articles using checklists and a Statistics Quick Reference Guide that the authors developed. Critiques were discussed in class, with instructors revisiting previously learned EBP knowledge. (Note that this occupational therapy program uses an integrated curriculum structure designed to link concepts and skills within and across courses to promote deeper learning and retention.) Therefore, ongoing formative assessment and learning were used throughout the course.
During the last class, students completed Version 2 of the Adapted Fresno Test and a short written survey. Students responded to open-ended questions regarding what influenced the development of their EBP knowledge, how they felt after completing Version 2, and whether they had recommendations for future courses. Because of pragmatic issues (i.e., time of semester and class schedules), survey responses were obtained from only two of the three sections. Because the data collected were part of the course, the study was deemed exempt by the university’s institutional review board.
Quantitative Data Analysis
We scored Versions 1 and 2 for the students in our respective course sections. For Version 1, to ensure consistency in scoring, each instructor scored two exams from the other section, made notes to indicate why scores were given, and met to discuss scoring decisions. For these exams, scores were modified to reflect joint scoring decisions, so the final scores for exams were identical. Subsequently, for Version 2, the instructors scored all exams from their respective sections and scored four additional exams (8.5% of the total exams) to ensure consistent scoring; no notable differences in scores were found.
Total scores from Versions 1 and 2 were used for quantitative data analysis. All data were checked for normality and homoscedasticity using the Kolmogorov–Smirnov Test of Normality (Massey, 1951) and Levene’s (1960) Test for Equality of Variances, respectively. Paired-samples t tests were used to evaluate differences in total scores between Versions 1 and 2. Independent-samples t tests were used to evaluate differences in total scores between thesis students and nonthesis students on Versions 1 and 2.
Qualitative Data Acquisition and Analysis
A qualitative content analysis method was used to analyze the open-ended written responses (Bengtsson, 2016). Specifically, a manifest analysis was performed, in which we stayed close to the students’ words to describe what they shared. All responses were entered into a Word (Microsoft Corp., Redmond, WA) document and read in entirety to obtain an overall understanding of the data. Next, we manually coded data into meaningful units using an inductive approach (Bengtsson, 2016). Sample codes included “critically appraised topic (CAT),” “checklist,” “discuss,” and “practice.” The research analyst (Atler) looked for ways to group the codes into mutually exclusive categories. For example, “cheat sheets,” “checklists,” and CAT framework were grouped into the category “helpful supports.” Strategies used to ensure rigor included use of direct quotes, return to the original data and codes to ensure all perspectives were included, and a peer review (Elo et al., 2014). Discussion and refinement of the theme labels continued until consensus was reached.
Results
Participants
Of 51 students, 47 completed both versions of the Adapted Fresno Test. Incomplete data were the result of 1 student not completing Version 1 and 3 students not completing Version 2 because of excused absences. All 47 students who took both versions finished within the allotted time, and the range of completion time was 20–30 min.
Version 1 Versus Version 2 Performance
Scores from Versions 1 and 2 were normally distributed and homoscedastic (ps > .05). In addition, there were no significant differences in Version 1 or 2 performance by students from different course sections (ps > .05); therefore, data were collapsed across course sections. As hypothesized, Version 2 scores (mean [M] = 108.28, standard deviation [SD] = 15.68) were significantly higher than Version 1 scores (M = 79.11, SD = 19.84), t(1, 46) = −8.77, p < .001, d = 1.63 (Figure 1). The students’ Version 1 mean scores were comparable to occupational therapy practitioners’ Version 1 mean scores of 78, and the students’ average improvement on Version 2, approximately 29 points, was slightly greater than that of occupational therapy practitioners, who had an average increase of 20 points. These occupational therapy practitioner scores were gathered during the development of the Adapted Fresno Test and are included for comparison with those of our student cohort (see McCluskey & Bishop, 2009, for more information about practitioners’ scores).
Students’ performance on Versions 1 and 2 of the Adapted Fresno Test.
Thesis Versus Nonthesis Students
Scores of thesis and nonthesis students were also normally distributed and homoscedastic (ps > .05). Thesis students (M = 92.57, SD = 14.23) had significantly higher scores than nonthesis students (M = 76.75, SD = 19.87) on Version 1 of the Adapted Fresno Test, t(1, 46) = 2.01, p = .05, d = 0.91 (Figure 2). However, no significant differences in scores were observed between thesis (M = 103.71, SD = 14.57) and nonthesis (M = 109.08, SD = 15.91) students on Version 2 (p = .41; see Figure 2).
Comparison of thesis (n = 7) and nonthesis (n = 40) students’ performance on Versions 1 and 2 of the Adapted Fresno Test.
Qualitative Findings
The qualitative results were organized by two questions: “What influenced your development of EBP knowledge?” and “How did you feel after completion of the posttest?” Three themes emerged from the first question: (1) helpful supports, (2) learning activities, and (3) the context of doing with others. Two themes emerged from the second question: (1) Students had a change in how they felt about their ability to engage in the EBP process after completing Occupational Therapy 611 (i.e., they felt “better,” “more comfortable,” or “more confident”), and (2) students recognized improvements in their learning.
What Influenced Your Development of Evidence-Based Practice Knowledge?
The first theme of helpful supports refers to materials (i.e., checklists and reference guides) and structures (i.e., format of the CAT) that aided or guided students through the steps of the EBP process. The following comment is representative of statements made by many students: “The checklists were also very helpful to guide me in reading research.” One student stated, “The CAT helped give me a framework of how to answer a question and a process on how to answer it.” Some students found these supports helpful, but also tiresome; for example, “Article evaluation resources—checklists, tables, CAT project . . . as tedious as they felt sometimes . . . were really helpful to organize critiques.”
Performing and being actively engaged in a variety of learning activities, the second theme, were consistently reported by all students as essential to their development of EBP knowledge. Actions included filling out critiques, doing article searches, practicing (writing) PICO (i.e., population, intervention, comparison, outcome) questions, critiquing and discussing research, and completing the CAT. One student reported that “class activities such as filling out article critiques and then having a discussion on the material helped me to understand the concepts.” A majority of the students shared how the CAT assignment influenced their knowledge. Several stated that it was because “we had to work through the process,” whereas others pointed out that they had to “apply” their skills in forming clinical questions and evaluating research articles. As one student stated, “The CAT assignment really tied all of this [critically reviewing and applying findings to practice] together.”
In the third theme, the context of doing with others, engagement in learning activities with peers and faculty, with the opportunity to receive feedback, was identified as important to students’ development. Fewer student comments addressed the context of doing with others, but this theme may have been implied in the following phrases from responses: “going over checklists,” “going through articles,” and “group discussion for skill development.” Explicit comments on the importance of working with others and obtaining feedback included “working with peers and discussing strategies with them in class,” “[conducting] peer review outside of class,” and “reading research and critiquing its validity, strength, and generalizability as a class was very helpful.” Several students acknowledged the importance of a relaxed, open environment for learning. One student shared, “I appreciated that the class was casual and nonjudgmental so we could have honest discussions, share ideas, and ask questions comfortably.”
How Did You Feel After Completion of the Posttest?
Most students who responded to this question reported a change in the first theme, how they felt about their ability to engage in the EBP process. Some students simply stated they felt better than before (i.e., when they took Version 1). Others stated that they were feeling more comfortable. One student described both quantitative and qualitative changes:
When I first took [the Adapted Fresno Test], I felt like I was just listing validity measures from research class; now I think I could actually recognize those things. I feel more comfortable searching for and applying evidence, especially creating a PICO question. I can say, “That looks promising, but it doesn’t match my population, so I won’t use it.”
In addition to creating PICO questions, other students identified increased comfort or confidence in using “terminology” and “critiquing articles.” Still other students expressed confidence in their abilities, supporting their answers with qualitative changes that had occurred. One student stated, “Before I knew the answers, but now I understand it.” Others talked about being quicker in answering the questions, “not hesitating,” or being “efficient.” Although many students noted changes in how they felt, two responses suggested that review of the grading rubric after the pretest influenced their performance on the posttest. For example, one student said,
Postcompetency: I felt a lot better. For one, I kind of remember the grading rubric so I made sure to provide multiple answers. But I also had concrete facts in my brain (PICO anatomy, how to limit search, how to assess validity) that made answers automatic and easy.
In the second theme of students recognizing improvements in their learning, a few students reflected on how the process of completing Versions 1 and 2 of the Adapted Fresno Test helped them to see changes in themselves. One student wrote, “It was helpful to see how much I learned, the precompetency test was challenging, as I didn’t get to all of the questions. The post was more encouraging.” Another student said, “I felt like I knew more than I thought I knew [after the posttest] (though I was a pretty empty vessel to begin with . . .), [and now] I feel competent to figure out/problem solve when necessary in practice.” Still another student shared, “I liked being able to identify areas of growth and competency, figure out my baseline; it showed me what I have learned.”
Discussion
The purpose of this study was to use the Adapted Fresno Test to assess change in students’ EBP knowledge. Our hypothesis that students would demonstrate substantial improvement in EBP knowledge was confirmed. Our secondary, exploratory hypothesis predicted that the subset of students completing a master’s thesis would have significantly higher Version 1 scores than their classmates not completing a thesis. This hypothesis was included because as occupational therapy education continues to evolve, we wanted to test whether learning about and practicing EBP, through the master’s thesis experience, would influence students’ understanding and use of EBP. This secondary hypothesis was also confirmed. Unexpectedly, however, thesis students did not have significantly higher Version 2 test scores, unlike their nonthesis classmates (see Figure 2). Importantly, no ceiling effects were observed in performance on Version 1 or 2, so thesis students could have obtained significantly higher scores, but they achieved only numerically higher scores.
The study findings imply that the Occupational Therapy 611 course enhanced EBP knowledge for all students and, moreover, that it may have been most beneficial for nonthesis students. In addition, the findings could indicate that thesis students had better baseline EBP knowledge, possibly because of self-selection to complete a thesis. In other words, students with greater interest in empirical research might have had more EBP knowledge and skills from previous research experience (e.g., undergraduate research experience), for which we did not control. Alternatively, it is possible that all occupational therapy students matriculated with equivalent skills, but the master’s thesis process enhanced skills for thesis students. Nonetheless, thesis students did not see significant gains in their EBP skills, which could simply represent regression to the mean or perhaps indicate that the course was insufficiently challenging for them. Because the majority of the class was composed of nonthesis students, the course likely focused on the skill limitations of those students, although thesis students may have had some overlapping skill deficits. These findings generally suggest that participating in research might enhance EBP knowledge, which could be explored by integrating more hands-on research experience in occupational therapy coursework.
Qualitative results illustrated what activities students perceived as useful in developing EBP knowledge and skills. Perhaps developing assignments based on clinical questions from students’ individual interests helped to maintain interest and facilitate motivation to stay engaged in the process. According to the MDL, learners’ interest in a topic is a critical part of students’ development (Alexander, 2003). In addition, students’ comments indicated that knowledge expanded (i.e., understanding new terminology and new concepts) and deepened. Applying the MDL to the students’ comments, we observed changes in both quantitative and qualitative results, suggesting that some students developed deep-processing strategies. As Alexander (2003) indicated, educators must guide and facilitate learning differently depending on where students are along their journey toward becoming competent evidence-based practitioners. Our findings suggest that use of best practices in teaching—engaging students in constructing knowledge and in carefully designed learning activities for formative assessment—supported their ability to learn.
Taking Version 1 may have provided students with a concrete indication of the important skills needed to engage in EBP. In addition, after taking Version 2, looking back on Version 1 to self-assess conceivably allowed students to see their improvements, which may have facilitated increased confidence. Confidence supports competence and links to one’s identity (Björklund & Svensson, 2006). Our qualitative themes support findings that confidence and knowledge are associated (Huber & Nichols, 2015). Holland et al.’s (2012) concept analysis of professional confidence stated that engaging in, reflecting on, or receiving feedback and thinking about or discussing professional expectations are intertwined and intricately related to confidence. Because students’ knowledge and confidence both affect their future use of EBP as practitioners (Huber & Nichols, 2015), both are important for educators to address. Although measuring confidence was not an objective of this pilot study, it proved important and should be addressed in future work.
Study Limitations and Future Research
This pilot study had several limitations. First, we used a convenience sample in the context of one academic program, so the results have limited generalizability. In addition, we do not have demographic information from our student participants because this information is not collected during standard class procedures or during standard administration of the Adapted Fresno Test. Second, there could be concern that instructors “taught to the test.” However, students did not keep the rubric or their exams. In addition, throughout the course, instructors used global EBP language, not exam-specific language. Third, the absence of a follow-up assessment beyond the end of the course restricted our ability to measure students’ maintenance and use of knowledge outside of the classroom. Therefore, future research should include assessment of classroom processes that accompany learning activities and use longitudinal designs that extend measurement into fieldwork and entry-level practice. Finally, the most important limitation of this study is that the Adapted Fresno Test does not measure how students use evidence for decision making and client collaboration—an essential skill for successful EBP (Cusick & McCluskey, 2001). Arguably, this is the most difficult skill to integrate into everyday practice. One potential way to evaluate students’ use of evidence in decision making is with client case analysis assignments (Evenson, 2013).
Implications for Occupational Therapy Education and Practice
The results of this study have several implications for occupational therapy education and practice:
Although many occupational therapy educators have taken steps to improve students’ EBP knowledge, the outcomes have not been empirically evaluated; however, our pre- and posttest measures offer an approach to assess student outcomes regarding EBP knowledge.
In both teaching and practice, formative assessment, such as the Adapted Fresno Test, can be an important educational or learning tool (Rushton, 2005) because it facilitates best teaching practice by helping students identify areas of weakness for tailored improvement of their skill set.
EBP resources such as critically appraised topic templates and critical appraisal guidelines (Law & MacDermid, 2014) can provide helpful structure as both students and practitioners engage in EBP.
Conclusion
Because using EBP is a core tenet of occupational therapy, having strong EBP skills is paramount for individual practitioners and the profession at large. Previous research focused on improving EBP skills in occupational therapy practitioners, but less empirical inquiry was dedicated to evaluating and improving EBP skills in students. Here, we evaluated students before and after a reflective and evidence-based practice course. We evaluated baseline and end-of-semester performance using the Adapted Fresno Test. We found significant quantitative gains in EBP knowledge, and qualitative results suggest that students had improved confidence and competence. Although longitudinal studies evaluating EBP knowledge transfer from school to practice are necessary, this study is an important first step in measuring EBP knowledge in students and evaluating teaching methods designed to improve that knowledge.
Footnotes
Acknowledgments
The authors thank Barb Hooper for her guidance in the development of this article. They declare no conflict of interest.