Comparison of On-Campus and Hybrid Student Outcomes in Occupational Therapy Doctoral Education

Terminology

Allen and Seaman (2006) identified four primary delivery models based on the percentage of the program delivered by technology (Table 1):

Student Learning Outcomes

In many studies that have compared undergraduate and graduate students in traditional versus online courses, the outcomes are often measured by student and faculty self-reports, test grades, and final course grades (e.g., Allen et al., 2006; Clark, 2002; Farber, 2013; Fredda, 2000; Hong, 2002; Johnston, 2008; Means et al., 2009; Neuhauser, 2002; Phye, 1997; Richardson, 2004; Van Schaik, Barker, & Beckstrand, 2003; Weiss, Schreuer, Jermias-Cohen, & Josman, 2004). Collectively, these studies’ findings revealed no significant difference in learning outcomes between students who were enrolled in traditional and online programs. The no-significant-difference phenomenon, Russell (1999) and other researchers have argued, may be attributed to lack of control for extraneous variables such as students’ existing knowledge and academic achievement and different delivery modes being delivered by different instructors (Hollis & Madill, 2006; Russell, 1999). Overall, however, these outcomes research studies have agreed with academic leaders’ beliefs by finding no significant differences between the effectiveness of traditional instruction and instruction using Internet technology (Allen & Seaman, 2013).

Online Education in Occupational Therapy

Distance education is commonly used in occupational therapy education. According to the Academic Year 2011–2012 AOTA Academic Programs Annual Data Report (American Occupational Therapy Association [AOTA], 2012), 50% of doctoral-level programs and 60% of master’s-level programs in occupational therapy and 56% of occupational therapy assistant programs reported offering some type of distance education to their students. To maintain and ensure the high quality of occupational therapy, educators in occupational therapy, as is the case for those in other professions, face the issue of how to replicate what happens in face-to-face courses in an online environment (Teeley, 2007).

Similar to other research studies, student learning outcomes in occupational therapy have been examined using student and faculty satisfaction and grades in didactic courses. Additionally, researchers in occupational therapy have also examined critical reasoning of occupational therapy students enrolled in online or distance education programs (Doyle & Jacobs, 2013; Farber, 2013, Mitchell & Batorski, 2009; Richardson, 2004; Simons, Baron, Knicely, & Richardson, 2002). In spite of the increased research studies on student learning outcomes, the literature on overall graduate outcomes in occupational therapy between on-campus and hybrid programs is still limited.

The purpose of this study was to examine and compare graduate outcomes between a traditional and a hybrid entry-level doctorate of occupational therapy (OTD) program. Specifically, we compared students’ grade-point average (GPA) at the end of each academic year, cumulative GPA on graduation, Level 2 Fieldwork Performance Evaluation (FWPE), grade on the National Board for Certification in Occupational Therapy (NBCOT) practice exam, and actual NBCOT pass rate.

Context of the Study

Participants in the study were enrolled in an entry-level OTD program at the urban campus of a private university in the Midwest. This university was founded in 1878 and consists of eight colleges and schools with more than 50 undergraduate and 20 graduate programs. This institution has a history of successful development of distance and professional doctoral education. In 1999, the university offered the first entry-level OTD program in the United States in response to professional and health care trends and to meet the needs of society (Runyon, Aitken, & Stohs, 1994). Currently, the entry-level OTD program is offered in 3 yr across eight semesters. The OTD curriculum emphasizes entry-level occupational therapy competencies, with a focus on occupation, professional identity, and practice; Ignatian values; and leadership. The detailed OTD curriculum can be found in the Appendix.

To address the shortage of occupational therapists in an underserved state, the university partnered with a public institution in that state to deliver a hybrid entry-level OTD program to students residing in the state. The hybrid entry-level program launched in fall 2008 and is a complete replication of the traditional program offered on campus at the university. A lecture capture system is used to record all lectures delivered on campus by faculty and to link lecture files for the hybrid student cohort to view and study. The students in the hybrid cohort interact with instructors and other students through the learning management system and other approaches such as email, Skype, and web conferencing. Courses that require laboratories are taught to the hybrid cohort by adjunct occupational therapy faculty at the partnering university in the state in which the students reside. All adjunct faculty are experienced practitioners or educators in occupational therapy and are employed by the university. Before instructing the lab components of the courses, all adjunct faculty completed a rigorous orientation and training pertaining to instructional methods, online teaching and learning, evaluation and assessment of student learning, and classroom management.

Method

This study used a retrospective, between-groups comparison method to examine whether a significant difference occurred between the graduate outcomes of on-campus and hybrid programs. Before the study, approval was obtained from the institutional review board of the university at which the investigators were employed.

Study participants were entry-level OTD students who graduated in 2011 and 2012 from both the traditional and hybrid programs. A total of 81 students from the traditional program and 13 students from the hybrid program took part in the study. The number, gender, and mean age of the participants from each cohort are displayed in Table 2.

The academic performance of the participating students was collected, including student GPA at the end of each academic year, overall cumulative GPA, Level 2 fieldwork performance, grade on the NBCOT practice exam, and final NBCOT pass rate. In occupational therapy education, the Accreditation Council for Occupational Therapy Education requires students to complete two 12-wk hands-on fieldwork assignments during their education and training. Students’ performance in Level 2 fieldwork is measured with the FWPE (AOTA, 2002).

The NBCOT test is a standardized assessment. Currently, the NBCOT test consists of two test item formats: multiple-choice questions and clinical simulation scenarios. In addition to the actual NBCOT test, NBCOT also offers online practice examinations. Graduates of occupational therapy programs can take the practice exam before taking the actual test. Before graduation, all students in the university’s occupational therapy program are required to take the NBCOT practice exam. Data from the NBCOT practice exam consisted of scores from the online multiple-choice and clinical simulation tests. An independent t test was used to analyze whether a significant difference occurred in the variables of interest. The significance level was set at .05.

Results

Results of the statistical analyses of the collected data are summarized in Table 3. As illustrated in Table 3, no significant differences were found in student 1st-yr GPA, overall cumulative GPA, first- and second-level fieldwork performance, grades on the NBCOT practice exam multiple-choice and clinical simulation sections, and overall NBCOT pass rates. Although the difference in cumulative GPA is close to significant, it did not reach a statistically significant level. A significant difference, however, was found in end-of-1st-yr and end-of-2nd-yr student GPA, with the traditional program students having a higher mean GPA.

Discussion

The purpose of this study was to examine and compare graduate outcomes of entry-level OTD on-campus and hybrid student cohorts. In general, study findings revealed no significant differences in students’ overall GPA, Level 2 FWPE, NBCOT practice tests (online multiple-choice and clinical simulation tests), and actual final NBCOT certification exam pass rate. Overall, the study findings suggest that the graduate outcomes of occupational therapy students in traditional and hybrid programs are comparable or the same. Such comparability and sameness exist in both knowledge as measured by didactic coursework and the NBCOT practice exam and the psychomotor dimensions as measured by laboratory course grades and FWPE, indicating that hybrid education is an effective alternative for occupational therapy professional education and training. Although caution is advised, the study findings may be applicable to other health care professions.

The significant findings regarding end-of-2nd-yr and end-of-3rd-yr GPAs are surprising and contradictory to the results for the other variables of interest. Whether such significant findings are due to the actual student performance outcomes or other extraneous variables is unknown. The significant finding may be caused by Type 2 error, given the small and disproportionate sample of the study. Future research studies are warranted to further examine such significant findings.

Caution needs to be taken when interpreting the study findings because of several limitations. First, the number of the participants from the hybrid program was small and disproportionate. For this reason, global generalizations from this study cannot be made. Moreover, with the small sample, Type 2 error may have occurred. Future studies need to include a large sample.

Second, although both knowledge and skill and psychomotor dimensions of the graduate outcomes were compared, the comparison between the two cohorts on the affective domain of Bloom’s taxonomy (Anderson et al., 2000) needs to be examined in future studies. Qualitative data need to be collected or qualitative research conducted to overcome such limitations. Finally, in this study only NBCOT pass rate was compared, not the actual scores obtained by graduates, due to their unavailability. Comparisons of the actual scores obtained by graduates on the NBCOT exam would be more informative and beneficial. Future research needs to explore ways to conduct such discrete comparisons.

Implications for Occupational Therapy Education and Practice

The findings of this study have the following implications for occupational therapy education: