Abstract
Student evaluations of teaching (SETs) are utilized by universities as one component in assessing course effectiveness, despite evidence in the research regarding their validity. With the global COVID-19 pandemic, many universities rapidly transitioned teaching modalities from face-to-face to online learning, regardless of the faculty experience. This study investigates the effects on SETs of the rapid transition in teaching modalities for all sections of courses occurring during COVID-19 compared with all sections of courses taught within a Communication department at a large public research university over the past 8 years. The results indicate moderate effects from the rapid transition to online learning.
Introduction
For more than 50 years, universities have been utilizing student-reported course evaluations as a major component for assessing teaching effectiveness. Despite the development of online and hybrid (part face-to-face and part online) teaching modalities, most instructors continue to teach in traditional face-to-face format. With the onset of the global COVID-19 pandemic, in the Spring of 2020 many universities rapidly transitioned to online teaching modalities to meet social distancing protocols recommended by governmental health agencies and organizations. In some cases, universities may have suspended the use of student evaluations of teaching (SETs) as a teaching effectiveness assessment tool. However, such suspension of SETs might not be universal or continued during the duration of the pandemic.
The purpose of this study is to investigate the effect of the rapid transition to online learning on the outcome of SETs by comparing those conducted during COVID-19 against those prior to the rapid transition. To accomplish this task, the SETs of all sections of courses occurring during COVID-19 are compared with all sections of courses taught within a Communication department at a large public research university beginning in Fall 2013.
Review of Literature
COVID-19 Effects on Students
The higher education community response to the COVID-19 pandemic has spurred considerable current research on effects on students and instructors. This research has investigated institutional and individual response as well as satisfaction to the response. According to Al-Fraihat et al. (2020), factors contributing to student satisfaction of online learning include ease of use of the online platform, quality of the information provided, service quality, and instructor quality. Instructor quality was measured as including communication, response timeliness, and attitude toward the e-learning system. In an earlier study, Sun et al. (2008) found that the critical factors influencing student satisfaction with online learning include anxiety in the use of computers, online course quality, perceived ease of use, and diversity in course assessments. Their findings related to students who voluntarily chose online learning comport with recent studies of students forced into online learning as a response to the pandemic.
The COVID-19 pandemic also elicited emotional responses to the rapid transition to online learning. For example, Mendoza et al. (2021) observed elevated anxiety levels among students during COVID-19. According to Cheng (2020), the rapid transition to online learning in China due to the COVID-19 pandemic created the challenges of learning how to use online technologies and how to compensate for the lack of direct emotional communication and immediate feedback which occurs in face-to-face instruction. However, student responses to the rapid transition to online learning may also be influenced by their own digital literacy. According to Poláková and Klímová (2021), students are ready and have the necessary skills to use technology for education. In addition, Jin et al. (2021) found that students discovered the convenience of online learning as a result of the emergency response of universities to the global COVID-19 pandemic.
Webb et al. (2021) posited that the rapid transition to online learning has highlighted the challenges associated with students from disadvantaged backgrounds or with limited access or digital skills. The rapid transition may have been uneven from institution to institution or from instructor to instructor. Students had to adapt to the variances in quality as college students typically take multiple classes from multiple instructors in a given semester. Added to those variances are digital literacy differences between students and/or differences in access. According to Yang et al. (2020), student participation in online learning, specifically viewing of online video content and completion rates, dropped off dramatically as COVID-19 progressed.
COVID-19 Effects on Instructors
Limiting factors to the effectiveness of teaching during the COVID-19 pandemic include the technological skills of both students and instructors and the time needed to provide student support (Camayd & Freire, 2021). In response, students, instructors, and technical support staff worked in tandem to transition to remote-learning during the COVID-19 pandemic, which included the need to learn new software applications and ways of communicating online (Mishra et al., 2020).
According to Watermeyer et al. (2021), the rapid transition to online learning stemming from responses to COVID-19 has brought about considerable disturbance to the roles of instructors as well as to their personal lives. More specifically, the rapid transition to online learning precipitated by the COVID-19 crisis response by universities illuminated the distinction between faculty who had prior experience with online learning and those that did not (Adedoyin & Soykan, 2020).
Student Evaluations of Teaching
According to LeBlanc (2012), factors such as whether the course was a first-prep, distance learning, or tenure status of the instructor did influence the outcome of SETs for Communication courses surveyed. LeBlanc (2012) also found a positive relationship between course grades and instructor ratings for tenure-track faculty but a negative relationship between course grades and instructor ratings for nontenure-track faculty. In addition, Adams and Umbach (2012) found that lower grades in a course and out of major status contribute to nonresponse bias for SETs. Adams and Umbach also found “that survey fatigue could suppress response rates” (Adams & Umbach, 2012, p. 586). Instructor-related variables, student-related variables, and bias can each affect the outcome of SETs for a given course.
Given these findings from previous research on SETs and current research on the effects and response to the current COVID-19 pandemic, the following hypothesis is proposed:
Method
The study procedures were reviewed by the local Institutional Review Board in March 2021 (IRB # FY20-21-156) and determined that the study did not meet requirements for federally regulated research, was exempt from human subjects protections, and required no further IRB oversight. For purposes of comparison, data were selected among all available course evaluations between Fall 2013 and Fall 2020 for all current faculty in the Department of Communication. For analysis, a distinction was made between pre-COVID (courses evaluated between the data start date and Fall 2019) and post-COVID (Summer and Fall 2020) due to the University requirement that all courses be converted to fully online beginning March 2020. Spring 2020 classes were not evaluated. In total, all faculty and course evaluations from 959 courses over the period were utilized (N = 20,307).
Results
To test the hypothesis that the conversion to fully online teaching modalities as a result of COVID-19 influenced student evaluations of Communication courses, a series of independent sample t-tests were conducted. To first contextualize the results of these analyses, the average response rate across all courses for the period under investigation is 65.57% (SD = 18.44%). A small reduction in the average response rate occurred between post-COVID evaluations (N = 105, M = 54.87%, SD = 20.51%) and pre-COVID evaluations (N = 854, M = 66.89%, SD = 17.74%), t(957) = −6.44, p < .001, Cohen’s d = .181 (see Sawilowsky, 2009). For undergraduate only courses (N = 903), the average response rate was 65.59% (SD = 18.54%), and for undergraduate courses only during the years 2019 and 2020 (N = 258), the average response rate was 63.09% (SD = 18.72%). Comparison of response rates with this subset also found a small reduction (12.25%) in the average response rate occurred between post-COVID evaluations (N = 97, M = 55.45%, SD = 21.00%) and pre-COVID evaluations (2019 only) (N = 161, M = 67.70%, SD = 15.53%), t(256) = −5.36, p < .001, Cohen’s d = .178. The total number of course evaluation responses for this study (student duplication potential) was 20,307.
The first hypothesis was supported. Moderate effects were found on student evaluations of faculty between pre-COVID (N = 854, M = 4.35, SD = 0.491) and post-COVID (N = 105, M = 4.19, SD = 0.698), t(957) = −2.89, p = .004, Cohen’s d = .518. Similarly, moderately strong effects were found on student evaluations of courses between pre-COVID (N = 854, M = 4.32, SD = 0.457) and post-COVID (N = 105, M = 4.19, SD = 0.644), t(957) = −2.52, p = .012, Cohen’s d = .481.
To account for the possibility of a long-term decline in student evaluations of faculty between the years 2013 and 2018 (up 2 years prior to COVID-19), data were filtered to compare only SETs for 2019, against SETs for 2020. With this data limitation considered, the first hypothesis was supported. Again, moderate effects were found on student evaluations of faculty between pre-COVID (N = 172, M = 4.48, SD = 0.422) and post-COVID (N = 105, M = 4.19, SD = 0.698), t(275) = −4.30, p < .001, Cohen’s d = .543. Even when equal variances are not assumed (Levene’s F = 27.195, p < .001), the results of comparison hold: t(151.184) = −3.84, p < .001. In addition, limiting the comparison to the years 2019 (pre-COVID, N = 172, M = 4.45, SD = 0.395) and 2020 (post-COVID, N = 105, M = 4.19, SD = 0.644), moderately strong effects were found on student evaluations of courses: t(275) = −4.12, p < .001, Cohen’s d = .504.
When separating out graduate courses from undergraduate courses, moderately strong effects were found on student evaluations of faculty between pre-COVID (N = 806, M = 4.33, SD = 0.489) and post-COVID (N = 97, M = 4.14, SD = 0.704), t(901) = −3.39, p = .001, Cohen’s d = .516. Similarly, moderately strong effects were found on student evaluations of courses between pre-COVID (N = 806, M = 4.31, SD = 0.454) and post-COVID (N = 97, M = 4.15, SD = 0.647), t(901) = −3.12, p = .002, Cohen’s d = .478. When limiting the comparison to the years 2019 and 2020 for undergraduate courses only, the differences for both student evaluations of faculty and student evaluations of courses hold. Moderately strong effects were found on student evaluations of faculty between pre-COVID (N = 161, M = 4.46, SD = 0.421) and post-COVID (N = 97, M = 4.14, SD = 0.704), t(256) = −4.54, p < .001, Cohen’s d = .545 (see Table 1). Even when equal variances are not assumed (Levene’s F = 28.705, p < .001), the results of comparison hold: t(138.066) = −3.96, p < .001. Similarly, moderately strong effects were found on student evaluations of courses between pre-COVID (N = 161, M = 4.43, SD = 0.454) and post-COVID (N = 97, M = 4.15, SD = 0.647), t(901) = −4.43, p < .001, Cohen’s d = .503.
Comparison of 2019 (Pre-COVID) to 2020 (Post-COVID) Student Evaluations of Faculty Averages.
Interestingly, a small negative correlation was found between the years in the department and faculty evaluation mean (r = −.075, p = .019, N = 958). When including only undergraduate classes prior to 2020, the negative correlation between years in the department and faculty evaluation mean increased slightly (r = −.188, p < .001, N = 806). The effect was nonsignificant during 2020, after COVID-19 protocol online delivery of courses.
Conclusion
The conversion to fully online teaching as a result of COVID-19 protocols did significantly lower the faculty and course evaluations. However, globally, the student evaluations of faculty during the Summer and Fall 2020 semesters were still relatively high (M = 4.31, SD = 0.519; thus, 1 SD below the mean for undergraduate-only classes was 3.79 on a scale of 5.00). More specifically, forced transition to online learning may constitute a form of measurement bias in SETs as the modality of instruction may be unrelated to measures of teaching effectiveness (see Kreitzer & Sweet-Cushman, 2021). Consequently, Kreitzer and Sweet-Cushman (2021) recommend against using SETs as critical components in personnel decisions (see also LeBlanc, 2012).
Other factors that may have influenced the outcomes include motivations of both instructors and students within the context of responses to a global pandemic. With a large segment of the higher education community previously inexperienced with the benefits of online learning, being forced into the modality may have had some positive consequences. Mishra et al. (2020) demonstrated the insight that teachers and students realized regarding the relevance of time-boundness. Space-boundness is also relevant as online learning provided the opportunity for participants to consider the benefits of not being tied to a time and space for meeting classes.
In addition, Peimani and Kamalipour (2021) recommend that the current pandemic provides an opportunity for institutions of higher education to reflect on the qualities and use of traditional face-to-face teaching modalities in light of what was learned from the rapid transition to online learning. One potential opportunity for institutions of higher education presented by the rapid transition to remote learning is the possibility of attracting more international students who may not be able to travel to another country to meet their educational needs (Peimani and Kamalipour, 2021). Jin and colleagues called the rapid transition to online learning a turning point for “promoting the development of online learning” (Jin et al., 2021, p. 10).
Adedoyin and Soykan (2020) suggested that the emergency remote teaching response to the global pandemic may produce opportunities for instructors and institutions to leverage what was learned toward development of more robust online and hybrid teaching modalities. During responses to the current pandemic, Cheng (2020) recommended that instructors pay close attention to the physical and mental health of students engaged in online learning. Lessons learned could and should be applied in the future to potential health and environmental crises. Recently, universities in the state of Texas had to respond to the effects of Hurricane Harvey which flooded large portions of coastal Texas including universities in Houston area. Universities in unaffected areas were asked to temporarily serve students who were displaced. Mahmood (2021) recommends that institutions seek feedback from students to enhance student learning in remote-learning environments and to offer flexible assessment policies for students. These health and environmental crises thus provide an opportunity to consider innovative approaches to teaching and learning.
Rapanta et al. (2020) suggest that reflections on development of online learning should spur leaders of educational institutions to focus on instructional design with different teaching modalities, reconceive teacher presence, and adopt a continuous assessment model. Institutions should also attend to instructors’ job satisfaction and turnover rate. Although Roch and Montague (2021) found higher levels of job satisfaction among faculty working in virtual environments (pre-COVID-19), other factors related to the added stress of a global pandemic as well as the rapid transformation into remote teaching and learning may have affected the calculus. Specifically, Sokal et al. (2020) found that as the pandemic wore on teachers’ exhaustion and negative attitudes increased, so did their attention to classroom management and sense of accomplishment due to change in teaching.
In addition, the current pandemic and forced rapid transition to online learning also exposed limitations brought about by unequal access to digital technologies. Although faculty typically have access to fast internet backbone on campus and may be used to that benefit, both remote teaching and remote learning demonstrated the limitations of technology for many participants in online learning. Despite those limitations, the current study showed considerable adaptability among higher education community members and overall satisfaction to the response to the transition to online learning. One lesson learned for educators from this rapid transition to online learning due to the global COVID-19 pandemic is the necessity to plan ahead for course delivery modality change and perhaps design courses from the outset to provide content through multiple media.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
