Abstract
Previous research studies have found that audience response systems or “clickers” provide significant classroom, learning, and assessment benefits in higher education. However, there are few to no research studies that have pitted popular clicker systems against each other using data collected from a sample of millennial students in public health–related professions. As a result, the purpose of this study was to compare two popular audience response systems in the college classroom with undergraduate students, specifically those studying public health. The study was implemented using a mixed methods quasi-experimental crossover design with three observation points over one academic year. Both qualitative and quantitative data were collected. Findings suggest that undergraduate students prefer using a dedicated “clicker” system to respond to questions from an audience response system than a system that requires them to use their personal cell phones.
Introduction
Research studies have found that audience response systems (ARS) or “clickers” provide significant pedagogical benefits in higher education (Burkhardt & Cohen, 2012; Carnaghan, Edmonds, Lechner, & Olds, 2011; Kay & LeSage, 2009; Lantz, 2010). Classroom benefits may include improved student attendance, engagement, and motivation (Hansen, 2007). Learning benefits may include improved student interaction, discussion, learning performance, and quality of learning (Dangel & Wang, 2008). And assessment benefits may include improved student feedback, and both formative and comparative assessment of learning (Kay & LeSage, 2009). According to Caldwell (2007), clickers “typically have a benign or positive effect on student performance” and often “create a more positive and active atmosphere” (p. 9) in the classroom.
Quite simply, the problem is not deciding whether to use clickers in the classroom, but deciding which clicker system to use in the classroom. Carnaghan et al. (2011) provided an overview of four major vendors (eInstruction, iclicker, Poll Everywhere, and Turning Technologies) with educational institutions as major customers. For instance, Turning Technologies is the campus-wide clicker standard (vendor) at several universities in the American Southeast, including North Carolina State University (Hoit, 2009), as well as the University of North Carolina Charlotte, Elon University (in North Carolina), Medical University of South Carolina, and the University of Georgia.
Carnaghan et al. (2011) reported that ARS “may use either proprietary hardware that must be purchased from the SRS [ARS] vendor (e.g., receivers and response units), existing hardware (e.g., cell phones and computers), or a blend of the two” (p. 267). The researchers further explained that all systems “collect and categorize student responses to multiple-choice questions, grade or summarize the responses relative to an instructor provided answer, and provide summary feedback to the instructor regarding the class’s performance” (p. 269). A recent review of the literature conducted for this research study found that Poll Everywhere and Turning Technologies appeared to be the two mostly widely adopted vendors of ARS by institutions of higher education in the United States.
At the time this study was conducted, Poll Everywhere was a free (up to 40 participants) Internet-based system that required students to use their mobile devices (cell phones) to respond to questions embedded within a Microsoft PowerPoint presentation or the Poll Everywhere website (Shon & Smith, 2011). To respond to questions, students were required to text their responses to a telephone number that was provided by the Poll Everywhere system. After students sent the first text to the Poll Everywhere system, they received a text reply that informed them that regular texting fees applied for all messages. Any cell phone that had the capability to send text messages could work with the Poll Everywhere system. On the other hand, the TurningPoint system (developed by Turning Technologies) was a proprietary system that required students to use a unique response pad (i.e., ResponseCard RF) to respond to questions embedded within a Microsoft PowerPoint presentation (Lantz & Stawiski, 2014). In this case, students did not use their cell phones, and only used the response pads that were given to them by the instructors. Both systems provided immediate feedback to respondents, included a host of technological advantages and disadvantages, and offered potential add-on features for additional fees. For more detailed information about these ARSs, please visit the individual websites for the vendors (Poll Everywhere and Turning Technologies), or review the article written by Carnaghan et al. (2011).
Although there are a large number of research studies that provide empirical evidence for the adoption of ARS in higher education, there is a lack of published research employing mixed methodology. For example, Mareno, Bremner, and Emerson (2010) suggested that there is a need for research that combines both experimental or quasi-experimental designs with a qualitative component, such as incorporating interviews or observations that assess and report on the impact of the technology. Furthermore, there is a dearth of literature about the best or most adequate vendor of ARSs for higher education, especially related to training undergraduate students in public health–related professions. In addition, there is minimal research about timing/administration of questions in the classroom (e.g., beginning, middle, or end of class), degree of question difficulty (e.g., hard to easy), type of questions presented (e.g., true/false, multiple choice, open-ended), and effect of system implementation on instructor performance (Carnaghan et al., 2011; Lantz & Stawiski, 2014). As a result, the authors developed this research study to evaluate the adequacy of two popular systems (TurningPoint by Turning Technologies and Poll Everywhere) in an undergraduate college classroom setting. Specifically, the authors wanted to know how well certain elements of these ARSs, that is, students using their personal cell phones versus dedicated respond pads, would be received by undergraduate students studying public health–related concepts.
Method
The timeline for carrying out this research study was August 2014 to April 2015. All methods were reviewed by the institutional review board at Coastal Carolina University, though it was determined that the study was exempt because it was conducted as part of regular instruction. Coastal Carolina University is a public institution located in Conway, South Carolina, and had a total enrollment of 10,263 students at the time the study was conducted. Participants for the study were derived from students enrolled in 15 undergraduate course sections in the Public Health degree program: Foundations and Principles of Public Health (four sections), Environmental Health (three sections), Epidemiology (four sections), and Family Life and Sexuality (four sections). Participants were also derived from two course sections in the Special Education degree program: Foundations and Principles of Special Education (two sections). All students enrolled in all 17 courses sections comprised the total sample size. The large majority of participants in the study were millennial undergraduate students (Elam, Stratton, & Gibson, 2007). The classroom size ranged from about 15 to 35 students. Three public health faculty members and one special education faculty member (all tenure-track) implemented each ARS, that is, TurningPoint by Turning Technologies (with ResponseCard RF; Turning Technologies, 2016) and Poll Everywhere (Poll Everywhere, n.d.), for half a regular academic semester in a total of 17 course sections. Only multiple-choice and true/false questions were used by faculty when implementing the systems in the classroom. Students were informed by their instructors at the beginning of each semester that they would be using two different ARSs in the classroom as part of a larger research study.
Below is an illustration of the mixed method quasi-experimental crossover design that was implemented during the two regular academic semesters (Fall 2014 and Spring 2015). R represents the start of all course sections during both academic semesters. Xa represents the implementation of the TurningPoint ARS in the classroom. Xb represents the implementation of the Poll Everywhere ARS. O represents the observation points, that is, when data were collected. This efficient crossover design maximized the data available to the researchers in an effort to identify and determine the most adequate system in the classroom.
Quantitative data were collected using a 25-item, 7-domain hard-copy instrument, the Audience Response Technology Questionnaire or ART-Q (MacGeorge et al., 2008). Four open-ended items were also developed for this study. At the first observation point (mid-semester), data were collected using the ART-Q to evaluate the adequacy of the TurningPoint system in the classroom during the first half of the academic semester. After the other system (Poll Everywhere) was used for the second half of the academic semester, the same data were collected at the second observation point (end of semester). Finally, at the third observation point (end of semester), data were then collected using a seven-item instrument (two items were open-ended) to evaluate and compare the student perceived differences between the two ARSs. All students enrolled in all course sections were invited to complete the survey instruments, though all students were informed that participation was voluntary. All data collected from students were analyzed at the aggregate level. Quantitative data were then analyzed in SPSS to compare statistical differences in mean scores. Qualitative data collected from the open-ended items were analyzed using the constant comparative method of data analysis to identify embedded themes.
Results
A total of 306 student participants completed the ART-Q at the first observation point. Three-hundred and seventeen students completed the ART-Q at the second observation point. And 323 students completed the comparison survey at the third observation point. Mean scores were calculated for all items on the ART-Q for both the first and second observation points (see Table 1). The mean scores for the TurningPoint and Poll Everywhere systems were then compared using a paired samples t test in SPSS software. In addition, data from Observation 3 were also compared across both ARSs (see Table 2).
Mean Scores for All Items on ART-Q for Both First and Second Observation Points, TurningPoint (n = 306) and Poll Everywhere (n = 317).
Note. ART-Q = Audience Response Technology Questionnaire; ARS = audience response system. Each item was measured using a Likert-type scale: Strongly Agree (5), Agree (4), Neutral (3), Disagree (2), and Strongly Disagree (1). Scores that are bolded and italicized are significantly different (p < .05) than the comparison score.
Frequency of Responses for All Items at Third Observation Point, System Comparison (n = 323).
Additionally, the qualitative data were analyzed and the themes that emerged about student “dislikes” of the two systems are presented in Table 3. Additional qualitative data were collected about student “likes” of the two systems. Student likes of the Poll Everywhere system included the following: engaging/interactive (n = 37), simple to use (n = 36), a good way to review material (n = 35), anonymous (n = 17), and using cell phone (n = 11). Student likes of the TurningPoint system included the following: engaging/interactive (n = 67), good to review course material (n = 57), simple to use (n = 42), and anonymous (n = 26).
Qualitative Themes About Student Dislikes That Emerged About Both Systems, Poll Everywhere and TurningPoint (Individual Student Responses Are Provided to Further Illuminate Each Theme).
Discussion
Overall, the findings indicate, both quantitatively and qualitatively, that millennial students studying public health–related concepts favored using the TurningPoint system over the Poll Everywhere system in the college classroom. It appears that students would rather use a dedicated “clicker” to respond to questions from an ARS than their personal mobile/cell phones for multiple reasons such as poor cell phone service and distraction. These findings align with previous research studies in that public health–related students are supportive of using ARSs because they may improve their ability to learn new concepts, attend class more regularly, and prepare for required course assignments (Caldwell, 2007; Lantz, 2010; MacGeorge et al., 2008).
The simple finding that the public health–related students enjoyed the use of technology-based instruction is not surprising. Past reviews on research into clicker-integrated instruction have been published within this decade demonstrating that students enjoy the use of the “clicker” technology within the classroom setting (Caldwell, 2007; Fies & Marshall, 2006; Kay & LeSage, 2009; Lantz, 2010). Furthermore, the results from this study extend the literature in that it offers insight into what type of technology is preferred by the millennial generation. Additionally, other researchers have suggested that more rigorous study designs be used when investigating clicker technology in the classroom (Boscardin & Penuel, 2012). This research study followed that suggestion by using a mixed method quasi-experimental crossover design.
Future Directions for Research
Moving forward, future use and research with ARSs should consider focusing on the question formatting (e.g., open- vs. closed-ended questions) and implementation approaches. For example, Turning Technologies have recently released a new cloud-based system as well as new response devices (QT Device). In fact, the new response device now allows for open-ended student responses. Investigations of implementation strategies (e.g., system implementation at the beginning vs. the end of class, and using group vs. individual responses for learning) using the newer Turning Technologies’ systems (e.g., the QT vs. the RF system) are needed to identify the most effective ways to enhance teaching and learning in the classroom. Research considering implementation of those types of strategies into the college classroom is currently needed.
Future use and research may also want to focus on implementing these technologies within community-based programming designed to improve population health. Given the consistent positive findings regarding the perceived enjoyment of using ARSs among students in higher education, it would appear that the use of these technologies may also benefit students in nontraditional learning environments, such as health promotion programming at local hospitals/clinics, health departments, and nonprofit organizations. However, additional research is needed to support this assumption.
Limitations
Although this research study reached its aims, there were some unavoidable limitations. First, student participants in this study were gathered as a result of a nonprobability, or convenience sampling. Additionally, they were sampled from mainly one academic discipline (Public Health). Future researchers may consider using random sampling techniques relative to classroom selection, from a variety of academic disciplines, in an effort to further explore and contrast the results found in this study. Additionally, students who did not own a smartphone were not able to participate in this pedagogical opportunity when the Poll Everywhere system was implemented in the classroom. Finally, because of time limitations of the grant supporting this research study, a larger sample size could not be obtained.
Conclusion
Regarding the assessment of student learning, it appears that due to the findings above, the TurningPoint system may be used for things such as reviewing course material and engaging students in classroom discussions. Further-more, the systems may also be used to collect student data from tests/quizzes and class attendance. Although it was not done during this study, collecting such data using the TurningPoint system may be very doable given the students’ perceptions about the ease of use and enjoyment with using the system.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this project was provided by the Office of the Provost at Coastal Carolina University, Conway, South Carolina.