Abstract
The current investigation tested the effectiveness of an online student research project designed to supplement traditional methods (e.g., lectures, discussions, and assigned readings) of teaching research methods in a large-enrollment Introduction to Psychology course. Over the course of the semester, students completed seven assignments, each representing a stage of the research process. Students formed hypotheses, tested their hypotheses using data from the class, interpreted their results, generated future directions, created PowerPoint slides summarizing their projects, and presented their results in a poster session. We found support for the hypothesis that the research methods intervention would lead to better performance on a research methods quiz compared to students in a nonintervention section taught by the same instructor. This intervention demonstrated that it is feasible to use project-oriented active-learning techniques to foster understanding of research methods in large classes.
Psychology departments and the American Psychological Association (2013) place a strong emphasis on fostering the understanding of research methods. This includes, among other things, gaining familiarity with the steps of the research process, interpreting the meaning of correlations, understanding the differences between experimental and correlational designs, and the implications of these designs for causal interpretations. It is argued that students benefit from an understanding of research methods regardless of whether they pursue a career in psychology. Indeed, the skills one develops from this understanding are beneficial outside the field of psychology because they enable people to evaluate claims made in advertisements, political campaigns, and other media and to make informed decisions regarding these claims. Thus, teaching students to apply the principles of research methods is a particularly important educational outcome for psychology majors and nonpsychology majors alike.
Students are especially likely to learn to acquire this understanding when they are actively involved in the research process (e.g., Healey & Jenkins, 2006; Jenkins & Healey, 2005; Leston-Bandeira, 2013; Piergiovanni, 2014). Although some forms of active learning can be readily incorporated into most courses (e.g., discussions and in-class activities), active involvement in research projects often requires a considerable investment of time and resources, making such techniques difficult to implement in large classes (e.g., 50–200+ students). In the present study, we created, implemented, and tested the feasibility and effectiveness of a highly structured, online active-learning program that involved students in conducting research. This program was developed for use in large introductory psychology courses.
Active Learning
Active learning has been operationalized in a number of different ways, including in-class writing activities, computer-based instruction, cooperative learning, and simulations (Sivan, Leung, Woon, & Kember, 2000). These techniques often result in increased student learning and interest (Butler, Phillmann, & Smart, 2001; Lake, 2000; Mathie et al., 1993; Schwartz & Bransford, 1998; Yoder & Hochevar, 2005). One reason why active learning may be so beneficial is that it encourages students to seek out and evaluate information for themselves, rather than having the instructor directly present all the material to them (Butler et al., 2001). Indeed, research indicates that teaching students to explain learned materials to themselves and to generate questions about such material is associated with improved comprehension (e.g., McNamara, 2004; Rosenshine, Meister, & Chapman, 1996). Our goal was to design an intervention that used active-learning principles to enhance the understanding of research methods in large classes beyond what is typically attained from in-class lectures and activities.
Teaching Research Methods Skills
Although much progress has been made in developing methods to teach undergraduates research methods skills (e.g., Bensley, Crowe, Bernhardt, Buckner, & Allman, 2010; Blessing & Blessing, 2010; Stevens & Witkow, 2014; Thieman, Clary, Olson, Dauner, & Ring, 2009), there has been little empirical work on teaching these skills in large classes. Of the empirical work conducted, efforts to teach research methods in large classes have focused primarily on in-class evaluations and discussions of past research (e.g., Blessing & Blessing, 2010; Mueller & Coon, 2013; Stevens & Witkow, 2014; Thompson & Fisher-Thompson, 2013).
Relatively few interventions to increase the understanding of research methods in courses not including a laboratory component have involved students actively participating in research. In one study, Butcher and Maunder (2014) used a longitudinal design and found that hiring undergraduates to work on pedagogic projects in partnership with faculty increased the undergraduates’ research skills and enhanced their relationship with faculty over time. Outside the field of psychology, Leston-Bandeira (2013) used a research-oriented approach to successfully teach political science students research methods over the course of two semesters. Both of these interventions were conducted at small institutions with ample resources (e.g., faculty time, funding for undergraduates, and small class sizes). Applying either of these approaches at large public universities would not be feasible because many introductory psychology courses have a large number of students. Our goal was to develop an effective intervention to foster an understanding of research methods that would be feasible to implement in large introductory courses.
Present Research
We developed a highly structured active-learning program designed to enhance existing instruction in research methods. The intervention was delivered online through our university’s course management system (Blackboard™), so that it did not require in-class time. Our intervention provided students with hands-on experience with research methods by allowing them to actively engage in a research project. Over the course of the semester, students completed seven assignments, each of which built on previous assignments. Each assignment represented a stage of the research process. Students formed hypotheses, tested their hypotheses using data from the class, interpreted their results, generated future directions, created PowerPoint slides summarizing their studies, and presented their results in a poster session.
We tested the effectiveness of the intervention using a quasi-experimental design in which an instructor with two large enrollment sections of an Introduction to Psychology course implemented the intervention in one section but not the other. However, both sections received the same in-class presentations, discussions, and assigned readings on research methods. Our primary hypothesis was that students who received our intervention would perform better on a research methods quiz at the end of the semester compared to students who had the same instructor but did not receive the intervention. We also hypothesized that students who received the intervention would perform better on the research methods quiz than students who had other Introduction to Psychology instructors who taught research methods by means of assigned readings and in-class presentations and discussions but did not receive the intervention.
Method
Participants
Participants were students from seven sections of an Introduction to Psychology course at a large state university. The two sections of primary interest were those taught by the same graduate student instructor; 202 students were in the intervention section and 147 were in the nonintervention section. Analyses for this study included scores for the 174 students (107 women, 67 men) in the intervention section and the 111 students (69 women, 42 men) in the nonintervention control section who provided consent for their data to be included in the study. The majority of consenting students in the intervention section were freshmen (74.1%), followed by sophomores (20.1%), juniors (2.3%), and seniors (2.9%). The majority of consenting students in the nonintervention control section were freshmen (59.1%), followed by sophomores (28.2%), juniors (7.3%), and seniors (5.5%).
Participants from the remaining five sections consisted of 566 consenting students (350 women, 202 men, 14 unknown gender) and were taught by five different graduate student instructors with an average class size of 128 people. Information on the year of school for participants enrolled in these five sections was not available. In accordance with departmental policies, all graduate student instructors had previously completed a two-semester teaching practicum and used the same introductory psychology textbook.
Procedure
In the first 2 weeks of the semester, no intervention occurred. During this time, all instructors used standard in-class techniques and reading assignments to introduce the field of psychology and research methods. Students in the intervention section of the course started the online intervention in the third week of the semester. Students in the nonintervention control section, taught by the same instructor, and students in the remaining five sections, taught by different instructors, did not receive the intervention. Importantly, students in all sections learned about research methods as part of the normal curriculum for the course; however, only students in the intervention section received the additional instruction in the form of the intervention.
The 11-week online intervention included a series of assignments that were spaced out across the semester. We estimate that, on average, the assignments took less than an hour per week. Detailed instructions and examples for each assignment were provided to students in order to minimize instructor/teaching assistant involvement. Students accessed these instructions, examples, and the grading information for each assignment through the university’s course management system (Blackboard). Students in the intervention section worked individually or in pairs (their choice) throughout the semester to complete a research project, which included providing data, generating hypotheses, reporting the method and results, interpreting the results, considering future directions, creating PowerPoint slides summarizing their project, and presenting the research in a poster format. Students received points toward their grade for each assignment. Graduate student teaching assistants were responsible for grading, providing feedback, and answering questions via e-mail, a discussion board, or in person (only one student requested an in-person meeting). Approximately 12 hr of graduate student time was used per week across 8 weeks of the project (no grading or feedback was needed in the first 2 weeks nor the last week of the intervention). Detailed rubrics were given to each graduate student to use when grading in order to ensure consistent grading across students and minimize the amount of time needed for grading. Finally, at the end of the semester, students in all seven sections completed a research methods quiz.
Student Assignments
Survey assignment
In the third week of classes, students were given 1 week to complete an anonymous online survey that included basic demographics and 15 self-report scales assessing the constructs introduced in the next assignment. After completing the survey, students were given a verification code that they turned in to prove that they completed the assignment, while keeping their responses anonymous. These questionnaires provided the data for students’ projects. Because the survey was anonymous and responses were used solely for the purpose of the class assignment, students were not asked to provide consent before taking the survey.
Hypothesis generation assignment
Students received a list of variables and their definitions. Each student selected two of the variables and developed a hypothesis about the relationship between them. There were four choices for one of the two variables: aggressiveness, social anxiety, need for cognition, and preference for physical attractiveness in a romantic partner. Each of these four variables had a list of potential correlates that students could choose for their second variable: aggressiveness (sensation seeking, narcissism, neuroticism, and self-esteem), social anxiety (introversion, impulsivity, self-esteem, and aggressiveness), need for cognition (introversion, impulsivity, openness to experience, happiness, and incremental view of intelligence), and preference for physical attractiveness in a romantic partner (self-esteem, narcissism, impulsivity, and happiness). Each student then hypothesized about the direction and strength of the correlation between the two chosen variables and explained why those variables could be correlated. Students were given 1 week to complete this assignment.
Method assignment
Students wrote a brief method section describing their project. The participants section included the number of participants, from where participants were obtained, and demographic information. The measures section included how students operationalized their variables (i.e., the name of the measure provided to students to measure each variable they used, the number of items, and the scale anchors). All measures were posted online to provide students with this information. The procedure section included the context in which the data were collected and the instructions survey participants received. Students were given 2 weeks to complete this assignment and the results assignment described below. Both assignments were due the same day.
Results assignment
One graduate student conducted the correlational analyses and posted the results online in a table. Students then reported the results of their projects by describing the data analysis, the resulting correlation coefficient, whether the correlation was significant, and the strength and direction of the correlation.
Interpretation assignment
Students interpreted the results by reporting whether the data supported their hypotheses for the strength and direction of the correlation. Students whose prediction was not supported indicated whether changing the method or procedure in a future replication could affect the results. In order to provide practice interpreting correlations, all students were then told to assume their correlation was significant (whether or not it actually was) and to explain three potential causal pathways linking their variables (i.e., Variable 1 causes Variable 2, Variable 2 causes Variable 1, or a third variable causes both Variables 1 and 2). Students were given 2 weeks to complete the assignment.
Future directions assignment
Students indicated why it is important for their correlational studies to be replicated and then described a potential replication with different measures, participants, or procedures. Then students were asked to propose a true experimental design that could test one causal pathway generated in the previous assignment and indicated whether the experiment would be ethical. Students were given 2 weeks to complete the assignment.
Presentation assignment
Students used their previous assignments and feedback on those assignments to create a short PowerPoint™ summary of their project. After each PowerPoint was graded online, students printed their slides to create a poster and presented it during an in-class poster session. Half of the students presented their posters to the other students for the first half of class and then students reversed roles during the second half of class. Students were given 3 weeks to complete the assignment.
Research Methods Quiz
To evaluate the effectiveness of the intervention, students in all seven course sections took a 15-item, multiple-choice research methods quiz in class during the last week of the semester (see Appendix). The quiz counted toward students’ course grades and, thus, only assessed material covered in all course sections. The items for the quiz were based on exam questions previously used by the authors to test students’ comprehension of research methods in psychology. All items were piloted for understanding and difficulty in a small sample of undergraduate students. After completing the quiz, students had the opportunity to provide consent for their grades on the quiz and previous exams to be included in data analyses.
Results
Research Methods Quiz
Four questions from the research methods quiz were excluded from the analyses due to the ceiling effects in which greater than 90% of students answered correctly. The reported quiz score means represent the total number of questions correct of the 11 questions (see Appendix).
The primary hypothesis was that students in the course section who received the intervention would perform better on the quiz than students in the section who did not receive the intervention but were taught by the same instructor. In line with our predictions, analysis of covariance revealed that students who received the intervention (M = 9.40, SD = 1.38) scored significantly higher on the research methods quiz than did students with the same instructor who did not receive the intervention (M = 9.04, SD = 1.74), F(1, 281) = 4.01, p = .05, d = .25. In this analysis, we controlled for whether the participants were freshmen and participants’ scores on items from the first exam of the semester that tested research methods. These variables are indicative of participants’ previous exposure to research methods. The same pattern of results was observed without controlling for either of these variables, t(283) = 1.97, p = .05, d = .24.
Next, we tested the prediction that students who received the intervention would perform significantly better on the quiz compared to students with different introductory psychology course instructors that did not assign the intervention. Exam scores and year in school were not available for students not taught by the instructor of the experimental section, thus the analysis used an independent samples t-test. Students in the intervention course section (M = 9.40, SD = 1.38) scored significantly higher on the research methods quiz than did students with different instructors who were in course sections that did not receive the intervention (M = 8.89, SD = 1.85), t(738) = 3.96, p < .001, d = .31. This result could be interpreted as providing additional evidence of the effectiveness of the intervention or as providing evidence that the intervention instructor was simply a more effective instructor than other instructors. To examine whether teaching effectiveness differed between the intervention instructor and nonintervention instructors, we performed a t-test comparing quiz scores of the nonintervention control section taught by the intervention instructor to the other nonintervention sections taught by different instructors, t(675) = −0.79, p = .43, d = −.08. This lack of significant difference suggests that the previous result is best interpreted as providing additional evidence of the effectiveness of the intervention.
Discussion
The current work found support for the use of an Internet-based intervention to foster understanding of research methods among undergraduates in a large introductory psychology course. Students who received the intervention scored significantly higher on a research methods quiz compared to both students with the same instructor and students with a different instructor who did not receive the intervention.
In addition to the effectiveness of the intervention, an important outcome of this study is that it demonstrated the feasibility of including a research project assignment in a large lecture course. This was accomplished by the online and highly structured nature of the project. Students were given extremely detailed examples for each assignment, and they were encouraged to model their own assignments after the examples. The goal of each assignment was to teach students about research methods by having them alter the example assignments to fit their own predictions, data, and conclusions. By taking this approach, we were able to create the detailed rubrics which graduate students used to grade each assignment, thus ensuring consistency in grading and reducing the amount of time needed to grade each assignment. An added benefit of this approach is that the intervention could be adapted for use in a distance learning course.
Despite the strengths of the present study, there were some limitations. Due to practical considerations, this study relied on a quasi-experimental design in which preexisting groups were compared. As with any quasi-experimental design, one critical interpretive issue is that it is possible that the groups we compared were not equivalent at the beginning of the study. That is, the results could reflect preexisting differences in the baseline research methods skills of the course sections rather than the effectiveness of the intervention. We think this is unlikely because we controlled for two variables that served as a proxy for baseline research methods skills: whether participants were freshmen and their scores on research methods questions from their first exam. Controlling for these variables did not alter our conclusions. Nonetheless, an ideal way to test the effectiveness of the intervention would be to conduct an experiment in which multiple sections of the course were randomly assigned to the intervention or nonintervention condition.
Another limitation is that, although the differences between intervention and nonintervention sections in quiz performance were statistically significant, they were small in magnitude. One reason why our effect sizes were small may be due to the nature of the questions on our quiz. Because the quiz counted toward students’ final grades in all sections, only material taught to all students was included on the quiz; thus, the quiz was not specific to the intervention. It also should be noted that small effects can have important consequences, particularly when considering cumulative effects over time (Cortina & Landis, 2009; Prentice & Miller, 1992; Rosenthal, 1990). The potential cumulative effects of our research methods intervention may be particularly relevant for the current sample, which consisted primarily of first- and second-year undergraduates. Future empirical work could investigate whether early direct experience with the research process in college leads to cumulative benefits. Students with early exposure to research methods, for example, may benefit when interpreting scientific work discussed in future content-specific courses. That, in turn, may further facilitate their research methods skills.
Another limitation might appear to be the number of assignments included in the intervention (seven) and the burden this might place on students. However, these assignments were spaced out over 11 weeks, and we estimated they required, on average, less than 1 hr per week. In fact, in response to a survey question, only 8.5% of the students agreed with the statement “The class research project was too much work.”
The biggest challenge to implementing this intervention is that it involved teaching assistants to grade assignments and help students use the online system. For our one section of 202 students, we used approximately 12 hr per week of graduate teaching assistant time over an 8-week period. This level of graduate assistant involvement may not always be feasible. However, it may be possible to reduce the extra resources required without diminishing the effectiveness of the intervention. First, it may be possible for the instructor to do some of the work. For control purposes in the present study, we did not do this because we did not want the instructor to have more interactions with students in the intervention section than students in the control section. Second, it may be possible to use advanced undergraduate assistants instead of graduate students. Third, because it was common for students to make similar errors, it may be possible for instructors to post a document containing scripted feedback and then refer students to the feedback that applies to them rather than providing individual feedback to each student. Fourth, another approach would be to adapt each assignment to contain only multiple-choice questions. Future research could explore these various methods for making this intervention even easier to implement.
Conclusion
Research methods skills are essential for students to develop, as they are related to academic and professional success and enable students to be educated consumers of information (e.g., Beasley & Cao, 2014; Lawson, 1999; National Association of Colleges and Employers, 2013; Paul & Binker, 1990). Although there has been some research investigating techniques for developing research methods skills (e.g., Bensley et al., 2010; Blessing & Blessing, 2010; Butcher & Maunder, 2014), these techniques have almost exclusively been tested in small classes or at universities with an abundance of resources. The current work adds to the literature by demonstrating the effectiveness of a research methods intervention in a large introductory psychology course taught at a state university.
Footnotes
Appendix
Author’s note
The author Lauren A. Holland-Carter is now affiliated to Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors received financial support from Florida State University for the research performed in this article. However, no financial support was received for the authorship or publication of this article.