The Strength of Mixed Research Methods for the Assessment of Learning Communities

At Georgia State, combining quantitative and qualitative techniques provides greater insight than using either method alone.

MORE AND MORE CAMPUSES are adopting freshman learning communities (FLCs) in order to boost retention. FLCs take many forms, but they typically involve enrolling small groups of students in two or more classes together during their first semester in order to create community and promote academic and social involvement. At our institution, a large public research university located in an urban area in the southeast, nine years of quantitative data on first-year student cohorts clearly demonstrate that FLC participants do better academically than non-FLC members. We share these quantitative outcomes internally each year and have found them crucial to garnering support from key campus stakeholders. We have also disseminated these data nationally; generally, they corroborate previous research on the effectiveness of learning communities. As Gary Pike noted in a recent Assessment Matters article, learning communities are increasingly popular because “they work” (p. 30).

In addition to improving academic performance, FLCs have other positive effects on students: lower risk of course withdrawal, increased cognitive skills and abilities, and higher overall satisfaction with college. Evidence also suggests that learning communities succeed across a range of institutional types and students. But while these documented outcomes of FLCs are valuable, we believe that the body of research on learning communities is limited by its focus on quantitative indicators. Pike supports our position, encouraging researchers to go beyond “identifying student learning outcomes to explaining and significantly improving them” (p. 32). He provides four research-based assessment lessons that promote a deeper understanding of the theoretical underpinnings of learning community programs. In her article “Learning Communities: Examining Positive Outcomes,” Maureen Andrade similarly argues that assessment efforts should examine the impact of learning communities in greater depth and notes a need for studies that use qualitative methods to provide a more complete picture of the learning community experience.

Valuable studies that explore how and why learning communities are successful do exist. Examples include Laura Soldner, Yvonne Lee, and Paul Duby's work, described in their article “Welcome to the Block: Developing Freshman Learning Communities That Work.” Using focus group data, these authors show that learning community students were academically and socially integrated; felt a close connection with peers, faculty, and the university; and formed academic and social networks of support. Stephanie Baker and Norleen Pomerantz, in their article “Impact of Learning Communities on Retention at a Metropolitan University,” describe their interviews with learning community students, who recognized that access to faculty who motivated, cared about, and respected them was critical to their success. Jennifer Crissman, in “Clustered and Non-clustered First-Year Seminars,” describes how she used focus groups to demonstrate that FLC students felt more comfortable talking with faculty, had more positive interactions in and out of class with faculty, and viewed faculty as more approachable than did non-FLC students.

Although both quantitative and qualitative methods serve important research purposes, combining the two methodologies into a general research frame remains problematic. Quantitative purists believe that social science inquiry should be objective and that social observations should be framed in the same way that physical scientists frame physical phenomena. On the other hand, some qualitative researchers believe that the subjective knower is the only source of reality and that multiple constructed realities exist. This dichotomy is often called the incompatibility thesis, wherein one research culture maintains that social science research should be predicated on time-free and context-free generalizations, while the other contends that time-free and context-free generalizations are not even possible because social science research is value-bound. Decades ago, Sam Sieber, in the article “The Integration of Fieldwork and Survey Methods,” described these paradigm wars as one side “professing the superiority of” deep, rich observational data’ and the other the virtues of ‘hard, generalizable’ … data” (p. 1335). This fervent debate continues and is well represented by Egon Cuba's comment in his chapter “The Alternative Paradigm Dialog” that “accommodation between paradigms is impossible…. We are led to vastly diverse, disparate, and totally antithetical ends” (p. 81).

Contrary to the purists’ beliefs, our recent experience with a mixed-methods research study is that it allowed us to look at the learning community experience much more comprehensively than we could have if we had relied on only one research method. Our research question was “What are the enduring qualities of the learning communities that may persist three or four years after participation in the program?” We believed that a mixed-methods research approach would allow us to use one method to verify findings stemming from the other method and probe further into the data to understand their meaning. Using a qualitative approach, we could study student narratives from focus groups to determine whether FLC membership had lasting effects. These narratives then could be linked to quantitative data on academic outcomes (grade point averages, retention rates, and graduation rates); survey items from the 2008 National Survey of Student Engagement (NSSE), and an internal instrument, the Survey of Recent Graduates (SRG).

Academic outcomes at our institution include consistently higher grade point averages (GPAs), graduation rates, and retention rates for FLC participants than for non-FLC participants. But because our program is voluntary, these findings raise the question of whether self-selection, not program effects, is a more plausible explanation of the differences in academic outcomes for FLC and non-FLC students. In testing this possibility, Julie Hotchkiss, Robert Moore, and Melinda Pitts, in their article “Freshman Learning Communities, College Performance, and Retention,” found that “students who are likely to perform worse than average are more likely to participate in a FLC” (p. 207). They also found that FLCs were particularly effective for minority students. Indeed, belonging to a FLC contributed to the probability that black men and women would be enrolled the following year (31 and 19 percentage points, respectively when compared to non-FLC black men and women). Except for white females (where there was little probable difference), students who participated in the FLC program were more likely to be underachievers than overachievers. These conclusions make our findings even more salient. For example, at our institution, FLC students have significantly higher GPAs than non-FLC students their first semester, and these GPAs remain higher even after two years. The freshman-to-sophomore retention rate for the FLC cohort is consistently higher, with significantly greater retention even two to four years out. Furthermore, FLC students consistently take more credit hours per semester and make greater progress toward graduation than non-FLC students.

But while these quantifiable outcomes suggest a long-term positive impact of FLCs, they represent only one perspective on the FLC experience. In addition, we cannot assume causation when assessing quantitative data on seniors because a number of other factors (for example, maturity, employment, or engagement in one's major) complicate our ability to isolate and measure the effects of FLC membership over time.

However, the qualitative results of focus groups allow us to explore the story behind the numbers and determine, from the students’ perspectives, whether elements of the FLC program endure years later. According to our focus group participants, participation in the program provided opportunities for students to become close to their professors and one another. Students reported that the FLC experience not only fostered collaboration with professors and other students but also helped them to be more at ease in our large university and city. FLC students felt encouraged to participate in campus and community activities and credited the FLC program with facilitating their transition into university life. Because the FLCs are theme-based, students further suggested that their experience in a particular FLC contributed to their choice of major, either reaffirming or leading them to reconsider their initial decision. Finally, they recommended that the program continue into the second semester in order to provide additional academic and social support.

These findings are consistent with previous research on learning communities. Moreover, findings from the focus groups suggest that the effects of FLCs persisted over time. Students associated four particularly strong, enduring qualities with the learning community experience: (1) student-professor connections, (2) collaboration, (3) engagement with the university or city, and (4) lasting friendships.

After analyzing the focus group findings, we linked the results to corresponding items from the NSSE and the SRG. Results from the survey items substantiated the focus group theme of student-professor connections, showing that FLC students were more likely than non-FLC participants to interact with faculty through activities other than coursework, even years later. Collaboration also appears to be an enduring quality of the learning community experience, with survey data for both freshman and senior FLC students indicating that they are more likely to report working effectively with others than non-FLC students. In addition, the FLC experience has a lasting impact on involvement in the community through activities like civic engagement projects and participation in elections at local, state, or national levels. Finally, learning communities have a multilayered effect on the quality of friendships; survey data support the focus group statements that FLC participants are more likely to venture out of their comfort zone to interact with others from different backgrounds. Essentially, data from the NSSE and the SRG validate student reports that the learning community experience fosters connections with other students, faculty, the university, and the city, as well as the idea that these connections persist throughout students’ university career.

This combination of qualitative and quantitative data provides evidence that learning communities are worth the investment in the long run. In addition, a mixed-methods research frame allows researchers to use data that may already be available at their institution (for example, GPAs, retention rates, national and internal survey data, and student narratives) to deepen understanding of the first-year experience.

Researchers can combine qualitative and quantitative approaches to serve any of the five purposes identified by Jennifer Green, Valerie Caracelli, and Wendy Graham in their article “Toward a Conceptual Framework for Mixed-Method Evaluation Designs”:

Three of these five purposes are represented in this study. We employ triangulation when we use multiple data sets to substantiate findings on the enduring impact of FLCs; complementarity occurs when we expand our understanding of academic outcomes by drawing on focus group data; and our use of survey data to further clarify focus group findings represents the purpose of development. If we had relied only on the numbers or only on the focus group reports, our research framework and understanding of outcomes would have been considerably more limited.

Our experience supports Anthony Onwuegbuzie and Nancy Leech's suggestion in their article “Enhancing the Interpretation of ‘Significant Findings'” that researchers employ a pragmatic lens, by using both quantitative and qualitative techniques, rather than a single lens, as mono-method studies do. The pragmatic lens allows investigators to zoom in on details and zoom out to the broader picture, with opportunities to combine macro and micro levels of a study. We were able to zoom in on the details of students’ perspectives on collaboration, lasting relationships with faculty, and types of engagement with the university and the city, substantiating these findings with survey data, and also zoom out to view the quantitative indicators of lasting impact on academic performance.

Onwuegbuzie and Leech call the use of mixed-methods techniques for data analysis the “gold standard” for studying phenomena. We agree. Too often, quantitative data become static, never moving knowledge beyond the limits of positivistic renderings (the numbers). By incorporating a qualitative data component, we enable knowledge to become dynamic; that is, the multiple layers of narrative meaning hidden by the numbers are revealed. As we found in the research reported here, student voices no longer inhabit the periphery of knowledge building, but become central to acquiring a fuller understanding of the freshman learning community experience.