The Value of Mixed Methods Research

Abstract

The purpose of this explanatory mixed methods study was to examine the perceived value of mixed methods research for graduate students. The quantitative phase was an experiment examining the effect of a passage’s methodology on students’ perceived value. Results indicated students scored the mixed methods passage as more valuable than those who scored the quantitative or qualitative passage. The qualitative phase involved focus groups to better understand students’ perceptions of the perceived value of mixed methods. Findings suggested graduate students view mixed methods passages as having rigorous methods, a newer history, and providing a deeper meaning of the phenomenon. This study adds to the literature base by revealing what value graduate students assign to quantitative, qualitative, and mixed methods research.

Keywords

mixed methods value graduate student perceptions

As a mixed methods researcher, one important question you must ask yourself is, “Is mixed methods going to add more value than a single method?” Mixed methods research has been practiced since the 1950s but formally began in the late 1980s and is increasingly used by a growing number of researchers (Creswell, 2003; Creswell & Plano Clark, 2007; Dunning, Williams, Abonyi, & Crooks, 2008). The increase in mixed methods research justifies the question of determining the perceived value of mixed methods research compared with a purely quantitative or purely qualitative study. It is important to understand the perceived value of combining two distinct methodologies, especially given the added resources, time, and expertise required to conduct a mixed methods study. Mixed methods research requires additional time due to the need to collect and analyze two different types of data (Creswell & Plano Clark, 2011). Researchers typically require additional funding for added supplies, extra space to interview participants or administer a survey, and assistants to help with data collection and data analysis. In addition, mixed methods research requires knowledge of both quantitative and qualitative methodology. Many researchers do not have training in quantitative and qualitative methodology; so, this can mean finding additional researchers with expertise in a particular area.

With the added burden researchers face conducting mixed methods studies, it is important to understand if mixed methods research adds any value to graduate students’ understandings and interpretations of a study as compared with a purely quantitative or purely qualitative study. It is important to understand graduate students’ perspectives because graduate students are one large group of consumers of published research. To date, there is very little work that examines graduate students’ perceived value of a mixed methods study (Creswell & Plano Clark, 2007; Tashakkori & Teddlie, 2003a).

Hurmerinta-Peltomaki and Nummela (2006) looked at the value of mixed methods in the field of business by reviewing studies published in the field. They found mixed methods added value by increasing validity in the findings, informing the collection of the second data source, and assisting with knowledge creation. The authors argue studies that use a mixed methods approach gain a deeper, broader understanding of the phenomenon than studies that do not utilize both a quantitative and qualitative approach. Another study looked at how mixed methods is used in the health services research. Researchers found that only 18% of the studies were noted as mixed methods. The researchers stated that the use of mixed methods is driven by the apparent shortfall of quantitative methods (O’Cathain, Murphy, & Nicholl, 2007). Another study in the business field found mixed methods articles received more citations than studies not utilizing mixed methods, which the researcher attributed to mixed methods studies being more valuable (Molina-Azorin, 2011).

Another value of mixed methods is the integration component. Integration gives readers more confidence in the results and the conclusions they draw from the study (O’Cathain, Murphy, & Nicholl, 2010). Mixed methods also help researchers cultivate ideas for future research (O’Cathain et al., 2010). In addition, researchers state mixed methods research is the only way to be certain of findings (Coyle & Williams, 2000; Sieber, 1973) and interpretation (Morse & Chung, 2003; Tashakkori & Teddlie, 2003b).

Some studies have looked at how to evaluate a mixed methods study. Sale and Brazil (2004) examined many articles and found no articles had discussed the criteria for judging a mixed methods article. Since this article, there has been additional work done. Creswell and Plano Clark (2011) included a section on evaluating a mixed methods study in their book. They listed such criteria as including both methods, having a rigorous method, and setting the study within philosophical assumptions. While it is important to have criteria to judge a mixed methods study, it is also important to understand the perceived value of a mixed methods study. If we do not understand graduate students’ perceived value of mixed methods, we have no way of knowing how a consumer group determines the worth of mixed methods.

The studies mentioned above address the quality of mixed methods and it may seem that value and quality are closely related. However, in this study, quality and value are not the same thing. This study examined the perceived value of mixed methods by looking at graduate students’ opinion of the usefulness of using mixed methods, not whether a mixed methods passage was the best mixed methods passage they have ever read (quality).

So, while these studies have made significant contributions to the field of mixed methods, they have not looked at graduate students’ perceived value of mixed methods as measured by confidence in results, better understanding of the findings, and perceived as a complete study. For this study, the perceived value of mixed methods studies was defined as a methodology’s ability to make sense of the world, help readers better understand the study, increase confidence in findings, improve accuracy and completeness, and inform and contribute to overall validity. This definition is based on what the researcher found in a previous study (McKim, 2013). This study looked at what researchers reported was the value of the methodology they chose. The researcher examined many published articles looking for what researchers reported were the reasons and strengths for utilizing a particular methodology. This information was synthesized to generate the value researchers report for the methodology they chose.

Determining the perceived value of mixed methods research from a graduate student perspective is important for a number of reasons. Conferences have begun focusing on mixed methods research. Martin Orland (as cited in Viadero, 2005) said “there is unprecedented interest now in the methodological quality of studies in education.” One group that has shown interest is graduate students. This interest is evident in thesis and dissertation methodologies as well as conferences. Table 1 shows the increase in the number of theses and dissertations containing the word “mixed methods.” A steady increase has been seen over the past 30 years. Students are also attending conferences containing mixed methods topics. At the American Educational Research Association, in 2013, graduate students were presenting in the Mixed Methods Special Interest Group as well as attending talks focused on mixed methods. Graduate students are also a consumer of research. Researching graduate students’ perceived value of studies that utilize both quantitative and qualitative approaches allows researchers to understand one of their consumer groups. The findings from this study can also help researchers who train graduate students. Knowledge of how graduate students value certain methodologies allows mentors to understand how they approach research.

Table 1.

Number of Dissertations and Theses Containing the Keyword “Mixed Methods.”

Year range	Number
2010-2013	2,538
2005-2009	2,524
2000-2004	532
1995-1999	100
1990-1994	26
1985-1989	17
1980-1984	3

Note. The number represents the number of dissertations and theses that contained the keyword “mixed methods.” This search was conducted in 2014 using the search engine “Proquest”. The most recent time frame only includes 4 years, while the others include 5 years.

The overall purpose of the study was to examine the perceived value of mixed methods research for graduate students at a Midwestern university using a mixed methods design. The intentions were multiple including educating researchers on mixed methods research and its perceived value in the research field while providing researchers with information about how graduate students perceive the value of the methodology of a study. This research may encourage other researchers to use multiple methodologies in their research and continue to study the perceived value of mixed methodology.

Method

Study Overall

This study utilized a sequential explanatory design (Creswell & Plano Clark, 2011) consisting of two phases where the quantitative phase was dominant, meaning more weight was placed on the quantitative phase (Creswell & Plano Clark, 2007). A sequential explanatory design was utilized so data from the focus groups could help explain the quantitative results for the purpose of complementarity (Greene, Caracelli, & Graham, 1989). The data were connected and the quantitative phase helped inform the qualitative phase. This connection happened in two places. The first connection of the quantitative and qualitative phase was the use of the quantitative results to create the focus group questions. The second connection was the mixing that happened after the qualitative data were collected and analyzed. The results were connected to gain a better understanding of the findings from both phases. The study aimed to answer the central question, “What value do graduate students perceive mixed methods has compared with quantitative or qualitative research?” This question was addressed by three subquestions:

Do students who read a quantitative, qualitative, or mixed methods passage differ in their perceived value of a study’s methodology? (Quantitative Phase)

How do graduate students assess the value of a study’s methodology? (Qualitative Phase)

What are graduate students’ perceptions of the value of mixed methods methodology? (Qualitative Phase)

Quantitative Phase Methodology

The first phase answered, “Do students who read a quantitative, qualitative, or mixed methods passage differ in their perceived value of a study’s methodology?” This question was answered by utilizing an experimental design where participants were provided with one of three methodologically different passages and asked to rate the passage on a value scale.

Passage Development

In this study, a methodological passage was a two-page summary of a study focusing on students’ perceptions of statistics. The content of the three passages were based on previous research conducted by the author (McKim, 2014). The three passages (a purely quantitative, a purely qualitative, and a mixed methods passage) were identical with regard to the Introduction and Discussion but different with regard to the methods used and the presentation of the findings. For example, in the purely quantitative passage the participants read the same introduction and discussion as the participants who read the qualitative and mixed methods passage. The introduction and discussion were uniform so graduate students’ scores were based on methodology not the information presented in the introduction or discussion. The passages differed with regard to the methods and findings. Passages intentionally varied so differences in scores reflected students’ perceptions of the value of the methods and results.

The purely quantitative passage informed the participants a questionnaire was administered and the results were presented in a typically quantitative format (see Appendix A). Example statistics included means, standard deviations, t values, and correlation values. This differed from the qualitative passage where the reader was informed the participants were interviewed and the result section included themes from the interview along with quotations from participants (see Appendix B for Qualitative Passage). The mixed methods passage was a combination of the quantitative and qualitative passage’s methods and results (see Appendix C).

Each student was randomly assigned to read only one methodological passage. The students were not informed that they were randomly assigned nor what passage they were reading. After reviewing the passage, the student was instructed to respond to a series of survey items addressing the value of the methodology. As opposed to all students reading all three passages, students were randomly assigned to one of three groups with each group being assigned to read a different passage. This random assignment was utilized for two reasons. First, reading all three passages might have affected the scores because participants would have been scoring the passages against each other. Second, reading all three passages would have required additional time and may have resulted in participant dropout.

Perceived Value Survey Development

The perceived value survey included 33 items measuring the value of the methodology. This survey was developed using themes found in a previous study (McKim, 2013). The previous study was based on a review of quantitative, qualitative, and mixed methods articles. The articles were examined for the value of methodology from the researcher’s perspective. The study details how the survey was constructed. Each questionnaire item was on a Likert-type scale where respondents rated how much they agreed or disagreed with a particular item. Example items included, “The study’s design is optimal for readers having a deeper understanding” and “This is a strong methodological study.” All participants were given the same items so responses could be compared.

Participant Identification and Access

Once the institutional review board granted permission to conduct the study, instructors teaching graduate courses in education and psychology were asked to forward an e-mail containing information about the study to their graduate students. If graduate students chose to participate in the study and clicked on the link, they were randomly assigned to one of the three passages by an online software survey tool. After they finished reading the passage, they were then presented with the value survey on a separate page. Participation required approximately 20 minutes.

A total of 113 graduate students completed the survey (overall completion rate of 58%). Of the 99 participants who provided their gender, 66 were female (67%). The mean age was 33 years (SD = 10.17). A majority of the participants were Caucasian (83%). The mean number of years as a graduate student was 3.35 years (SD = 2.39). Participants reported they had participated in slightly more than four research projects on average (M = 4.43, SD = 4.21) with most of those projects being quantitative (M = 3.61, SD = 3.76). The graduate students were also randomly assigned to one of three passages. Thirty-seven graduate students read the quantitative passage, 43 read the qualitative passage, and 33 read the mixed methods passage.

Quantitative Phase Results

Descriptive Statistics

Descriptive statistics were calculated for each item based on what passage the graduate students read. These results, calculated using SPSS, are presented below (see Table 2). The descriptive statistics revealed several interesting differences. Overall, the mixed methods group perceived the passage as providing readers with a deeper understanding (M = 3.84, SD = 0.95), a stronger methodological study (M = 3.72, SD = 0.89), a better understanding of the results (M = 3.97, SD = 0.70), more reliable findings (M = 3.38, SD = 0.83), and an optimal design for readers having a deep understanding already mentioned above (M = 3.56, SD = 0.98). The graduate students who read the mixed methods passage also felt the mixed methods passage was a better design for the study (M = 3.84, SD = 0.88) compared with the graduate students who read the qualitative (M = 3.00, SD = 1.05) and quantitative passages (M = 2.81, SD = 0.81).

Table 2.

Means and Standard Deviations by Group for Each Value Survey Item.

Survey item	Quantitative group, n = 37	Qualitative group, n = 43	Mixed group, n = 33
I am confident in the interpretation of the results.	3.08 (0.92)	3.14 (0.87)	3.73 (0.72)
I think more evidence could have been provided.^a	2.14 (0.79)	2.00 (0.77)	2.52 (1.06)
This study had the participants’ voice in the results.	2.49 (1.04)	4.02 (0.64)	4.00 (0.67)
I think the methodology is sufficient to address the study’s purpose.	3.08 (0.92)	3.52 (0.97)	4.00 (0.66)
The chosen methodology provides readers with a deeper understanding of the findings.	2.65 (0.92)	3.43 (0.99)	3.84 (0.95)
I have a clear understanding of what the researcher did.	3.41 (1.09)	3.55 (0.92)	3.94 (0.86)
I have a clear understanding of what the researcher found.	3.32 (1.06)	3.67 (0.61)	3.91 (0.84)
I have a clear understanding of the methodology the researcher chose.	3.32 (1.08)	3.50 (0.83)	3.94 (0.90)
This methodology explored students’ experiences in their statistics course.	2.73 (1.12)	3.86 (0.99)	3.75 (1.24)
I would have a better understanding of the findings with a different method.^a	2.97 (0.76)	3.12 (0.99)	3.66 (0.83)
I would have a better understanding of the findings if more information about the methodology was provided.*	2.32 (0.82)	2.38 (1.06)	2.81 (1.03)
The results are useful.	3.46 (0.87)	3.60 (0.85)	3.94 (0.62)
This is a strong methodological study.	2.62 (0.86)	2.74 (0.95)	3.72 (0.89)
Nothing could be done to improve this study.	1.81 (0.70)	1.95 (0.84)	2.66 (0.97)
This study would be stronger with a different method.^a	2.62 (0.83)	2.81 (0.73)	3.31 (0.69)
I have a deeper understanding of the study after reading the results.	2.89 (0.94)	3.50 (0.89)	3.87 (0.66)
This study’s methodology provides me with a better understanding of students’ perceptions of their statistics course.	2.97 (1.09)	3.67 (0.81)	3.97 (0.70)
The study’s methodology did not influence the findings.^a	2.95 (0.81)	3.40 (0.91)	2.75 (0.84)
This methodology is the best for ensuring the results are not influenced by the researcher.	3.05 (0.91)	2.51 (0.91)	3.34 (0.90)
The sample is sufficient for the conclusions that were drawn.	3.30 (0.91)	3.02 (0.99)	3.41 (0.98)
Participant selection was appropriate for this methodology.	3.46 (0.77)	3.42 (0.91)	3.75 (0.76)
This methodology is sufficient to generalize to other college students enrolled in statistics.	3.05 (1.03)	2.21 (0.94)	3.44 (1.24)
After reading the results, I have a clear understanding of what the participants were reporting.	3.14 (1.06)	3.31 (0.87)	3.81 (0.78)
There is sufficient evidence for the interpretations drawn.	3.00 (0.88)	2.86 (0.92)	3.66 (0.97)
The chosen methodology is appropriate based on the study’s purpose.	2.97 (0.83)	3.53 (0.91)	4.09 (0.73)
The research design is the best design for what the study wanted to address.	2.81 (0.81)	3.00 (1.05)	3.84 (0.88)
I would have had a better understanding of the results had the researcher provided more evidence.^a	2.16 (0.73)	2.31 (0.78)	2.63 (0.87)
Selection of the participants was appropriate based on the study’s purpose.	3.35 (0.82)	3.45 (0.77)	3.84 (0.72)
I think another methodology would better address the study’s purpose.^a	2.68 (0.78)	3.02 (0.78)	3.28 (0.85)
The findings from this study are reliable because of the chosen methodology.	2.92 (0.86)	2.83 (0.88)	3.38 (0.83)
The study’s design is optimal for readers having a deeper understanding.	2.54 (1.02)	3.07 (1.09)	3.56 (0.98)
The design is appropriate for this study.	3.16 (0.76)	3.48 (0.67)	3.78 (0.71)
The chosen methodology provides readers with a better understanding of the findings.	2.70 (0.78)	3.48 (0.83)	3.72 (0.96)

Note. Items were rated on a 5-point Likert-type scale, where 1 = strongly disagree and 5 = strongly agree. Standard deviations are presented in parentheses.

Denotes a reverse-coded item.

After descriptive statistics were calculated a total perceived value score was calculated. This score was an average of all 33 items. There were no subscales since all items assessed the components of perceived value. The perceived value survey had a reliability of .95 (Cronbach’s alpha). This was calculated on all survey items for this study.

Group Differences

There was a significant difference between participants who read the three passages on their perceived value of the study, F(2, 112) = 15.52, p < .01. Least significant difference post hoc tests revealed the group that read the quantitative (M = 2.89, SD = 0.51) and the group that read the qualitative passages (M = 3.08, SD = 0.55) were significantly different from the group that read the mixed methods passage (M = 3.59, SD = 0.61) on their perceived value of the study. Overall, participants who read the mixed methods passage rated it higher than the groups that read the quantitative and qualitative passages.

Controlling for Prior Experience

Because the analysis of variance (ANOVA) revealed differences between the three groups, an analysis of covariance (ANCOVA) was conducted to control for prior experience since it was assumed prior research experience could affect participants’ views of certain methodologies. The researcher assumed graduate students with more experience conducting certain types of studies might place relatively more value on those methodologies. For example, students with past involvement in mixed methods studies may view mixed methods methodology as more valuable since they are more frequently involved in those types of studies. Participants reported they had participated in slightly more than four research projects (M = 4.43, SD = 4.21) with most of those projects being quantitative (M = 3.61, SD = 3.76).

The assumptions for ANCOVA were met. In particular, the homogeneity of the regression effect was evident for the covariate. The ANCOVA was significant, F(2, 98) = 12.60, p < .01. When controlling for prior experience, the group that read the mixed methods passage had the largest adjusted mean (M = 3.57), followed by the group that read the qualitative passage (M = 3.15), and the group that read the quantitative passage had the smallest adjusted mean (M = 2.88). Least significant difference follow-up tests were conducted to evaluate pairwise differences among the adjusted means. There were significant differences in the adjusted means between the quantitative and mixed methods groups and the qualitative and mixed methods groups.

Quantitative Phase Summary

The ANOVA showed graduate students perceived the mixed methods passage as more valuable than the quantitative or qualitative passage. The ANCOVA further expanded on these results by revealing that even controlling for prior experience graduate students perceived the mixed methods passage as more valuable. Overall, results for the ANOVA and ANCOVA both revealed students who read the mixed methods passage reported the highest perceived value.

Qualitative Phase Methodology

In a sequential explanatory study, the quantitative piece is a follow-up by a qualitative piece (Creswell & Plano Clark, 2011). The section will focus on the qualitative piece, which served as a follow-up to the dominant quantitative phase.

The qualitative component of the study sought to answer two questions not addressed in the quantitative phase: “How do graduate students assess the value of a study’s methodology?” and “What are graduate students’ perceptions of the value of mixed methods methodology?” The qualitative component included focus groups allowing the researcher to further understand how graduate students determine the value of a methodology. More specifically, the researcher wanted to know what perception students hold about the value of mixed methods research.

Focus Group Protocol Development

To best answer the research questions, the researcher utilized a focus group. Focus groups are a great methodology for gathering lots of information from a group of individuals. Focus groups also allow participants to expand on each other’s responses (Beyea & Nicoll, 2000). Since each graduate student read only one passage, this was a way to bring everyone together and collaborate on the various methodologies. Using a focus group also allowed the researcher to reach a consensus on the topic as well as obtain several perspectives on the same topic (Morgan & Spanish, 1984).

The focus group protocol included open-ended questions that sought to further understand the value participants place on certain methodologies and how graduate students judge the merit of a study. These questions were created based on the findings of the quantitative component. Focus group participants were asked about their experiences with the various methodologies, how they evaluate the methodology, rigor, and value of a study, and their perception of the perceived value of mixed methods. The focus groups were structured to answer two research questions. The first research question was “How do graduate students assess the value of a study’s methodology?” The second research question was “What are graduate students’ perceptions of the perceived value of mixed methods methodology?” The semistructured focus groups were invited to share their perceptions of quantitative, qualitative, and mixed methods methodology and asked the following questions: “How do you judge the merit of a quantitative study?” How do you judge the merit of a qualitative study?” “How do you judge the merit of a mixed methods study?” and “What is the value of mixed methods research?”

Participant Selection

Participants for the qualitative component of the study volunteered at the end of the quantitative study to be contacted later about participating in a focus group. Participants were not purposefully sampled because of the time required to participate and the fact that contact information was needed. Students also were not forced to identify themselves in the study; so, purposefully sampling could not be conducted. There were 11 students who volunteered and participated in two focus groups. The number of participants was based on Stake’s (2006) recommendation of sampling 4 to 10 participants. The majority of the participants were female (n = 8, 72%) and there were three males who participated. All participants were graduate students who participated in the quantitative phase. Students were not asked what passage they read since they were not informed in the quantitative phase that they were being randomly assigned to one passage. What passage participants read in the quantitative phase was not known or collected. The researcher met with the focus groups for approximately 45 minutes outside of class time. Notes were taken during both focus groups and reviewed later for key findings; no audio recording device was utilized. During the focus groups, notes were also taken on the whiteboard, so students could provide immediate feedback if the researcher recorded something incorrectly. Thematic analysis was used to analyze the qualitative data collected during the focus groups (Braun & Clarke, 2006). The thematic analysis was done by hand and findings were taken back to a few participants (n = 5) who volunteered to review the findings for accuracy. The qualitative data collection was used to support and further understand the findings from the quantitative component of the study.

Qualitative Phase Findings

Research Question 1

Focus groups revealed three findings for each methodology addressing how graduate students assess the value of a methodology. The findings are presented below based on the methodology.

Qualitative Methodology

When students were asked how they assess the value of qualitative studies, students discussed such things as design type, sampling, coding, quotations from participants, and validation. Many graduate students reported the value of a good qualitative study was in the author’s description of the design type. They believed the design type description was twofold. First, they expected to see exactly what design type was used. Second, students expected the proper components of that design throughout the study. Students judged the value of a qualitative study based on “the information about the sampling procedure” used in the study. Students also mentioned when they judged the value of a qualitative study, they looked at the coding methodology used and evaluated how appropriate it is for the study. Additionally, they mentioned the “depth of information” provided about the coding method is important. Students felt when information about coding was provided, it provides readers with a clear idea of how the findings were obtained.

Another component students evaluated when judging an article was the use of participants’ voices. They stated that in a good qualitative study, a reader would “hear voices” and “stories” throughout the study’s findings. One graduate student believed that quotes are “part of the story” and without them, “it is hard to hear the voices or stories.” Another graduate student stated that “without quotations from the participants in the study,” a qualitative methodology has not truly been employed. Graduate students wanted the participants’ voices to validate the findings of the study.

Another key component of qualitative studies participants mentioned was not only the use of quotations to validate the study but also the use of techniques such as member checking to validate the study. They believed validation techniques should be explained in detail and should convince readers that the findings are accurate. In summary, participant feedback suggested a superior qualitative study has a clear design type, discussion of sampling procedures, details about coding procedures, quotations from participants, and discussion of validation techniques.

Quantitative Methodology

When students were asked how they assessed the value of quantitative studies, they discussed such things as instrument selection, research questions, sampling procedures, design, and limitations. Students considered the common components of a method section to be important to a quantitative study. These common components included the instrument, research questions, sampling procedures, and the study’s design. Students judged the study based on the instrument chosen and “the instrument statistics like reliability and validity.” One student said, they “expect to see part of the instrument, including reliability and validity.” Students stated that they assessed the value of a study based on how important the research questions are and how the “methodology answers the research questions” presented in the study. In addition, they stated how important the sampling procedures are to the value of a quantitative study. One graduate student stated that they wanted to see “information about sample like participants’ background” and how the researcher chose the participants.

Graduate students also referred to the study’s design when judging the value. In general, they wanted to see a “design that is useable” and a “design that controls for the effects of independent variables” in a valuable study. Since the design is so crucial to a quantitative study in their eyes, they thought this should dictate other components of the study such as the philosophical assumptions, “tables and graphs,” and the analyses used. One graduate student said, “analyses are not enough, they should also include if the assumptions were met.”

Graduate students also mentioned limitations when asked how they assess the value of a study believing that in a valuable quantitative study the author lists the limitations “so that future researchers know what problems they might have.” They also shared that if the study’s limitations were extensive, then it raised a red flag about the quality of the study. In summary, a superior quantitative study had a discussion of instrument selection, appropriate research questions, description of sampling procedures and design, and discussion of limitations.

Mixed Methods Methodology

When graduate students were asked about how they judged the value of a mixed methods study, students mentioned rationale, research team, data collection timeline, description of both components, and integration. They mentioned throughout the focus groups the importance of the study’s rationale when judging any methodological study. In their views, a valuable study has a strong rationale backed by a “mixed methods purpose,” as was phrased by one graduate student. In addition, they also wanted to have a discussion “of the team and their expertise” in the paper. One student stated that “the team should have both types of researchers” and that one type of researcher was not enough. Students regarded the expertise of the team as very valuable to a mixed methods study. They believed researchers “need to know quantitative and qualitative” methods in order to combine both methods into a strong mixed methods study.

With regard to the methodology of a valuable mixed methods study, graduate students discussed the need for a complete detailed timeline of when each component of the study took place and exactly what was collected in each phase. One student stated, it is “important to know when and what was collected,” so one can “have a better idea of exactly what was going on.”

A broader theme related to this feedback was the idea of having a detailed description of both components. Students not only talked about knowing when data were collected but also what was collected, from whom, and what was done with the data collected. They believed a strong mixed methods study should make readers feel like they are the researcher. Graduate students wanted enough information, so they could picture exactly how the study was conducted, almost as if they had done it themselves. Graduate students said a strong mixed methods paper contains a “blueprint” of what was done. This blueprint should also be followed by a clear description of how the data were “mixed” or “integrated.” One student stated, a high-quality mixed methods study discusses how the researcher(s) “combined both types of data” and how the “qualitative informs the quantitative” and vice versa. In summary, they believed a superior mixed methods study has a strong rationale, discussion of a qualified research team, details of data collection timeline, description of both components, and discussion of integration of both components. As one student put it, “when done correctly, there is something in it for everyone” or as another student stated, “everyone can gain something.” Students summed it up by saying no matter what the reader’s philosophical worldview, he or she can walk away understanding a mixed methods study.

Research Question 2

Focus groups revealed three themes when asked about their perceptions of the value of mixed methods methodology. The three themes, rigorous method, audience, and history, are presented below.

Rigorous Method

When graduate students were asked about their perception of the value of mixed methods studies, they mentioned mixed methods studies have rigorous methods. Students discussed such things as the strength of the approach used, the objectivity, and complexity. Specifically, students discussed how a mixed methods study is more complex than a purely quantitative or qualitative study because it requires “knowledge of both,” a design that utilizes both quantitative and qualitative methods and a mixing component. One graduate student expanded by saying because mixed methods research is so complex, “no one really wants to do mixed methods studies” but when done correctly, the study is very rigorous.

Audience

Graduate students also discussed how important the audience is in a mixed methods study. Students stated, before a researcher starts a mixed methods study he or she wants to have a good idea who their audience is. While graduate students felt that “everyone can gain something” from a mixed methods study, they did not believe that everyone is open to a mixed methods study. In fact, they stated some people do not perceive mixed methods study as rigorous and therefore considering your audience is important when deciding whether to conduct a mixed methods study. Another component to the audience theme was the “deeper meaning” readers walk away with from a mixed methods study. Students discussed how a mixed methods study leaves readers with a “full story” composed of “multiple perspectives.” One student stated a mixed methods study “gives multiple inputs” and paints the “complete picture.”

History

Another theme that arose when graduate students were asked about their perception of the value of mixed methods was the history of mixed methods research. Graduate students discussed how mixed methods has a “short history” and is still “building a reputation.” Students discussed the limited references that exist to help researchers interested in mixed methods. When asked what references exist, students mentioned work by researchers such as Creswell, Plano Clark, Tashakkori, and Teddlie. They commented how the field is not as frequently populated with studies, as are the quantitative and qualitative fields. One graduate student stated that once people stop “trying to be perfect and realize the world is a mess and so is research,” maybe more would try mixed methods. Students believe mixed methods methodology is contemporary and once researchers start to realize that people with various worldviews “can gain something,” more researchers will consider mixing quantitative and qualitative methods.

Connecting the Quantitative and Qualitative Findings

Overall, the quantitative results show students judged the mixed methods passage as more valuable than students who read the quantitative or qualitative passage. Qualitative findings showed students judge the value of a study based on the method chosen and, while mixed methods may be newer than quantitative and qualitative research, students felt people with different worldviews and paradigms can gain something from a mixed methods study.

The findings from the qualitative phase also revealed that students believed mixed methods studies present more evidence for the findings and interpretations. This aligns with the quantitative findings dealing with graduate students perceiving the mixed methods results as more valuable. By providing more evidence for the findings, the graduate students believed people who read the mixed method passage would have more confidence in the study. This increased confidence in mixed methods research was first reported in the quantitative component of the study and then further understood when asking graduate students about the value of mixed methods studies.

Overall, the qualitative component sheds additional light on the quantitative component by providing further understanding of all the criteria students use to judge an article. The quantitative results showed graduate students perceive mixed methods studies as more valuable and the focus groups further expanded on this when graduate students stated they believe mixed methods results are more complex in nature when done correctly and mixed methods studies have something for everyone, no matter what their philosophical worldview.

Discussion

Quantitative results indicated students scored the mixed methods passage as more valuable than those who scored the quantitative or qualitative passage. The qualitative phase revealed mixed methods passages as having rigorous methods, newer history, and providing readers with a deep meaning of the phenomenon.

Researchers have criticized quantitative and qualitative methods for years. Qualitative research has been criticized for lacking things such as objectivity (Nagel, 1986) and generalizability (Gelo, Braakmann, & Benetka, 2008), while quantitative research has been criticized for lacking participants’ voice and a meaningful interpretation (Toomela, 2008). Many researchers have turned to mixed methods methodology as a way to address the critiques of quantitative and qualitative methods. Graduate students scored mixed methods higher with regard to perceived value and further explained that when done correctly, mixed methods has something for all readers, regardless of their philosophical worldview. They also stated that mixed methods is more rigorous than quantitative and qualitative methods.

Mixed methods methodology has received support in the literature for numerous reasons. Choosing mixed methods research combines the strengths of each methodology and minimizes the weaknesses (Creswell & Plano Clark, 2007). Another reason for selecting mixed methods research was the need to understand what information is encoded in a variable so the interpretation is meaningful (Toomela, 2008). Graduate students in the focus groups also supported this stance in the literature mentioning mixed methods is critical in understanding complex phenomena because it allows readers to understand and explain. This expands on Schulze’s (2003) findings that mixed methods research provides more breadth, depth, and richness as compared with either quantitative or qualitative methods alone.

Bryman’s (2006) work focused on the rationale for using mixed methods research. Graduate students mentioned in the focus groups that the rationale in the study was very important. They stated that the reason the author mentions for using mixed methods is critical in judging the value of the study and a valuable mixed methods study has a strong rationale for using the methodology along with a clear purpose. In addition to the rationale, graduate students also wanted to see a detailed timeline of when the quantitative and qualitative components took place. Bryman (2006) found most researchers say the rationale for using mixed methods is to enhance the findings. This was an area graduate students touched on during the focus group. Graduate students said a valuable mixed methods study uses one methodology to inform the other and the ability of one methodology to inform the other often creates a more complete picture.

Researchers have claimed mixed methods research provides a more balanced perspective (Morse & Chung, 2003) and is therefore needed (e.g., Coyle & Williams, 2000; Johnson & Turner, 2003; Morse & Chung, 2003; Schulze, 2003). Graduate students confirmed these statements by rating the mixed methods passage as having more value than students who read a quantitative or qualitative passage. Graduate students expanded on the findings by revealing how they judged a study. Students judged the various components (i.e., rationale, research team, data collection timeline, description of both components, and integration) of mixed methods studies more critically than quantitative and qualitative studies, but students saw more value in mixed methods study. When asked what value students see in mixed methods methodology, students discussed confirmation of results, deeper meanings, multiple perspectives, and rigor. This expands on what other researchers have stated about the value of mixed methods. Coyle and Williams (2000) stated mixed methods is the only way to be certain of findings, and other researchers stated mixed methods is the only method that provides the most accurate interpretation (Morse & Chung, 2003; Tashakkori & Teddlie, 2003b). Creswell and Plano Clark (2007) clarified that the value of mixed methods is the combination of two methods with the goal of providing readers with a better understanding than a singular method can. Greene (2008) stated that mixed methods invites everyone to participate because it provides multiple views and standpoints.

The findings from this study contribute to the field of mixed methods. Researchers continue critiquing the value of mixed methods compared with the other, more established methodologies. Few empirical studies have examined the perceived value of mixed methods. This study also utilized an innovative experimental methodology in the quantitative phase. This study is unique since it provides feedback from graduate students with regard to the value of particular methodologies and how they judge a mixed methods study. Since graduate students are not only conducting research using various methodologies but also consuming research, it is important to understand their perceptions of various methodologies. This study lays the groundwork for future researchers to continue to create interview protocols or additional surveys assessing the value of mixed methods from the eyes of researchers or other research consumers.

Since value and quality are often intertwined in responses received from participants, future researchers could examine the difference between value and quality. This study illustrates how students discuss quality, such as a clear rationale and a well-rounded research team, when asked about the value of a particular methodology. This connection between the two concepts is something that could further be explored to better understand the overlap and/or distinction between value and quality. Instructors of research methods courses can also use the findings from this study when structuring a methodological course. For example, instructors could design discussion around methodological strengths and weaknesses, a study’s rationale, and the importance of considering your audience. Instructors of mixed methods course could specifically focus on the mixed methods rationale, the importance of a strong research team, the discussion of the timeline for data collection, the description of the quantitative and qualitative components, and the integration of both components.

The results of this study are specific to the treatment conditions. Specifically, one methodological limitation to this study is the fact that participants only read one passage. It would have been stronger to have them read all three passages so comparison could have been made among all three passages, ruling out individual differences. This approach was not utilized due to time and potential carryover effects. It would also have been stronger to have all three passages exactly the same length. With variations in length, it is hard to rule out the concern of additional information influencing participants. Future researchers could try using three unique passages. This would allow students to score three unique passages and may lower dropout due to boredom. When creating unique passages, researchers should also consider length as a confounding variable. Longer passages may lend to higher scores not because of a higher perceived value but because readers were provided with more information. The use of a recording device during the focus group would also have provided longer quotes from participants, maybe providing a better understanding of their perceived value of the methodologies.

Another limitation of this study is the characteristics of the sample. Only graduate students at one university in the areas of psychology, education, and administration were contacted to participate. This sample is not representative of all graduate students because of the diverse training various universities provide. Researchers should consider examining the value of mixed methods in different domains and in other fields. Researchers in other fields such as art, English, or physics may value certain methodologies differently than social science researchers (i.e., education and psychology). These fields also conduct quantitative, qualitative, and mixed methods studies and the value in those methodologies may be different for different fields. This study also only looked at a narrow definition of value, which could be closely tied to quality. Future researchers could examine value and quality in one study to delineate the difference between the two concepts.

The field of mixed methods would benefit from additional research on perceived value of mixed methods. The area of value with regard to mixed methods has been lacking for a while (Creswell & Plano Clark, 2007). Being able to understand the value of mixed methods could provide further justification for the use of mixed methods in various fields. Future researchers might examine the perceived value of mixed methods for new mixed methods researchers. Future researchers may also examine the perceived value for more senior researchers who carry out mixed methods research and those who do not conduct mixed methods studies. This would help the field understand the perception and value attributed to mixed methods research. Once researchers and consumers see the value, time can be spent on expanding the field. This expansion could include areas such as design types and special mixed methods topics.

Footnotes

Appendix A

Appendix B

Appendix C

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Beyea

S. C.

Nicoll

L. H.

(2000). Learn more using focus groups. Association of Operating Room Nurses Journal, 71, 897-900.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101.

Bryman

(2006). Integrating quantitative and qualitative research: How it is done? Qualitative Research, 6, 97-113.

Coyle

Williams

(2000). An exploration of the epistemological intricacies of using qualitative data to develop a quantitative measure of user views of health care. Journal of Advanced Nursing, 31, 1235-1243.

Creswell

J. W.

(2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage.

Creswell

J. W.

Plano Clark

V. L.

(2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Dunning

Williams

Abonyi

Crooks

(2008). A mixed method approach to quality of life research: A case study approach. Social Indicators Research, 85, 145-158.

Earp

(2007). Development and validation of the statistics anxiety measures (Unpublished doctoral dissertation). University of Denver, Colorado.

10.

Gal

Ginsburg

Schau

(1997). Monitoring attitudes and beliefs in statistics education. In Gal

Garfield

J. B.

(Eds.), The assessment challenge in statistics education (pp. 37-51). Amsterdam, Netherlands: IOS Press.

11.

Gelo

Braakmann

Benetka

(2008). Quantitative and qualitative research: Beyond the debate. Integrative Psychological & Behavioral Science, 42, 266-290.

12.

Greene

J. C.

(2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2(1), 7-22.

13.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.

14.

Hilton

S. C.

Schau

Olsen

J. A.

(2004). Survey of attitudes towards statistics: Factor structure invariance by gender and by administration time. Structural Equation Modeling, 11, 92-109.

15.

Hurmerinta-Peltomaki

Nummela

(2006). Mixed methods in international business research: A value-added perspective. Management International Review, 46, 439-459.

16.

Johnson

Turner

L. A.

(2003). Data collection strategies in mixed methods. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in the social & behavioral research (pp. 297-319). Thousand Oaks, CA: Sage.

17.

Loftsgaarden

D. O.

Watkins

A. E.

(1998). Statistics teaching in colleges and universities: Courses, instructors, and degrees in fall 1995. The American Statistician, 52, 308-314.

18.

McKim

(2013, October). The value of quantitative, qualitative, and mixed methods: The value researchers report in their studies. Paper presented at the annual meeting of the Northern Rocky Mountain Educational Research Association, Jackson Hole, WY.

19.

McKim

(2014). Understanding undergraduate statistical anxiety. Journal of Research in Education, 24, 204-210.

20.

Molina-Azorin

J. F.

(2011). The use and added value of mixed methods in management research. Journal of Mixed Methods Research, 5(1), 7-24.

21.

Morgan

D. L.

Spanish

M. T.

(1984). Focus groups: A new tool for qualitative research. Qualitative Sociology, 7, 253-270.

22.

Morse

J. M.

Chung

S. E.

(2003). Toward holism: The significance of methodological pluralism. International Journal of Qualitative Methods, 2(3), 1-12.

23.

Nagel

(1986). The view from nowhere. New York, NY: Oxford University Press.

24.

O’Cathain

Murphy

Nicholl

(2007). Why, and how, mixed methods research is undertaken in health services research in England: A mixed methods study. BMC Health Services Research, 7, 85. doi:10.1186/1472-6963-7-85

25.

O’Cathain

Murphy

Nicholl

(2010). Three techniques for integrating data in mixed methods studies. British Medical Journal, 314, 1147-1150.

26.

Sale

J. E. M.

Brazil

(2004). A strategy to identify critical appraisal criteria for primary mixed-methods studies. Quality & Quantity, 38, 351-365.

27.

Schau

Stevens

Dauphinee

T. L.

Del Vecchio

(1995). The development and validation of the survey of attitudes toward statistics. Educational and Psychological Measurement, 55, 868-875.

28.

Schulze

(2003). Views of the combination of quantitative and qualitative research approaches. Progressio, 25(2), 8-20.

29.

Sieber

S. D.

(1973). The integration of fieldwork and survey methods. American Journal of Sociology, 78, 1335-1359.

30.

Stake

R. E.

(2006). Multiple case study analysis. New York, NY: Guilford.

31.

Tashakkori

Teddlie

(2003a). Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage.

32.

Tashakkori

Teddlie

(2003b). Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. International Journal of Social Research Methodology, 6, 61-77.

33.

Toomela

(2008). Variables in psychology: A critique of quantitative psychology. Integrative Psychological & Behavioral Science, 42, 245-265.

34.

Viadero

(2005). “Mixed methods” research examined. Education Week. Retrieved from http://www.edweek.org/ew/articles/2005/01/26/20mixed.h24.html