Abstract
This article provides an examination of a range of mixed methods research projects that employ Internet-mediated technologies (IMT) for data collection. Using a case study approach, this article allows for the uncovering of a process by which IMT are used as a data collection medium in mixed methods praxis. Under the theoretical position of medium theory, the impact of how going online and changing the “mode” of traditional research will be explored. Advantages, drawbacks, as well as ethical issues that emerge for mixed methods using the online medium are highlighted throughout the article. Finally, implications and directions for future research using IMT are discussed.
The use of Internet-mediated mixed methods research holds the promise of enabling mixed methods researchers to harness the power of emergent Web 2.0 technology to address difficult and complex social issues within and across their disciplines and across the global society as a whole. This article provides an examination of a range of mixed methods research projects that employ Internet-mediated technologies (IMT) for data collection, allowing for the uncovering of a process by which IMT are used as a data collection medium in mixed methods praxis. In so doing, we will assess the advantages, drawbacks, as well as ethical issues that emerge for mixed methods praxis and the extent to which IMT add synergy to the practice of mixed methods research.
There are currently 1.5 billion Internet users, which amounts to 22% of the world’s population. In the United States alone, 77.3% of U.S. households are connected to the Internet (Internet World Statistics, 2010) with a frequency access rate ranging from weekly to daily contact (Horrigan, 2007). The growth of user-generated Internet data has, in turn, challenged the research community to think outside their traditional method practices and research concepts such as the idea of what a field site should look like. Internet-mediated research is already transforming the way researchers practice traditional research methods such as survey research and ethnography as they confront the challenges of taking these traditional methods online (Andrews, Nonnecke, & Preece, 2003; Denissen, Neumann, & van Zalk, 2010; Dicks & Mason, 2008; James & Busher, 2009; Robinson & Schulz, 2011).
To date, few research studies have examined the extent to which Internet-mediated research facilitates the practice of mixed methods research. Earlier research studies that focus specifically on mixed methods designs suggest that the gathering of mixed methods data over the Internet is an important way mixed methods researchers can validate their offline research findings (Hewson, 2003, 2007, 2008). In addition, some mixed methods studies using the Internet for one component of a research project found that this practice serves to complement research findings from the offline component (Davis, Bolding, Hart, Sherr, & Elford, 2004). Taking mixed methods research online is also said to produce a more representative sample that improves researchers’ ability to generalize their overall results (Hewson, 2003). However, the implications of moving traditionally offline methods of study online may have major implications for the praxis and findings of social science research. Hine (2008) questions if we move away from the security of predefined offline “fields” to an online medium, how can we be certain about making claims regarding a study being “in depth” when environment is constantly fleeting and changing. Through the presentation of a variety of a cases that provide different examples of the way mixtures between mode and type of method (qual/quan) can vary, we aim to explore not only the problems and prospects mixed methods research bumps up against as Internet technologies become incorporated but also how different modes affect the way the message is delivered to and interpreted by the receiver.
Marshall McLuhan (1964/1994), in his path-breaking book Understanding Media: The Extensions of Man, published in 1964, notes the character of the medium through which a given message is delivered is of paramount importance regarding how it is understood by the recipient. He coined the phrase “the medium is the message,” because he firmly believed that a given medium has an independent impact in its own right regardless of the content of the message. Joshua Meyrowitz (1985, 1994) extended McLuhan’s ideas by outlining a theoretical communications perspective he terms medium theory, to denote the importance of the “means” of communications’ impact on recipients’ perception of message content.
The theoretical perspective of “medium theory” is therefore especially important to consider when looking at mixed methods as it interfaces its praxis with new modes of communication. Given the already fast-pace integration of emergent Internet technologies into social research praxis, it behooves the researcher to be cognizant of medium theory’s perspective on social research praxis. We need to be aware how varying modes in timing (synchronous/asynchronous), direction (unidirection/bidirectional), and public/private forums affect the meaning and interpretation of the message being delivered. These new technologies “are a transformative force that will challenge received knowledge, generate original empirical insights, and catalyze new theories” (Hackett, 2011, p. 26). Researchers need to be able to adapt to new communication technologies and the norms they create within society as they begin to challenge “existing research assumptions and premises” (Sudweeks & Simoff, 1999, p. 30).
Internet-mediated research has been spearheaded by the commercial sector of the economy, especially among market researchers. The application of Internet technologies to social research projects has grown dramatically over the past two decades, although usage of the Internet for research is not evenly distributed among the disciplines. An examination of research articles published between 1994 and 2008 that use online technologies reveals that the highest rate of Internet usage is found in the natural and health sciences, education, computer science, and engineering, with the social sciences lagging significantly behind (Farrell & Peterson, 2010; Meyer & Schroeder, 2009).
Barriers to Usage of New Technologies for Social Science Researchers
Many barriers to the use of new technologies in social science research still remain for a variety of reasons. Accessing new modes of data collection may challenge a researcher to come out of his or her methods “comfort zone” and to develop new skills in both data collection and analysis. There may also be a bias against collecting new forms of data whose veracity has not been determined (Schonlau, van Soest, Kapteyn, & Couper, 2009; Travers, 2009). There are some researchers who may feel, for example, that these data are not representative or that obtaining a representative sample is not possible, given the ever-changing nature of cyberspace (Lieberman, 2008). There may also be a concern about how these data can be generalized to a population, when the nature of that population itself is also changing (Andrews et al., 2003).
Online research also opens up new possibilities and challenges along several dimensions of a research project’s design. One important dimension of a project is location in physical space. As a researcher moves a project’s data collection phase to an online environment, the research landscape’s spatial boundary becomes more fluid and it may be difficult to mark the boundaries of the research field site. How can qualitative Internet researchers define the boundaries of their projects (see Markham & Baym, 2010)? The time dimension of an online study may also vary from a synchronous to asynchronous mode. A synchronous mode of data collection is ongoing and streaming, whereas an asynchronous data collection mode (such as conducting an email survey) has a time lag in communication between the researcher and the respondent. As research moves online, the overall tenor and form of interaction between researcher and researched begins to shift, especially as new forms of social networking communication tools such as blogs and Twitter arise. These ICT boundaries between researcher and researched may become more fluid such that there is a blurring of the line between the real world and the cyber world, whereby new forms of social networking (interaction) can crossover into the “real” world as researcher and researched communicate offline (Hine, 2009). There is also the possibility that a lack of face-to-face interaction may cause meaning to be lost in the collection of data because nonverbal cues are missing from the interaction.
Ethical boundaries are also ever changing as the line between private and public information is blurred, especially when sometimes highly personal information can be obtained publically from social networking sites such as Facebook. To what extent do traditional ethics rules apply, once research moves online? How, for example, does the meaning of informed consent change as new forms of online interaction arise? What is the meaning of confidentiality in a cyber environment (Buchanan & Hvizdak, 2009; Buchanan & Williams, 2010; Markham, 2008)? Is it ethical for a researcher to use public information that may also be private (personal information gathered from Facebook; Buchanan & Hvizdak, 2009; Buchanan & Williams, 2010, Parry, 2011)?
Mixed Methods and Mixed Modal Research: A Case Study Approach
We examine a convenience sample of four articles to begin the conversation of design issues for mixed method Internet-mediated research. More specifically, the articles were selected to demonstrate the use of mixed methods designs (sequential and concurrent) and also use either a single Internet technology (used in one or both mixed methods components) or multiple Internet technologies (using two different Internet-mediated forms of data collection in either one or both mixed methods components). These case studies are not meant to be an exhaustive list of mixed methods studies or generalize toward all cases that comprise mixed modes/mixed methods Internet-mediated designs but are meant to be used as a heuristic device for extending our knowledge base with regard to exploring how mixed methods researchers incorporate Internet technologies into their mixed methods praxis. Are researchers cognizant of the potential impact a new technological mode might have on any aspect of the research process? And if so, how? The cases will allow us to get at process that is context specific. We will compare and contrast the strengths and weaknesses across the range of Internet-mediated mixed methods designs, noting the specific types of research questions that lend themselves to these different types of designs as well as explore the research challenges and synergies that arise in the application of this design for mixed methods praxis.
Case Study 1: Studying the Dynamics of Social Capital: The role of Social Media
Williams et al.’s (2006) article, “From Tree House to Barracks: The Social Life of Guilds in World of Warcraft,” seeks to address Putman’s (2000) claim that social capital is constantly degrading by suggesting that new forms of social media can positively influence the degree of one’s social capital. To pursue this question, the authors chose the online interactive game World of Warcraft (WoW) as an example of an online site where people interact in the pursuit of goals and engage in relationships that extend offline, providing evidence that upends Putman’s claims.
Research design
The authors devised three research questions to examine the social content of online gaming.
[1] What kinds of social organizations do players create in [multiplayer online games] such as WoW? [2] What kinds of roles and social relationships do players develop within guilds and what were the social consequences? Did bridging or bonding social capital go up or down as a result of membership? [and 3] To what extent does the WoW social interface impact social interactions? (pp. 340-341)
To answer these questions, the authors developed a sequential and concurrent mixed methods and single modal (all online) research design (see Figure 1). First, the research team created their own players to engage and learn about the online game WoW as active participants for a period of time ranging over 16 months. All researchers joined different groups (known as guilds) for a 3-month period to better understand the game mechanisms as well as its social context. Additionally, participation in the game allowed the researchers to learn and establish the current typology of guilds used in WoW, gathering data about the individuals that make up such guilds. Concurrently, a census was derived from bots, which are automated characters that log into the system and collect data around the clock. These data allowed the variety of guilds, type of servers, and players to be accounted for, thus represented in the sample used in the second part of their study.
Williams et al.’s (2006) research design: Concurrent and sequential mixed methods, single modal
The qualitative component and the final step in their design involved conducting in-depth ethnographic interviews. Participants were randomly selected from the sampling frame categories established in the first phase of the project. A total of 48 semistructured interviews were conducted through the systems online chat system over a 2-week period that incorporated 13 base questions and follow-up questions.
The authors analyzed both the quantitative and qualitative data, and much of the quantitative “talked” to the qualitative data, informing the issue of sampling frame and procedures (e.g., best time to collect data) and an understanding of the overall dynamics of the game. The analysis of the qualitative data suggests that playing WoW is another avenue wherein the players strengthen and maintain existing offline relationships, while additionally new online relationships have the potential to develop offline.
Research design strengths and weaknesses
This study exemplified many of the positive characteristics of conducting mixed methods research online. This mixed methods multimodal design enabled the researchers to explore a new form of community that takes place online and allows for the collection of large amounts of data in a short period; the authors were able to collect more than 1,000 responses in approximately 90 days. Additionally, their design allowed for ascertaining the behavior of an online community through network analysis. The utilization of a sequential design facilitated a representative sample to be created for the ethnographic component of their project. Finally, an Internet-mediated mixed methods design allowed not only to quantify many factors related to WoW players (i.e., number of players, player makeup of guilds, usage time, etc.) but also understand the social component of what this online play means to offline users by conducting qualitative interviews.
Many common ethical challenges emerged that researchers typically encounter in Internet-mediated research. The researchers engaged as players in the game to collect quantitative data and develop an operational understanding of the site and players; their methods give rise to interesting questions that plague many researchers, particularly the concept of “lurking.” Initially, the authors created “bots” to get an exploratory understanding of the mechanics of the game and different uses of play. However, it is likely that as they played the game to gather further information, they did not tell other players that they were researchers. The question of disclosure of a researcher’s motivations is an ethical concern for all forms for research, traditional and online. As we get better acquainted with taking our research designs online, the issues of lurking and ethics will be readdressed in this article.
In bringing traditional research methods to the online venue, researchers are not only able to study new concepts but also study “old” concepts in a new light. Virtual communities are redefining how we traditionally understand the word community. This medium of an online gaming venue indeed fostered social interaction, social identities, behaviors, and established a social life as theorized by medium theory (Meyrowitz, 1986, 1994, 1997). Through the mixture of traditional methods (i.e., interviews) and emergent technologies (i.e., all online, continuous data collection, use of a “bot”), the researchers were able to better understand what a community means to these users and how they interpret and create this reality. Gunkel (2011) argues the difficulty in proving the veracity of reality when it comes to online research. Williams et al. (2006) brought the traditional concept of social capital online to see if social capital offline translates to social capital online. Researchers are now able to investigate how social capital garnered online may translate into offline social capital accumulation.
Case Study 2: Building Self-Identity Online
Nip’s (2004a) study, “The Queer Sisters and its Electronic Bulletin Board: A Study of the Internet for Social Movement Mobilization,” examines how the Internet affects “identity building capacity” in social movements. Identity building within social movements refers to the development of a sense of solidarity, a collective consciousness, and a culture of direct opposition to the dominant culture. For these tenets of identity building to exist, a social network of relationships has to be created and maintained. Nip posits that the Internet allows for such identity formations to be created and nurtured, similar to direct face-to-face contact or offline relationships, which traditionally create identity-building capacity for individuals.
Research design
To explore if the Internet indeed can foster this sense of identity building, Nip developed four research questions to address this issue:
[1] Do participants on the electronic bulletin board develop a collective identity among themselves. [2] If yes, does the identity extend to offline organizations? [3] If yes, does the identity help action mobilization of the offline organization? [4] What role does the offline organization play in affecting the development of a collective identity on the bulletin board?
The development of these research questions stemmed from previous literature suggesting the Internet helps build identity.
To begin to answer these research questions, the researchers focused on the oldest electronic bulletin board for queer and lesbian individuals in Hong Kong. The bulletin board claims to be a “human rights organization fighting for the sexual rights of women.” The board is accessible to the public, as there is no registration or membership needed to post on the bulletin board or read posts by others. Nip developed a sequential and triangulated mixed methods and multimodal design as seen in Figure 2. This design gave her a better understanding of the development of social networks from online community to offline life. Additionally, it helped to begin to understand the many perceptions and uses of the bulletin board.
Nip’s (2004a) research design: Sequential and triangulated mixed method, multimodal
Nip announced her research intent at an offline meeting of the volunteers who manage the electronic bulletin board. Her desire to be an unobtrusive observer but record the postings, for later content analysis, was agreed on by the managers. During a 1-month period, Nip acted as an unobtrusive observer on the bulletin board to provide her background for the content analysis, collecting and evaluating 603 messages during two separate 1-month periods. At the end of the content analysis periods, Nip posted on the website an invitation to complete on online questionnaire. The invitation was continuously reposted for 6 weeks at which point the survey link was shut down. A total of 102 surveys were collected for analysis. The final point of data collection involved offline interviews. The interviews were conducted with volunteers who run the bulletin board and individuals who post on the bulletin board. Each portion of the research design sequentially flowed into the next portion of the design.
Research design strengths and weaknesses
Nip’s research design allowed her to begin to understand the phenomenon of identity building in an online context. She was able to easily secure a sample from the participants on the bulletin board for not only the online questionnaire but also for the in-depth interviews. This was an efficient and cost-effective method of data collection. However, one issue that Nip faced was temporal fluctuation. There was potential for certain times of the year to have greater or fewer posts compared with other times of the year, thus potentially affecting the reliability of the results.
Nip’s findings suggest that indeed the Internet allows for such identity formations to be created and nurtured. For some members, their participation in the bulletin board stemmed from preexisting offline relations, whereas for others, their participation led to many new offline relationships. Nip’s results suggest the online/offline connection exists and relationships can be strengthened and created through the online arena. In the context of the lesbian community using this bulletin board, the creation of an online identity as a lesbian may have strengthened or helped make sense of their offline identity as a lesbian. Nip was able to triangulate her findings from the online content analysis, online questionnaire, and offline interviews. To explore the phenomenon of collective identity through the Internet, an Internet-mediated mixed methods design provided informative results.
The use of the sequential and triangulated mixed methods and multimodal design in this study allowed for the research of a typically offline phenomenon (identity building in social movements) to be studied in an online context. This research adds a better understanding of the potential effectiveness of social movements in a time where advocacy and information is maintained online. The author was able to understand the community of users through content analysis, in-depth interviews, and online questionnaires. Additionally, the multimodal design allowed Nip to trace connections between the online and offline community in terms of understanding usage, posts, a sense of belongingness and consciousness to the queer community, and a culture formed in opposition to the dominant heterosexual community.
Nip understood through her content analysis that the communication created through the online mode (i.e., blog) was able to create its own reality for the participants. Similar to the findings of Williams et al. (2006), the content analysis confirmed that this online medium fostered social interaction, social identities, and established a social life as theorized by medium theory (Meyrowitz, 1986, 1994, 1997; Nip, 2004b). A community, in the newest sense of the word, no longer tied to a geographical location, was able to form through the online mode of a chat room, even though the timing was asynchronous and many of the users will never meet in person. People felt they developed relationships and had meaningful interactions through this online medium. Nip uncovered this finding by bringing traditional research methods of content analysis onto the Internet.
Nip confronted ethical issues when deciding how to inform members that she would be observing their posts on the bulletin board. It is interesting that the researcher obtained the permission of the managers of the bulletin board (offline) but did not make an announcement on the bulletin board (online) itself that she would be observing the posts. Again the issue of “lurking” or being an unobtrusive observer arises, similar to the Williams et al. (2006) study. It is not agreed on how researchers should make their presence known on something that is in constant flux such as a blog or bulletin board, or even gaming communities. Messages are constantly being added or players are constantly logging in, leaving the researcher uncertain as to how often to announce their presence, or even how far back to analyze archival postings for the use of research. This ethical area of online research needs to be explored further so as to not continually violate the rights of the target population being studied.
Case Study 3: Studying Social Movements Online
Biddix and Park’s (2008) study, “Online Networks of Student Protest: The Case of the Living Wage Campaign,” aims to study social networks and examine how the Internet can be used to bridge communities, in this case as a part of a social movement. Their study provides an example of how to conduct social network analysis using an online research technique called hyperlink. This study evaluated the hyperlinks between student groups and noncampus organizations’ websites. Specifically, the authors focused on activist groups addressing the issue of a living wage at a variety of universities and their connection to nonprofit organizations.
Research design
To identify the communities focused on the issue of a living wage, the researchers asked two research questions:
[1] Which individuals comprise the core and group network. [2] In what ways does the network affect mobilization and latency of the campaign?
The researchers hoped that answering these questions would allow them to better understand specific membership among networks, map the relationships between networks, and look at the different degrees of relationships and potential influence among them.
The authors created a single modal, triangulated mixed methods design to answer their proposed research questions. The first step in their design was to study hyperlinks between different organizations’ websites. Hyperlink analysis considers the website a “node” and the hyperlinks between sites a form of relationship, between not only the sites but also individuals at the respective organizations. These links can be hierarchical or lateral, depending on who is producing and who is consuming the information. Studying the directionality of links (what organizations have connections), isolated communities (not linked with other organizations), and the potential influence of one community (determined by the number of links connected to them from other organizations) over others helped the authors better understand the different organizations and how they communicated and potentially worked in conjunction with others. It has been noted that hyperlinks between groups may point to the importance of campaigns and effective coordination (see Figure 3).
Biddix and Park’s (2008) research design: Triangulated mixed methods design, single modal
This study evaluated the hyperlinks between the websites of 27 student groups and noncampus organizations currently active in the living wage campaign. Using the hyperlink method, the authors were able to rank and evaluate the websites’ connections as social networks and follow up with more in-depth online interviews through instant messenger (IM). Only 10 interviews were conducted through IM and email, with five alumni, two current students, and three representatives from noncampus organizations.
Research design strengths and weaknesses
Similar to the Williams et al. (2006) and Nip’s (2004a) studies, Biddix and Park’s (2008) research focused on how the use of the Internet affects offline relationships. Interestingly, the authors’ findings differ from the findings of the previous two case studies. They discuss the issue of the “digital divide” suggesting that overreliance on technology is detrimental to personal relationships, especially when it comes to outreach, when those you are advocating for (if a vulnerable population) may not have access to the same technology or be as technologically savvy. The research focused on university student groups and nonprofits, but results suggest that relying on spreading awareness solely through online mediums may not reach the entire community and groups of people may be “left behind.”
Conducting a mixed methods design with a focus on social network analysis and qualitative interviews allowed researchers to identify the networked living wage campaign community. Studying hyperlinks specifically gave the researchers a sample from which to find the leaders of targeted organizations and analyze the online connections between multiple organizations. Online interviews were used due to ease and cost efficiency; however, the findings from the qualitative portion of the study may not have yielded truly saturated results, due to the small sample size.
Hyperlink analysis is a new way to research social networks and analyze relationships through an online perspective, but it is not able to discern the extent to which offline relationships affect the strength of the online hyperlink relationship (creation and maintenance). Hyperlinks only provide an informational and exploratory perspective allowing for the exploration of hegemonic power and the online environment for political purposes. Hyperlink analysis provides researchers a new way of “quantifying” the degree of importance and potential influence in online relationships not only for organizations but potentially for individuals as well. Biddix and Park’s design also allowed them to evaluate how the composition and structure of social networks change as a result of digital media use.
The authors only briefly discussed the rationale of conducting interviews through IM, as a means to validate their decision. Opposed to Madge and O’Connor (2002), the authors did not discuss how they built rapport with their interviewees, or how emoticons or “silence” were interpreted through the interviewing process. Additionally, the informal nature these IM (and one email) interviews potentially may have affected their findings. Respondents may have felt at ease during the interview, due to the informal nature of the program, but it also may have diminished the quality of their responses by not taking the interview as seriously. The authors did not discuss these potential pitfalls and how they account for the particular online mode, which is something all researchers need to move toward as more and more traditional methods move online.
Case Study 4: Studying Online Social Support Groups
Madge and O’Connor’s (2002) study, “Online with E-Mums: Exploring the Internet as a Medium for Research,” explores the potential benefits and problems associated with online surveys and online synchronous interviews. To explore these issues, the authors examined the users of Internet-based website Cyberparents. The authors looked at how this website is used by new parents, how the information is derived from the site, and if the site is used as a form of social support. At the same time, the authors were interested in learning the benefits and pitfalls of conducting research methods solely online.
Research design
To explore the above stated goals, the authors developed a sequential mixed methods single modal study. The methods used for this project were an online survey and semistructured virtual group interviews (see Figure 4).
Madge and O’Connor’s (2002) research design: Sequential mixed methods, single modal
Madge and O’Connor used an online survey to reach a large sample size quickly, save money in lieu of printing and mailing costs, and analyze responses faster as they are collected around the clock, which loaded into an automatic analytical package. Additionally, user-friendly online surveys tend to have higher response rates compared with the traditional mail surveys. Madge and O’Connor’s survey gathered demographic data as well as data about the parents’ general Internet usage patterns, why they may use the Cyberparents website compared with other websites, and their attitudes about the website. The survey also asked if respondents could be contacted for a more in-depth online interview, providing a sample for the second stage in the research project.
Semistructured online group interviews occurred in a real-time, synchronous fashion. The purpose of these interviews was to gain a better understanding of themes found in the online questionnaire. The targeted population was familiar and comfortable with the online medium, as they are regular visitors of an online parents cyber network. Internet-mediated research was the most feasible option for this population because travel and time for parents with young children are rare and expensive for researchers. The group interviews were all done through text because audio and visual data collection was not available features of the software used by the researchers.
Research design strengths and weaknesses
One of the advantages of this research design is the equalizing nature of using Internet-mediated research. The power dynamics between the researcher and participants seem to be more equalized when the identities of both are unknown. However, the authors in this study posted pictures of the research team to add a sense of humanity and verify the credibility of the study. An additional reason for posting pictures and small biographies of the team was to build rapport, typically feared to be a missing link in online interviews. The researcher conducting the group interview also established rapport by sending personal emails to the participants to answer any questions before the actual interview.
One of the challenges in conducting online written interviews is the added layer of deciphering emoticons and acronyms for real emotions and fuller expressions (e.g., “lol”; “omg”). Researchers must be able to decipher this online language without being able to account for any nonverbal cues. Madge and O’Connor also struggled with how to interpret “silence” (a lack of typing) during an online interview. Additionally, a skilled researcher needs to know how and what to write in times of sensitive topics. Online interviews are increasingly able to incorporate video, as video conferencing technologies (e.g., Skype) become common in standard operating systems. However, these newer technologies may require certain computer upgrades and system requirements. In the present study, Madge and O’Connor sent all participants the necessary software and directions for installation of the interview operating system. This may be too costly for some researchers and too complicated of a process for respondents who are not as tech savvy as this particular group.
Another problem encountered by the research team was that the initial slew of responses to the survey tapered off after a 6-week period. This could be attributed to the link losing its appeal on the homepage of the Cyberparents website as users potentially began to overlook the link over time. Madge and O’Connor note that without access to the central registry for the research, they were not able to ascertain if they received a representative sample of users, and there is no guarantee that respondents did not fill out the survey more than once. The limited cooperation of the host website did allow for the link to be posted on the homepage, which proved tremendously helpful to the researchers.
One noted strength of Internet-mediated research that emerged from this study was that women were open and “vocal” with their opinions. The success of these group interviews may have been a result of the fact that women participated in the interviews from their homes, providing them a sense of “safety” to speak openly. The influence of the setting in an interview is an issue for both online and offline interviews and concern for which online interviews may prove beneficial.
The authors for this particular study needed to conduct an online mixed methods study because they were not only exploring an online phenomenon but also how it affects people’s offline lives. The quantitative portion of the study allowed the researchers to get a sense of the website’s users and their pattern of behaviors. The researchers were then able to more thoroughly explore how the website affects the users’ lives through in-depth interviews. The online mode enabled researchers to discuss and identify new challenges and benefits from bringing traditional methods online. However, they did not discuss how the online mode may have affected the answers they received either through the survey or through the online groups. The fact that online groups and surveys took place in the same environment that participants use, the networking site potentially further authenticated the findings (although this was not stated) as they were in their “natural setting.” The authors were aware that by changing the traditional mode of interviewing to an online venue would affect findings. They refer to Crang, Crang, and May (1999) who note the “use of communication media involves the creation of new forms of action and interaction in the social world, new kinds of social relationship and new ways of relating to others and to oneself” (p. 11). This type of research helps contribute to the literature on Internet-mediated mixed methods research as it begins to build a better understanding of not only how to effectively conduct research online but also illuminates many of the limitations that occur along the way.
Assessing the Role of Internet-Mediated Technologies in Facilitating and Hindering Mixed Methods Research Praxis
These four cases provide the foundation in which we can now tie together a number of thematic strands to understand the ways that Internet-mediated research methods can hinder or facilitate mixed methods research.
Table 1 presents a list of advantages provided to mixed methods research by Internet-mediated methods. These studies along with other research that have brought methods online have found the most common advantages are the reductions in cost and time to collect data (e.g., Beddows, 2008; Bell et al., 2010; Fox, Morris, & Rumsey, 2007; Joinson, 2005; Strickland et al., 2003; Teddlie, Tashakkori, & Johnson, 2008). Cost savings are found in travel, mailing, and office supplies for a pen-and-paper survey or simply travel costs for face-to-face interviews. Online interviews and other Internet-mediated methods also have the advantages of being directly loaded into data analyses packages, which saves time, diminishes the potential for human error in transcription or data entry, and strengthens validity (e.g., Fox et al., 2007; Kelly & Oldham, 1997; Strickland et al., 2003). Additionally, collecting data online may reduce an unnecessarily large time gap for researchers using a sequential design, whereas more traditional offline methods would require additional time for data collection.
Advantages of Internet-Mediated Technologies to Mixed Methods Research
Single or multimodal mixed methods design may hold the promise of enhancing the validity of the findings. Orgad (2009) suggests that there is an assumption among some researchers that, for example, doing a multimodal study will add a sense of “authenticity to the findings obtained in the online study component” (p. 51). Yet, as we have seen in the case studies presented, a more likely use of online research is for the enhancement of the overall understanding of the research problem rather than validation of the online research findings. Orgad further notes that before the researcher decides on whether or not to move their study online, it is important to be clear on the specific set of research goals for a given project. How can moving a study online assist with answering the research question or set of questions? Does going online enhance the overall research process? What about the drawbacks of taking a research project online? To what extent will going online add synergy of understanding of the research problem (Orgad, 2009)?
Researchers have also found an abundance of other advantages in using online research methods, such as studying “hard to reach” populations, specific subsets of the populations, or a wide range of individuals across dispersed populations (Biddix & Park, 2008; Madge & O’Connor, 2002; Nip, 2004a; Teddlie et al., 2008; Williams et al., 2006). Online research provides a great scope for targeting and accessing samples with certain desired qualities, whether these are gender-specific, age-specific, or more broadly representative, whereas practical constraints might make accessing such samples offline difficult. Of course, certain populations are still underrepresented on the Internet (e.g., the homeless), and this is why using offline and online sampling procedures concurrently may prove valuable (see also Matthews & Cramer, 2008).
Internet-mediated mixed methods research also allows for new research questions, especially with regard to revealing subjugated knowledge. New forms of data collection, such as observing the behaviors or traces of behaviors from social networking websites and blogs, may in fact allow researchers to “study up” by providing some information on the inner workings and behaviors of a range of elite populations (Mazur, 2010).
Another advantage for using online methods to conduct research is the potential for higher participation rates. Online questionnaires can be made very user friendly, thus potentially encouraging higher response rates (Madge & O’Connor, 2002; Nip, 2004a). In terms of conducting interviews, when an asynchronous method is chosen, an online survey provides more flexibility for the respondent to partake in the study and thereby increases participation rates. Conducting interviews online allows discussion groups to occur between people from different geographic locations, when otherwise it would have been impossible to find a common meeting time or space (Beddows, 2008; Madge & O’Connor, 2004; Strickland et al., 2003).
Finally, conducting research online may allow respondents to divulge more sensitive data. Participants are able to participate in interviews in the comfort of their own homes or venue of their choice, providing a sense of “safety,” which may allow for gathering more sensitive data (Madge & O’Connor, 2002; Nip, 2004a). This may counteract the traditional power dynamic that is found in research between the interviewer and the interviewee. The divide and equalizing effect of the online medium may in fact neutralize this power dynamic, allowing for more candid interviews (Madge & O’Connor, 2002). Additionally, online research has shown the potential for increased honesty and lessened social desirability regarding sensitive issues when research is brought online, potentially a result of the increased feeling of anonymity (Fox et al., 2007; Griffiths, 2010; Joinson, 1999). This is a double-edged sword of sorts, as people may feel “safer” in their own space during the interview, but they also may not be as engaged in the interview as if it were conducted in person. Online interviews and focus groups can also lack the interpersonal dynamics with the research and other group members, and it will be more difficult to establish rapport online than in person. However, as will be discussed later, the expertise of a moderator may help increase the engagement of online discussions.
Another potential advantage of bringing research online is the ability for researchers to ask “new” research questions. The emergence of research involving the online and offline continuums of relationships and identity (e.g., in real time, interactive games, online support groups, online social network sites) offer new potential for researchers. This new age of Internet research allows social scientists the ability to make meaning of complex social environments (Feldon & Kafai, 2008), connect online and offline realities, challenge established methodologies, and stimulate “the imagination of the researcher in new directions” (Hine, 2008, p. 533).
Potential Drawbacks of Online Research
Online research is not without its drawbacks, a full list of which is presented in Table 2. The first noted drawback is a lack of generalizability. Internet research may not be generalizable to the greater population for a variety of reasons. Internet users still tend to be composed mostly of White, well-educated, and younger individuals (Pew Internet and American Life Project, 2007). Sampling and self-selection is a major issue for research using any venue; however, online research may be skewed to the technologically savvy, better educated, those who can afford computers, and younger populations (Bell et al., 2010; Madge & O’Connor, 2002; Nip, 2004a; Strickland et al., 2003). Additionally, with technology constantly changing, it is important for researchers to be aware if the majority of desired respondents have access to required technology. Bandwidth can vary across countries if one is conducting a global study, and inability of the participants to load the research may lead to lower participation rates (Beddows, 2008; Fox et al., 2007). The potential for excluding portions of the population appears to be a great threat for online research and makes it relatively impossible to achieve a random sample for generalizing purposes (Beddows, 2008; Hewson, 2008; Ignacio, 2012). Temporal fluctuation, regarding when people respond or would be able to participate, also affects participations rates, sampling, and generalizability (Fox et al., 2007; Nip, 2004a; Williams et al., 2006). Individuals who do not have access to computers or the knowledge to engage in such research are more likely to be left out. Researchers may need to take the extra step and target websites that are known to cater to different populations of individuals or advertise at public libraries where individuals without personal computers may engage in computer use. In a study by Strickland et al. (2003), for a participant who did not have access to a personal computer a research assistant helped the individual participate in the study at a public library. This type of one-on-one assistance may not be feasible for all online research studies but is one way to help bridge the “digital divide.”
Potential Drawbacks to Internet-Mediated Technologies to Mixed Methods Research
The challenge of verifying identities online is a common area of criticism when conducting Internet-mediated research. It is virtually impossible to verify respondents’ age, gender, race, or any other demographic or personal characteristics (Biddix & Park, 2008; Griffiths, 2010; Madge & O’Connor, 2002; Nip, 2004a; Williams et al., 2006). This information is completely dependent on the honesty of the respondents. Respondents may misrepresent their identity, have an online persona different from their offline persona, and additionally have many “virtual selves” and complete the research as multiple “people” (Williams et al., 2006). This too affects the generalizability of the results of online studies (Beddows, 2008).
A lack of nonverbal cues is another potential drawback of conducting online research (Biddix & Park, 2008; Madge & O’Connor, 2002; Williams et al., 2006). When conducting interviews online, whether asynchronous or synchronous, the researcher lacks nonverbal cues that can be observed and interpreted in face-to-face interviews. These interpersonal connections with offline interviews also add to establishing rapport (Hewson, 2008). Tone of voice, body language, and gestures all provide a certain richness to qualitative data that may be missing from Internet-mediated research (if the voice and video component are not present; Beddows, 2008; Madge & O’Connor, 2002). However, the development of emoticons, use of punctuation (!!), and online “slang” (OMG, LOL, etc.) add a level of emotion in online interviews (Madge & O’Connor, 2002). Correct interpretation of these new online cues is crucial to the findings, but interpreting sarcasm and ironic statements may be more difficult in the online venue compared with an in-person interview. Referencing online web-based dictionaries on emoticons will help researchers be able to fully appreciate not only the participants’ tone but also the full meaning of their contributions (Fox et al., 2007).
Madge and O’Connor (2002) faced the difficulty of interpreting “silence” (delays in typing). The silence may have meant the participant was formulating a response, slow in typing a response, typed a response and did not hit “send,” left the interview, or merely declined to answer the probe but did not explicitly state refusal. As such, online interviews may alienate people who are not as comfortable typing or have poor typing abilities (Fox et al., 2007; Madge & O’Connor, 2002; Strickland et al., 2003). Typing speed may be a key factor in participants’ level of contribution in online focus groups or discussions. The loudest personality is no longer the dominant force in group interviews but rather the person who is the most comfortable typing. Researchers need to be cognizant of the fact that some people may not feel as comfortable working through the online medium, affecting their participation and engagement in the study. Researchers who aim to bring their research online, especially in terms of synchronous focus groups, need to not only have the skills of a moderator but also be fast typists in order to effectively moderate the group, as dynamics of online chat can be “fast, furious, and chaotic” (Fox et al., 2007, p. 542). Large online groups may even require a second moderator to be able to read and make sense of all the input data.
Ignacio (2012) suggests that the timing of responses should be taken into account when analyzing online data (through the use of time stamps) adding an “additional layer of observation” (p. 244). As people read the questions posed to them online there will be an effect of how and when they answer, another factor that may support the idea that the “medium is the message.” Researchers need to be aware in moving forward in a technologically driven world of these new found research issues as they analyze their findings and reflect on the research process. Potential follow-up with a subsample of participants regarding their feelings on the involvement in research process may help illuminate how people react to being studied online and cross-validate findings.
These issues all converge on the need for researchers to be aptly trained on Internet-mediated research, in order to be able to not only conduct research but also know how to interpret findings. Researchers need to be aware of the possibilities and potential downfalls of going online and the options open to them. This may require researchers to broaden their skill set or potentially rethink what we mean by relearning research methods and bringing research online.
As Internet-mediated methods continue to grow, mixed methods researchers will need to be mindful of the “ethics gaps” they will confront as the line between personal and public data becomes blurred (see Collste, 2011; Crasborn, 2010; Winston & Edelbach, 2009). For example, social networking sites such as Facebook contain personal information that is sometimes available to the general public. When a respondent gives their “informed consent” to participate in a study, they may have already been selected for a study based on information the researcher acquired about them online. Is it ethical to collect and analyze “public” data that are personal?
Many of the traditional ethical concerns of conducting research remain whether online or offline. The issue of “lurking” is present for many online research studies (e.g., Beddows, 2008; Griffiths, 2010). Properly informing individuals that their involvement in an online community may subject them to online research is an ethical issue for online researchers. At what point should researchers inform the online community that they are observing their posts or activity when the information is in constant flux? This is a definite confound for researchers to ensure that everyone is aware of their presence and ethics are not violated. There is a debate among social scientists as to what is considered public and what is private online (see Hine, 2008; Hookway, 2008; Parry, 2011). However, in many of our case studies (Madge & O’Connor, 2002; Nip, 2004a; Williams et al., 2006), the authors needed to initially get a sense of what the online community consisted of in order to ask the correct and appropriate questions during the qualitative portion of the study. As we get better acquainted with the process of taking our research designs online, the issues of lurking and ethics need to be readdressed in order to not violate participants’ rights but also collect valid and reliable data. Institutional review boards need to be educated on the ethical “grey area” of online research and what is public and private before signing off on research.
Conclusions
As Internet-mediated communications continue to become a central aspect of how individuals interact with one another, mixed methods researchers will be drawn increasingly into Internet-mediated data collection designs. The cases studies presented in this article reveal the enormous strengths of Internet technologies for asking new questions and getting at subjugated knowledge, especially in accessing hard to find populations. In addition, Internet-mediated methods will help save costs and close the time gap in sequential mixed methods designs. This time gap often prevents researchers from discussing the findings from both components in a mixed methods project because of pressures to publish and a lack of research funds to start the second (or subsequent) component of their mixed methods design. These cost and time savings will perhaps spur the needed integration of qualitative and quantitative findings that is often lacking in many traditional mixed methods research publications (see Hesse-Biber, 2010).
It is also important to note that Internet-mediated research practices are evolving over time in tandem with the continued evolution of Internet technology itself. New technologies that use geographic interactive systems allow for the representation and analysis of geographical information. For example, global positioning system devices allow researchers to examine social behavior in both space and place (see Steinberg & Steinberg, 2011). New mobile technologies are now used as sensors for social research to gather ongoing “real-time” streaming data, including information about a respondent’s location, as well as prompt the respondent through mobile surveys and the gathering of more in-depth qualitative information (Eagle, 2011). Given the expansion of emergent technologies over time, mixed methods research is in a unique position to take advantage of these technological tools. However, there is also a research-skills gap that also needs to be addressed; the next generation of researchers will need to be trained in the use and impact of these Internet-mediated tools in the social research process. There appears, in fact, to be a dynamic interaction between the research process and the deployment of new technologies (Hesse-Biber, 2011; Hesse-Biber & Leavy, 2008).
As this article has shown, there is a need for the further discussion of how to properly bring mixed methods online. This article aimed to highlight the issues that are present in such research. There are many positive features to Internet-mediated technologies that lure researchers online, but many of the traditional issues that plague researchers conducting mixed methods remain. Taking into account medium theory, researchers need to better understand how the medium affects the message they are presenting to participants, a discussion that is currently missing in the field. What are the social and psychological effects that differing modes may have on the research process and potentially affecting the findings? What are the new implications for traditional research methods? There are different meanings between sending someone a text message, compared to email, to a handwritten note, to a phone call, and even visiting in person, so a natural question emerges as how do these differing modes effect how messages are received and responded to by the participants? Online modes of research change the nature of discourse. Michaelson (1996) states, “The relative anonymity that IT provides also changes the rules of discourse” (p. 59). In online research, people are dependent on the written word and not to the spoken word. This will naturally affect the conversation and responses provided and researchers need to be mindful of this impact.
Researchers are now confronted with a variety of new issues in the rapidly changing online environment. With the shift of social research online, researchers are publishing a wealth of new studies to address the drawbacks of Internet-mediated methods (Gosling & Johnson, 2010) in order to foster the synergistic promise of doing mixed methods research. In the same light, there is a need for continued methodological research to ethically conduct, interpret, and build off Internet-mediated research. As such, social science researchers need to continue to learn and adapt their own methods to the ever-changing technological driven society.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.