Leveraging the Power of Online Qualitative Inquiry in Mixed Methods Research: Novel Prospects and Challenges Amidst COVID-19

Abstract

The rare circumstances of COVID-19 have transformed research toward increased dependence on online spaces. This article examines related challenges and opportunities, focusing on how philosophical and ethical implications are differentially manifest amid crisis. Anchored by a transformative perspective, our framework recognizes heightened vulnerabilities amid COVID-19; it seeks dexterous strategies for implementing qualitative strands that adapt well to a virtual context while remaining philosophically grounded and ethical. Our findings highlight issues of unequal access, disembodiment, safety and vulnerability, researcher positionality, anonymity, and the delineation between private and public spaces; we also showcase an array of virtual qualitative methods. We conclude that ethical practice in the use of online methods is likely to be broadly applicable and adaptable to the mixed methods research community.

Keywords

COVID-19 integration mixed methods research online qualitative research transformative paradigm

This article details the opportunities and challenges of carrying out the qualitative strands of mixed methods research (MMR) amid the unparalleled circumstances promulgated by the COVID-19 pandemic. While scholars have commented on the bourgeoning opportunities to explore the pandemic’s effects on humanity (Fetters & Molina-Azorin, 2020; Graber, 2020; Jowett, 2020; Kamei et al., 2021; Riha et al., 2021; Meixner & Spitzner, 2021), some of those very effects, in the form of social distancing and stay-at-home edicts, have recessed face-to-face qualitative research efforts or, in novel cases, pushed them online. This turn, and the possibility that online research will become a new normal, necessitates adroit consideration of how to use qualitative methods in a virtual form, ethically and practically, within MMR. How do inquirers harvest tenets like rapport building, inclusion, intersubjectivity, and privacy in ways safeguarded in face-to-face encounters? What new opportunities and challenges unfold online that may inform MMR’s future? Such questions form the center of this article.

Our discussion begins with an orientation to the context within which online methods have emerged, particularly during the coronavirus’s callous plight. We discuss social and philosophical contexts, highlighting inequities associated with the pandemic, and a broadening potential suitability of the transformative paradigm in MMR that they evoke. We explore opportunities and cautions in the use of online methods, underscoring a number of ethical considerations. The final sections cover specific online qualitative tools, followed by instances of exemplar MMR studies, discussion, and conclusions. The article aims to surface methodological opportunities for scholars to build from in the hoped-for diminution of the pandemic—and in future years.

Context of Online Mixed Methods Research

Well before the birth of the Internet, researchers gathered qualitative data absent face-to-face contact. We recall a time, years ago, when families would take the telephone off the hook during the dinner hour, to avoid being interrupted by multiple calls. The calls were often from telemarketers, but some were from researchers hoping to indulge an adult member of the household in “a few” questions. It seemed an odd request to discuss opinions with a faceless stranger, who could craftily twist and re-represent words at will. Today, participants of online research may feel the same, which is why researchers must be sensitive to the social, philosophical, and ethical context of their work (see Clarke, 2005); these are explored next.

Social Context

At the time this article was conceived, the first author had been conducting synchronous online interviews when the seriousness of the COVID-19 pandemic crystallized. In the ensuing weeks, aches in the hearts of humankind grew and were starkly revealed in terms of mounting cases of severe illness and death, strains on front-line healthcare workers and systems, acerbic political divisions, and plummeting well-being and happiness especially among poor, marginalized, and vulnerable persons (Graham, 2020). The world had changed, as was palpably evident—the term unprecedented entered universal discourse (Major, 2020)—and so had the backdrop in which the interviews transpired.

Observers of the pandemic highlighted the severity of the situation from a variety of angles: Healthcare workers and first responders worked interminable shifts, subjugating their own health; providers’ children, overwhelmed by fear and anxiety, experienced adverse effects to psychological well-being (Mahajan et al., 2020). Meanwhile, doctors and psychologists forecasted the abiding toll, manifest in post-traumatic stress disorder and other conditions, for those on the front lines (Madani, 2020; Wu et al., 2020). As economies tanked and unemployment levels reached unfounded peaks, exclusion hierarchies took center stage, widening the gulf between those with and without resources (Casale, 2020; Graham, 2020; Qureshi, 2020). Racism and xenophobia reared their grotesque heads in bigoted anti-Asian memes, closed borders, and hacked Zoom forums (Gao & Sai, 2021; Sanchez-Rivera, 2020). Domestic violence grew to an epidemic of silent proportions (Bradbury-Jones & Isham, 2020; Goh et al., 2020) with few, if any, to see the fading of bruises or to recognize subtle arcs in a survivor’s temperament.

Amid global lockdown, families and individuals sheltered within bounded spaces: apartments, houses, and skilled nursing facilities; church, school, and gym floors for many without homes. Children across the globe transitioned to school-at-home arrangements and virtual learning. The intensity at which professional activities took place online drastically increased; the Internet, for those with access, became a place of connection, stabilization, and refuge—though for those with lower incomes, digital gaps only widened (Vogels et al., 2020).

These developments, together with auspicious efforts to counter such ails (South et al., 2020), bring to light quandaries and questions, hence calls for doing research (Fetters & Molina-Azorin, 2020). Shannon-Baker (2016) challenges researchers to interpret research with attention to “the social and historical contexts of the community” (p. 328). Who cares for whom? How does advocacy for the destitute, the elderly, and those with disabilities function absent face-to-face encounters? Amid these crucial questions, we ask this: What are the challenges and opportunities of doing research virtually? How are ethical principles and practices to be upheld? What tools are amenable to online research?

Philosophical Context

To address issues of social context—of overwhelming fear, anxiety, and stress, of exacerbation of exclusion hierarchies, racism, xenophobia, and domestic violence, and of tightly confined shelter spaces with accompanying unevenness in Internet access—we ground our inquiry by looking to MMR’s long recognized imperative to attend to the philosophical underpinnings of inquiry. Mertens (2003) reminds us that even “science, an empirically based tradition, is also influenced by values” (p. 136). Amid inequities revealed or heightened by the pandemic, the values accompanying a phenomenon’s social context take on great importance. Hence, the transformative paradigm, advanced for MMR by Mertens (see Mertens, 2003, 2009, 2012, 2018; Mertens et al., 2008; Mertens & Tarsilla, 2015), is an attractive foundation for contemplating ethical issues in online research. It is described in Mertens et al. (2008) as “a framework of belief systems that directly engages members of culturally diverse groups with a focus on increased social justice” (p. 82).

To further understand the influence of values and philosophies, a helpful concept is the researcher’s mental model. This is defined in Greene (2007) as a “set of assumptions, understandings, predispositions, and values and beliefs” (p. 53) guiding inquiry and interpretation. Smith (1997) cites the premise that “action is impossible unless one has constructed a mental model, image, or definition of the situation” (p. 73). Similar concepts are paradigm, popularized by Kuhn (1962), and worldview, which Guba and Lincoln (2005) characterize along the dimensions of axiology, ontology, epistemology, and methodology. The bounds on use and contextualization of these terms may be loose; Denzin (2010) views paradigms as “human constructions…[that] define the shifting worldview of the researcher-as-bricoleur” (p. 421). See also Mertens (2018), Morgan (2007), and Shannon-Baker (2016).

These concepts reside in the gravelly terrain of MMR, alongside lingering issues around the role of foundational paradigms in guiding research. Denzin (2010) recalls a combatant state (1990–2005), wherein “two warring paradigm camps” (p. 421) created a climate in which constructivist qualitative approaches to inquiry became wholly incompatible with postpositivist quantitative approaches. Here, postpositivism entails an attitude of distance on the part of the researcher toward the research subject, ostensibly to avoid undue bias (Creswell, 2014); a researcher would subscribe to the ideal of an independent reality approximated through objective measures (Crotty, 1998). Constructivism admits multiple realities coexist amidst researcher-participation interaction (Creswell, 2014; Guba & Lincoln, 2005; Mertens, 2003).

Subsequent MMR scholarship contributes novel perspectives that would unify paradigms and integrate corresponding methodologies. The transformative paradigm is one such perspective; its assumptions “emanate from an ethical stance that emphasizes the pursuit of social justice and the furtherance of human rights” (Mertens, 2012, p. 256). It catalyzes integration by prescribing the use of MMR when those methods would support its aims. Indeed, the transformative paradigm emerged in response to inquiry that fell short of representing the experiences of those in marginalized communities (Mertens, 2018). Accordingly, such researchers often engage with critical theoretical frameworks and collaborate in inquiry and advocacy.

By now, a host of perspectives is available for addressing cross-paradigm tensions and integrating MMR studies. Among the most prominent, a pragmatist position acknowledges differences among philosophical stances, asserting that “assumptions are logically independent and therefore can be mixed and matched, in conjunction with choices about methods, to achieve the combination most appropriate” (Greene & Caracelli, 1997, p. 8). Many MMR studies claim this position as its grounding (Bryman, 2007). Cherryholmes (1999) cautions that pragmatism is “ferociously complex” (p. 1), requiring effort to prevent a principle-deficient pragmatism focused on expediency. Dialectical pluralism welcomes multiple perspectives and methodologies into spaces of dialog (Greene & Hall, 2010, p. 124). The aim is not to integrate discordances, but honor differences among varying values (Greene, 2007; Shannon-Baker, 2016). See Fetters and Molina-Azorin (2020) for relevant perspectives and a call to bring in more from across the world.

Relatedly, Hall and Howard (2008) offer a synergistic strategy for integrating MMR designs, which encompasses both the “linear or logical in flow” (p. 248) of typological designs (e.g., Creswell & Plano Clark, 2018) and the interactive features of Maxwell and Loomis’s (2003) systemic design approach. It operates under four principles: “the concept of synergy, the position of equal value, the ideology of difference, and the relationship between the researcher(s) and the study design” (p. 250). While other approaches are available (e.g., Creamer, 2017; Denzin, 2012; Hesse-Biber, 2010; Nastasi et al., 2010), we highlight Watson’s (2020) methods-braiding design strategy for its incorporation of arts-based research, characteristics of which are taken on by some online methods. Methods braiding “actively seeks complex and creative integration” (p. 68), managed by implementation in multiple phases: within a phase, the methods are to the extent possible given equal consideration; between research phases are “intensive periods of reflexive evaluation…including the purposeful consideration, justification, and articulation or research questions, rationales, and outcomes” (p. 69).

Online Qualitative Research: Emergence, Cautions, and Opportunities

Early online qualitative research featured analyses of two basic types: on individuals using the Internet or of text exchanged among users (Mitra & Cohen, 1999). With the internet’s evolution, opportunities to conduct qualitative research grew precipitously, providing ease, access, and cost savings (Rogers, 2015). New methods—and recycled versions of face-to-face protocols—came into prominence, such as synchronous interviews and focus groups, e-mail interviews, ethnographic observations, and thread analyses. Eysenbach and Till (2001) distinguished three classes of online methods: passive, where the researcher uses data with no other involvement; active, where the researcher participates in the online community; and traditional, where the researcher generates data through interviews. Roberts (2015) resituated these as mutually inclusive and interchangeable typologies.

Some writers (e.g., Gregory, 2018; Rogers, 2015) cautioned attempts to transpose traditional methods to online forums, given the lack of physical presence, different power dynamics, and “the disembodied aspects of applying what in practice should be rooted in a humanistic inquiry” (Gregory, 2018, p. 1612). Disembodiment is summarized in Oderberg (2012) as an absence of “the gestures, inflections, indicators of emotion and attitude...which are typical of communication when bodies are directly present” (p. 382); this may trouble both the exchange of information and the meaning-making experience. As an example, Gregory (2018) recounts the disembodied exchange of life-threatening weight loss techniques in an asynchronous space for pro-anorexic teenagers, wherein text eluded human interaction and could even be “spambot programmed” (Oderberg, 2012, p. 383). Analysis of such spaces can be difficult, particularly for novices (Gregory, 2018).

Other writers emphasize issues that are particularly worrisome during crises. Access to technology, even outside of crisis situations, diminishes for lower income persons and those living in rural areas; observing COVID-19, Vogels et al. (2020) describes this: “As Americans turn to the Internet for critical purposes, there are rekindled debates about how the digital divide...may hinder people’s ability to complete everyday tasks or even schoolwork” (n.p.). Further, Graber (2020) cautions researchers to maintain awareness of vulnerability in an “emotionally precarious” social context and to shift definitions of who is vulnerable during crises; researchers must also remain aware of their own well-being.

Having already noted ease, access, and cost savings as benefits of online research, opportunities to broaden or deepen knowledge in various ways are also part of the picture. The particular space in which online research is conducted may offer participants a safe environment in which to tell their stories, hence it may assist in efforts taken under a transformative perspective to engage.

Illingworth (2006) recounts asynchronous interviews with 15 women experiencing infertility and assisted reproductive procedures. “Within this research,” Illingworth recalled, “many participants welcomed the chance to participate and share...This eagerness may, in part, reflect the specific nature of their experiences—infertility remains a sensitive and socially isolating experience” (p. 5). Similar experiences are relayed by Orgad (2005) in work with breast cancer patients. Studying bullying, Bouchard (2016) asserts that anonymity online allowed adolescents “to candidly discuss sensitive topics” (p. 62). Synnot et al. (2014), while researching family experiences with multiple sclerosis, conducted in-person focus groups and an asynchronous forum; while focus groups engendered discussion, the authors observe deeper, more thoughtful exchanges in the online forum. Rogers (2015) lists such opportunities among intrinsic benefits of online inquiry: “deeper context, entirely new insights, and the ability to engage...in ways that are challenging or impossible in the traditional settings” (p. 37). Babcock (2020), curator of a blog on fieldwork during coronavirus, expresses similar enthusiasm.

As with any forum, a researcher’s positionality influences the character of an online space. Gregory (2018) imagines a scenario in which two researchers, one identifying as Black and the other white, observe a supremacist forum; both “could not have approached the site from the same position given how the social organization of the group’s mission is based on a negation of the Black student’s personhood and the racial supremacy of the white student” (p. 1615). In contrast, a student who observes a support group for women experiencing infertility might disclose her own conception difficulties, and instantly boost rapport within the group.

Several nuances to access as a benefit of online research are worth mentioning. Synnot et al. (2014) additionally observes that an online forum accommodates geographical dispersion and the complexities of participants’ schedules. Participant access to online spaces may initially require support, though newer online interfaces may improve accessibility, particularly for some disabled persons. Vine Foggo et al. (2020), noting communication difficulties encountered by persons with autism, found private online forums to be potent tools—elucidating rich data that may have been more challenging to gather face-to-face. Seymour (2001) opined, “persons with disabilities have the potential to be among the major beneficiaries of the technological revolution” (p. 149).

Indeed, online qualitative inquiry encompasses factors (e.g., access, vulnerability, sensitivity) that may, depending on social context, be beneficial or limiting; it is, thus, incumbent on researchers to contemplate philosophical positions and ethical considerations.

Ethical Considerations for Mixed Methods Researchers Pursuing Online Inquiry

Ethics in online research are disputed by scholars and institutional review boards (Gregory, 2018; Roberts, 2015). For example, some debate whether to treat online data as generated by human subjects or as secondary text sources created by authors (Roberts, 2015). Mixed methodologists new to online methods may find value in consulting disciplinary association guidelines, or general guidelines for Internet research (see Franzke et al., 2020). Although ethics can be broad-ranging and widely relevant, the transformative paradigm, with its emphasis on social justice, human rights, and engaging members of diverse groups, is helpful for anchoring our discussion of ethics online.

Buchanan (2000) writes, The researcher has the ultimate personal responsibility to be true to the participants. Presumably, no one would know if I were not a Latino woman if I wanted to study Latino women online and be sure I “fit in” the virtual culture: A mutual trust must predicate virtual research. (pp. 84–85)

While this quote underscores trust, voice, and self-representation, it also highlights the ethical terrain around anonymity. Gregory (2018) warned against a condition wherein researchers “can effortlessly exploit the role of anonymity or self-misrepresentation for the sake of gaining unwarranted access to online communities, especially those built and sustained by vulnerable populations” (p. 1614). Transparency and disclosure, fundamental tenets for online researchers to uphold, carry risks: a report from the Association of Internet Researchers highlights the “comparatively new risks and levels of risk posed to researchers whose work—and/or simply their public identity…triggers strong ideological reaction” (Franzke et al., 2020, p. 11). Research participants may exploit anonymity, too, by presenting faux selves who obfuscate the inquiry process, or pursue other nefarious aims. Though deception is not absent from face-to-face research encounters, the mystifying anonymity and avatar statuses provided by the Internet make deceptive schemes befuddling for researchers.

Anonymity, to Bouchard (2016), can be a double-edged sword in online research. As suggested earlier, participants in an online forum may feel disinhibited, “free to exchange their true attitudes and opinions without fear of reproach” (p. 62); consequently, a researcher may “‘hear’ things [they] would not in a face-to-face interview or focus group” (Buchanan, 2000, pp. 84–85), like revelation of illegal activity, which complicates the researcher’s ethical responsibility. Such considerations are relevant to Sharifimonfared and Hammersley’s (2020) mixed methods study of strategies employed by heavy MDMA users to quit or curtail drug use; the authors note that online methodology “may have recruited participants who would have been reluctant to discuss an illegal activity face-to-face due to concerns about confidentiality and being ‘outed’” (p. 8).

Another challenging ethical concern for online research is delineation between private and public spaces. This is a subject of debate within ethnographic circles (Gregory, 2018; Roberts, 2015), where it is noted that classifying online communities as public would make “many of the ethical considerations disappear” (Roberts, 2015, p. 317). Absent that solution, questions regarding the perimeters around consent abound, with answers depending on source, context, and motive. If a web space is password-protected, is it private? Does the sensitivity of the topic inform the delineation? Roberts (2015) recommended attending to “the accessibility of the community to the general public, the perceptions of members, community statements, topic and setting sensitivity, the permanence of records and the intended audience” (p. 318). Jowett (2020) advises researchers to consider, while not making assumptions, how online research might “be received by those individuals or communities whose content has been used” (para 7).

When inquiry involves private online spaces, membership versus non-membership may become a parameter of ethical consideration. Research jurisdiction over a forum, network, blogging site, or newsfeed is granted neither by the status of member nor follower alone. Gregory (2018) advises, “while membership in a group might not necessarily be required as a criterion for studying an online community, reflexivity is in most cases necessary to better understand one’s position as a researcher” (p. 1616). Here, reflexivity has the sense defined by Lincoln and Guba (2000) as “the process of reflecting critically on the self as researcher, the ‘human as instrument’” (p. 183). Importantly, Roberts (2015) asserts that reflexivity can reveal and counteract possible, and increasingly frequent, decisions to select an online space to circumvent procedural ethics that would be upheld face-to-face, such as interviewing children online without attaining parental consent.

The principle of avoiding harm is widely acknowledged, but may be difficult to maintain online. For example, signs that “participants are becoming discomforted” (Bouchard, 2016, p. 63) may be muted. Imagine a face-to-face study of suicidality, where third-party therapists are made available as needed; online provisions of this sort may be less effective. Bouchard (2016) advises researchers, especially when conducting studies of sensitive nature or with vulnerable populations, to attend vigilantly to participant responses and language choices, and devise methods for frequent check-in.

In sum, mixed methodologists conducting online qualitative inquiry are counseled to consider a study’s benefits in the context of respect for persons (e.g., anonymity, privacy), reflexivity (e.g., delineation between private and public spaces), and harm avoidance.

Online Qualitative Tools for Mixed Methodologists

The following discussion highlights a number of online qualitative tools that are nascent, gaining traction, or prominent amid current times. Recalling Eysenbach and Till’s (2001) categorization, the present emphasis is on active and traditional methods; however, an exemplar passive observational study appears in a later section. Each tool, as it is described below, centers researchers’ experiences navigating exceptional circumstances. Many methods are inspired by Lupton’s (2020) crowdsourced document on conducting pandemic fieldwork—bringing to the fore the most contemporary tools.

Synchronous Interviews and Focus Groups

Synchronous interviews and virtual focus groups may be held using computers with web cams, tablets, and smart phones. Researchers can launch from a bevy of utilities, free-of-charge and fee-based: Zoom, WebEx, Google Hangouts, and FaceTime are widely used, and many offer video, audio, and/or closed captioning options; while less commonly used for research, instant-messaging utilities such as Google Chat provide another forum. Studying the suitability of Zoom for online research, Archibald et al. (2019) finds that benefits (e.g., convenience, security, recording options) far outweigh costs.

Because online interactions constrain attention toward nonverbal cues, group dynamics, and researcher-participant affinity (Biedermann, 2018), rapport building is fundamental. On one hand, rapport may be fostered by natural comfort some gain in participating from home; others’ spaces may be ridden with distractions. As appropriate, interviewer self-disclosure may foster affinity; this “has the potential to produce supportive, empathetic spaces, where participants feel secure” (Bouchard, 2016, p. 65).

As with face-to-face studies, consideration should be given to what transpires before the study begins. A protocol that has worked well in personal experience is as follows: one week before a scheduled interview, e-mail the participant a list of questions and an informed consent form, the latter of which they are to review, sign, and return prior to the interview; subsequently, share not only a password-protected Zoom link, but a telephone number in the event of an Internet glitch. Excellent organization and transparent communication make for a fluid, relatively error-free experience. Such strategies also benefit scholars collecting quantitative data prior to, during, or after interviews.

Synchronous Online Interviews

Among the benefits of online interviewing is its capacity to expand geographic scope; as Biedermann (2018) wrote, “For researchers, the world has opened up exponentially over the past two decades” (p. 26). Whereas, some interviews encompass one meeting instance per participant, others entail many interviews with a participant over time. Kamei et al. (2021), for instance, detailed a fully longitudinal mixed methods case study encompassing telenursing interviews over more than 24 months (to include the pandemic period).

We advise attending to participants’ temporal and spatial preferences. An interview that spans multiple time zones may require of the researcher an early morning or late evening session. Quiet, unencumbered places to talk are likely favorable, though for individuals in caretaking roles they may not be easily arranged. Some researchers use scheduling software (e.g., Calendy, Doodle) to set up online interviews (Lupton, 2020); these adjust for time zones and streamline logistics.

Synchronous Virtual Focus Groups

Synchronous virtual focus groups are, like face-to-face focus groups, opportune spaces to observe interactions among diverse or homogeneous persons in the context of research questions. Among the challenges of hosting virtual focus groups is attending to participant talk-time absent subtler and overt cues (e.g., eye contact) easily provided in face-to-face encounters. To manage logistical challenges, we recommend taking participants through a robust orientation to procedures, such as for signaling a wish to contribute (e.g., clicking a hand-raising icon), that would emphasize the researcher’s role as both interviewer and facilitator. Reflecting the advice of multiple scholars, Merriam (2009) suggested six to 10 participants per group.

Asynchronous Interviews and Focus Groups

Asynchronous interviews and virtual focus groups are growing in practice. Flexibility is among their benefits: neither interviewer nor participant need to be available at the same time; responses can occur in accord with a negotiated schedule. Asynchronous techniques provide opportunities for researchers to attend to tone, style, and affect—not just in participants’ responses, but in written communication. Debenham (2007) borrows from conversational interviewing techniques, and is influenced by theories emphasizing empathy, encouragement, and non-judgment. Indeed, how one phrases and responds, absent nonverbal cues and a face-to-face context, is vital in creating a trusting space (Ratislovavá & Ratislav, 2014). Vine Foggo et al. (2020) propose that care in setting up the space for discussion (e.g., posting a welcome message) may facilitate rapport. They furthermore recommend that methodologists critically evaluate how tonality differs geographically, cross-culturally, and for disabled persons.

Asynchronous Online Interviews

Introduced as “epistolary interviews” by Debenham (2007), and reviewed in Lupton (2020), asynchronous online interviews are virtual exchanges, usually through e-mail or a messaging. Debenham (2007) observes that an asynchronous protocol afforded her interview participants, some with chronic health conditions subject to daily fluctuation, time “to read, digest and reflect on the questions posed and prepare their responses at leisure, when they were feeling freshest” (p. 1). It moreover provides an immediate text record, precluding the need for transcription. Ratislovavá and Ratislav (2014), who used this method to study the grieving process in Czech women experiencing perinatal loss, underscored the potential therapeutic effect of the epistolary process. They noted challenges, among them the lack of “nonverbal and paralinguistic cues” (p. 458) that can be helpful in face-to-face contexts to facilitate connection.

Asynchronous Virtual Focus Groups

Im and Chee (2012) observe that asynchronous virtual focus groups are among the most widely used qualitative methods. Early documentation by Murray (1997) indicates that they were initially conducted via automated mailing lists; today, they are commonly hosted within online forums. Biedermann (2018) used this method in a private Facebook group to explore the experiences of deployed military spouses: after recruiting participants and receiving consent, the researcher posts a question to the forum; participants respond, to the researcher and each other, until saturation is attained or responses dwindle, after which a new question is posted. This method maximizes participant flexibility, “further empowering the population under study” (p. 28).

Creative Qualitative Methods

A number of methods are available that share qualities with arts-based research, and are amenable to online implementation. According to McNiff (2008), “knowing through the arts takes place in ways that are distinctly different yet complementary to more logical cognition” (p. 3). In a seeming echo of the transformative perspective, Gerber et al. (2020) assert that arts-based research involves “artistic integration in order to activate knowledge around the necessity to address social constructs, activism, and advocacy” (p. 10).

Multimedia Elicitation

By this method, participants utilize a smartphone or other recording device to document, in the format of digital photos, video, and voice recordings, their everyday activities (Lupton, 2020). Individuals relay their experiences artistically and longitudinally. Prompts may be co-constructed with participants, “to promote dialogue and potentially introduce new dimensions to the research that the researcher had not considered” (Bates et al., 2017, p. 461), making such methods more participant-directed than traditional interviews. As an MMR example, Peroff et al. (2020) combines “ranking of landscape photographs with photo interviews to investigate place attachment” (p. 381). They observe that multimedia elicitation is effective at transcending researcher-participant barriers, like literacy and socioeconomic status; such transcendence would extrapolate to online research.

The sensory richness of multimedia elicitation, and arts-based methods generally, adds resonance to the primacy of ethical considerations detailed earlier. Bates et al. (2017) advises being transparent about power balances and to tend carefully to consent when selecting photos, keeping mind of their nature. Philosophical grounding is critical; within an MMR context, elicitation is to be “driven and understood much more from the participant’s perspective” (p. 468), relative to researcher-driven processes and postpositivist analysis.

Digital Diaries

Digital diaries overlap with multimedia elicitation, but focus on writing as the core medium. Lupton (2020) noted that digital diaries may range from unstructured to structured and may include photographic, visual, and artistic images. A diarist may write on a regular or irregular schedule, or episodically, in response to events. Further, diaries may be used privately by participants within a broader study, left unshared with the researcher, or they may be shared. Mammen et al. (2019) elicited daily digital diaries from teenagers with asthma; concurrently, they evaluated symptoms and patterns quantitatively. The worth of this study includes a “better understanding of existing symptom patterns and how these are perceived by teens” (p. 300).

Collaborative Autoethnography

An innovative technique integrating autoethnography with teamwork, collaborative autoethnography (Chang et al., 2013) invites researchers to explore their own and each other’s experiences of a phenomenon; they “work in community to collect their autobiographical materials and to analyze and interpret their data collectively” (pp. 23–24). An advantage of collaborative autoethnography, particularly during crises, is its methodological flexibility: co-inquirers might use any combination of videos, photographs, drawings, poetry, diaries, or interviews, and exchange responses digitally. Since researchers interrogate their own lived experiences, informed consent may be unnecessary (Chang et al., 2013). An example of the use of collaborative autoethnography in MMR appears in the next section.

MMR Applications

We turn to several examples, traditional and creative, of MMR studies that showcase well-conceptualized online qualitative strands. Each presents a poignant ethical or philosophical model, and attends to principles discussed.

Asynchronous Virtual Focus Groups

The exploratory MMR study by Ruihley and Hardin (2011) is noted for its depiction of a transparent, ethically-contemplated six-step process utilized to study motivations related to participation in fantasy sports. First, the authors created an online blog to house three respective virtual focus groups; next, they recruited participants via e-mail, placing them into groups. The authors then shared detailed instructions via e-mail, followed by a post to which participants responded, using pseudonyms, in a discussion-style format; from there, the authors added questions across a five-day time period. The groups ended with expressions of the authors’ gratitude, demonstrating the centrality of participants’ contributions. Qualitative themes, once coded, helped the researchers finesse items for an instrument that was later deployed.

Collaborative Autoethnography

Explicitly guided by a transformative ethos, Manyweathers et al. (2019) used MMR to explore the congruence between gender equity policies and their own experiences of gender at an Australian university. Strategically, the researcher-participant team chose collaborative autoethnography, reflecting an ethical obligation to analyze power relations and center lived experiences. Through journaling, interwoven with weekly virtual meetings, the team chronicled “interactions, observations and reflections on their engagement with [a policy designed to promote equity]” (p. 115); concurrently, they examined de-identified institutional data related to the policy’s actual use. The results elucidated the potency of gender equity policies, while expressing nuances and limitations requiring further action (Manyweathers et al., 2019).

Passive Online Observation

As another MMR exemplar, Colditz et al. (2019) studied vaping culture; they analyzed 12-hours of Twitter data, amounting to N = 5,149 “tweets,” collected live on World Vaping Day. The researchers argue that Twitter data are suited for MMR since “(a) the data sets can be extremely large, lending well to quantitative analyses and (b) the textual content itself can be highly unstructured, which is appropriate for qualitative syntheses” (p. 200). The team embraced a phenomenological approach, situating the study in a philosophical stance that invites “a more ‘essential’ understanding” (p. 198) of vaping culture. For ethical and analysis-design purposes, they de-identified Twitter handles (i.e., user-chosen names) prior to coding.

Epistolary Approach: Radio-SMS Messaging

Published in JMMR’s special edition on COVID-19 and MMR is a piece by Riha et al. (2021) elucidating the promise of an interactive radio-SMS approach for social research amidst crisis—particularly in low and middle income countries (LMICs). The authors described a study conducted in the wake of the Somalian cholera outbreak; to ascertain community perceptions of risk and preparedness, “qualitative and quantitative data…were collected from SMS responses texted in by radio audiences” (p. 307). Respondents were not charged for SMS use; a cascading question structure, common to the epistolary approach, also collected consent, demographic information, and follow-up data. From a transformative gaze, the approach invites perspectives from “more vulnerable populations such as internally displaced persons” (p. 321). Further, it circumvents the access issues prominent in face-to-face inquiry (Riha et al.).

Discussion

Though global disasters are not new, what is novel to the current crisis is the interface between a malignant virus that separates, divides, and isolates and a virtual medium that can achieve the opposite. In response, this article is partly a showcase of promising online tools that would be used in MMR; at the same time, it is a study of a context in which those tools would be used, one that is relevant to the implications of COVID-19. This is to reiterate that context has dictated the scope of our study, which invites forth the transformative paradigm as a potential means to address, at least partially, the “host of challenges” that Fetters and Molina-Azorin (2020, p. 1) connect to the pandemic, and as a guiding framework for ethical decision-making.

We, the authors, nevertheless expect that online methods supported by philosophical and ethical considerations similar to those discussed here would be widely valuable. After all, few, if any, of the inequities and vulnerabilities exposed by COVID-19 were absent prior; in other words, the pandemic is not a source of inequity and vulnerability, but an influence in making these more severe. Relevant social context was present before the pandemic, and is likely to remain so; contexts paralleling that worsened by the pandemic are extant in realms seemingly distinct. This said, we do not attempt to delineate this article’s scope of applicability, except to suggest that our recommendations for using online methods with deliberate consideration to ethical practice is likely to be broadly applicable to inquiry under embrace of a transformative position.

We have offered in this article a few handholds for potential further extrapolation of the specific ideas discussed. Though our path coalesces with the concerns of social justice, human rights, and other assumptions of the transformative paradigm, we have provided in this article a context for philosophical positions—situated in long-standing debates within MMR—that cultivate appreciation of the crucial influence of philosophical vision when devising ethical principles and implementing research practice. Having also touched on issues of research design, this article offers support for integrating qualitative and quantitative methods. We hope that methodologists find ideas that would allow them to be nimble, creative, and deliberately ethical when formulating and implementing MMR studies.

Contribution to Mixed Methods Research

This article presents pathways for conducting online qualitative research strands that are philosophically grounded, overtly ethical, methodologically dexterous, and virtually adaptive. The inquiry described is borne out of recognizing deep inequities exposed or worsened by the COVID-19 pandemic, and a corresponding increase in the reliance of MMR researchers on online tools. Our attention to philosophical matters holds up the transformative paradigm as potentially helpful in addressing the challenges of conducting research amid crisis, though other paradigms need not be forgotten. We advise that use of online tools would come with awareness of: ease, access, and cost savings provided by online technology alongside unequal access to these benefits; disembodied aspects of virtual interactions; vulnerability of participants, researchers, and research sites; opportunities to create safe environments for participants; and influences of researcher positionality. Our consideration of ethics, anchored by the transformative paradigm, highlights anonymity as double-edged; it also underscores the delineation between private and public spaces, issues connected to membership versus non-membership, and hurdles that arise in online space to upholding the principle of avoiding harm. We invite the community of reflective methodologists to explore, deepen, and expand the array of active, passive, synchronous, asynchronous, traditional, and creative online methods featured in this article to effectuate transformative methodological approaches that also advance the MMR field.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Cara Meixner

Dan J. Spitzner

References

Archibald

M. M.

Ambagtsheer

R. C.

Casey

M. G.

Lawless

(2019). Using Zoom videoconferencing for qualitative data collection: perceptions and experiences of researchers and participants. Journal of Qualitative Methods, 18, 1–8. https://doi.org/10.1177/1609406919874596

Babcock

. (2020, May 1). Introduction: Fieldwork in a time of coronavirus. anthrodendum. [blog Post]. Retrieved from https://anthrodendum.org/2020/05/01/introduction-fieldwork-in-a-time-of-coronavirus-new-series

Bates

E. A.

McCann

J. J.

Kaye

L. K.

Taylor

J. C.

(2017). Beyond words: A researcher’s guide to using photo-elicitation in psychology. Qualitative Research in Psychology, 14(4), 459–481. https://doi.org/10.1080/14780887.2017.1359352

Biedermann

(2018). The use of Facebook for virtual asynchronous focus groups in qualitative research. Contemporary Nurse, 54(1), 26–34. https://doi.org/10.1080/10376178.2017.1386072

Bouchard

K. L

(2016). Anonymity as a double-edged sword: reflecting on the implications on online qualitative research in studying sensitive topics. The Qualitative Report, 21(1), 59–67. Retrieved from https://nsuworks.nova.edu/tqr/vol21/iss1/5

Bradbury-Jones

Isham

(2020). The pandemic paradox: The consequences of COVID-19 on domestic violence. Journal of Clinical Nursing, 29(13-14), 2047–2049. https://doi.org/10.1111/jocn.15296

Bryman

(2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10.1177/2345678906290531

Buchanan

E. A.

(2000). Ethics, qualitative research, and ethnography in virtual space. Journal of Information Ethics, 9(2), 82–87. https://pubmed.ncbi.nlm.nih.gov/12530453/

Casale

(2020). COVID-19: Can this crisis be transformative for global health? Global Public Health, 15(11), 1740–1752. https://doi.org/10.1080/17441692.2020.1811366

10.

Chang

Ngunjiri

F. W.

Hernandez

K.-A. C.

(2013). Collaborative autoethnography. Taylor and Francis.

11.

Cherryholmes

C. H.

(1999). Reading pragmatism. New York, NY: Teachers College Press.

12.

Clarke

(2005). Situational analysis: Grounded theory after the postmodern turn. SAGE Publications.

13.

Colditz

J. B.

Welling

Smith

N. A.

James

A. E.

Primack

B. A.

(2019). World Vaping Day: contextualizing vaping culture in online social media using a mixed methods approach. Journal of Mixed Methods Research, 13(2), 196–215. https://doi.org/10.1177/1558689817702753

14.

Creamer

E. G.

(2017). An introduction to fully integrated mixed methods research. SAGE Publications.

15.

Creswell

J. W.

(2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). SAGE Publications.

16.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.

17.

Crotty

(1998). The foundations of social research: Meaning and perspective in the research process. SAGE Publications.

18.

Debenham

(2007). Epistolary interviews on-line: A novel addition to the researcher’s palette. York, UK: TechDisc. Retrieved from http://www.debenham.org.uk/personal/Epistolary_Interviews_On-line.pdf

19.

Denzin

N. K.

(2010). Moments, mixed methods, and paradigm dialogs. Qualitative Inquiry, 16(6), 419–427. doi: 10.1177/1077800410364608. https://doi.org/10.1177/1077800410364608

20.

Denzin

N. K.

(2012). Triangulation 2.0. Journal of Mixed Methods Research, 6(2), 80–88. https://doi.org/10.1177/1558689812437186

21.

Eysenbach

Till

J. E.

(2001). Ethical issues in qualitative research on internet communities. British Medical Journal, 323(7321), 1103–1105. https://doi.org/10.1136/bmj.323.7321.1103

22.

Fetters

M. D.

Molina-Azorin

J. F.

(2020). Call for papers for a special issue on COVID-19 and novel mixed methods methodological approaches during catastrophic social changes. Journal of Mixed Methods Research, 14(3), 281–287. https://doi.org/10.1177/1558689820920098

23.

Foggo

V. R. S.

Webster

A. A.

Dixon

(2020). Utilisation of an on-line forum to engage adolescents with autism in direct participation in qualitative research. British Journal of Special Education, 47(2), 208–229. https://doi.org/10.1111/1467-8578.12305

24.

Franzke

A. S.

Bechmann

Zimmer

Ess

The Association of Internet Researchers . (2020). Internet research: Ethical guidelines 3.0. AOIR. Retrieved from https://aoir.org/reports/ethics3.pdf

25.

Gao

Sai

(2021). Opposing the toxic apartheid: The painted veil of the COVID-19 pandemic, race and racism. Gender, Work, & Organization, 28(suppl 1), 183–189. https://doi.org/10.1111/gwao.12523

26.

Gerber

Biffi

Biondo

Gemignani

Hannes

Siegesmund

(2020). Arts-based research in the social and health sciences: Pushing for change with an interdisciplinary global arts-based research initiative. Forum Qualitative Sozialforschung, 21(2), 1–15. https://doi.org/10.17169/fqs-21.2.3496

27.

Goh

K. K.

M.-L.

Jou

(2020). Impact of COVID-19 pandemic: Social distancing and the vulnerability to domestic violence. Psychiatry and Clinical Neurosciences, 74(11), 612–613. https://doi.org/10.1111/pcn.13130

28.

Graber

(2020). Guidance on conducting and supervising community-oriented psychology research during COVID-19. Elsevier. Retrieved from https://cris.brighton.ac.uk/ws/portalfiles/portal/12025345/Guidance_on_conducting_community_oriented_psychology_research_during_COVID_19.pdf

29.

Graham

(2020, November 17). The human costs of the pandemic. Brookings Institution. Retrieved from https://www.brookings.edu/research/the-human-costs-of-the-pandemic-is-it-time-to-prioritize-well-being

30.

Greene

J. C.

(2007). Mixed methods in social inquiry. Jossey-Bass.

31.

Greene

J. C.

Caracelli

V. J.

(1997). Defining and describing the paradigm issue in mixed-method evaluation. In Greene

J. C.

Caracelli

V. J.

(Eds.), Advances in mixed-methods evaluation: The challenges and benefits of integrating diverse paradigms (pp. 5–17). Jossey-Bass.

32.

Greene

J. C.

Hall

(2010). Dialectics and pragmatism: Being of consequence. In Tashakkori

Teddie

(Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd ed., pp. 119–143). SAGE Publications.

33.

Gregory

(2018). Online communication settings and the qualitative research process: Acclimating students and novice researchers. Qualitative Health Research, 28(10), 1610–1620. https://doi.org/10.1177/1049732318776625

34.

Guba

Lincoln

Y S.

(2000). Paradigmatic controversies, contradictions, and emerging confluences. In Denzin

Lincoln

Y. S,

(Ed.), The handbook of qualitative research (2nd ed., pp. 1065–1122). SAGE Publications.

35.

Guba

Lincoln

Y. S

(2005). Paradigmatic controversies, contradictions, and emerging confluences. In Denzin

Lincoln

Y. S,

(Ed), SAGE handbook of qualitative research (pp. 191–216). SAGE Publications.

36.

Hall

Howard

(2008). A synergistic approach: Conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2(3), 248–269. https://doi.org/10.1177/1558689808314622

37.

Hesse-Biber

(2010). Qualitative approaches to mixed methods practice. Qualitative Inquiry, 16(6), 455–468. https://doi.org/10.1177/1077800410364611

38.

Illingworth

(2006). Context, content, reflexivity, and the qualitative research encounter: Telling stories in the virtual realm. Sociological Research Online, 11(1), 1–12. https://doi.org/10.5153/sro.1216

39.

E. O.

Chee

(2012). Practical guidelines for qualitative research using online forums. Computers, Informatics, Nursing, 30(11), 604–611. https://doi.org/10.1097/NXN.0b013e318266cade

40.

Jowett

(2020, April 20). Carrying out qualitative research under lockdown—practical and ethical considerations. LSE. [blog Post]. Retrieved from https://blogs.lse.ac.uk/impactofsocialsciences/2020/04/20/carrying-out-qualitative-research-under-lockdown-practical-and-ethical-considerations

41.

Kamei

Kawada

Nishimura

Kakai

(2021). A fully longitudinal mixed methods case study design: An example based on the illness trajectory of a person with chronic conditions in home-monitoring-based telenursing during the COVID-19 pandemic. Journal of Mixed Methods Research, 15(3), 413–436. https://doi.org/10.1177/15586898211019512

42.

Kuhn

T. S.

(1962). The structure of scientific revolutions (1st ed.). University of Chicago Press.

43.

Lupton

(2020). Doing fieldwork in a pandemic. ResearchGate. [crowdsourced document]. Retrieved from https://docs.google.com/document/d/1clGjGABB2h2qbduTgfqribHmog9B6P0NvMgVuiHZCl8/edit?ts=5e88ae0a

44.

Madani

. (2020, June 14). Therapists are under strain in COVID-19 era, counseling clients on trauma they’re also experiencing themselves. NBC News. Retrieved from https://www.nbcnews.com/news/us-news/therapists-are-under-strain-covid-era-counseling-clients-trauma-they-n1230956

45.

Mahajan

Kapoor

Prabhakar

(2020). Psychological effects of COVID-19 on children of health care workers. Anesthesia & Analgesia, 131(3), Article e169–e170. https://doi.org/10.1213/ANE.0000000000005034

46.

Major

(2020). Unprecedented times and innovation. Innovative Higher Education, 45(6), 435–436. https://doi.org/10.1007/s10755-020-09528-4

47.

Mammen

J. R.

Java

J. J.

Rhee

Butz

A. M.

Halterman

J. S.

Arcoleo

(2019). Mixed-methods content and sentiment analysis of adolescents’ voice diaries describing daily experiences with asthma and self-management decision-making. Clinical & Experimental Allergy, 49(3), 299–307. https://doi.org/10.1111/cea.13250

48.

Manyweathers

Lymn

Rurenga

Murrell-Orgill

Cameron

Thomas

(2019). The lived experience of gender and gender equity policies at a regional Australia university. Social Sciences, 9(115), 1–11. https://doi.org/10.3390/socsci9070115

49.

Maxwell

Loomis

(2003). Mixed methods design: An alternative approach.In Tashakkori

Teddie

(Eds.), Handbook of mixed methods in social & behavioral research (p. 241–272). SAGE Publications.

50.

McNiff

(2008). Art-based research. In Knowles

J. G.

Cole

(Eds.), Handbook of the arts in qualitative research: Perspectives, methodologies, examples, and issues (pp. 29–41). SAGE Publications.

51.

Merriam

(2009). Qualitative research: A guide to design and implementation. Jossey-Bass.

52.

Mertens

Fraser

Heimlich

(2008). M or F?: Gender, identity, and the transformative research paradigm. Museums & Social Issues, 3(1), 81–92. https://doi.org/10.1179/msi.2008.3.1.81

53.

Mertens

D. M.

(2003). Mixed methods and the politics of human research: The transformative-emancipatory perspective. In Tashakkori

Teddie

(Eds.), Handbook of mixed methods in social & behavioral research (pp. 135–164). SAGE Publications.

54.

Mertens

D. M.

(2009). Transformative research and evaluation. Guilford Press.

55.

Mertens

D. M.

(2012). What comes first? the paradigm or the approach? Journal of Mixed Methods Research, 6(4), 255–257. https://doi.org/10.1177/1558689812461574

56.

Mertens

D. M.

(2018). Mixed methods design in evaluation. SAGE Publications.

57.

Mertens

D. M.

Tarsilla

(2015). Mixed methods evaluation. In Hesse- Biber

S. N.

Johnson

R. B.

(Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 426–446). Oxford Press. https://doi.org/10.1093/oxfordhb/9780199933624.013.27

58.

Meixner

Spitzner

D. J.

(2021). Mixed methods research and social inclusion. In P. Liamputtong (Ed.), Handbook of Social Inclusion. Springer. https://doi.org/10.1007/978-3-030-48277-0_19-1

59.

Mitra

Cohen

(1999). Analyzing the web: Directions and challenges. In Jones

(Ed.), Doing internet research: Critical issues and methods for examining the net (pp. 179–202). SAGE Publications.

60.

Morgan

D. L.

(2007). Paradigms lost and pragmatism regained. Journal of Mixed Methods Research, 1(1), 48–76. https://doi.org/10.1177/2345678906292462

61.

Murray

P. J.

(1997). Using virtual focus groups in qualitative research. Qualitative Health Research, 7(4), 542–549. https://doi.org/10.1177/104973239700700408

62.

Nastasi

Hitchcock

Brown

(2010). An inclusive framework for conceptualizing mixed methods design typologiesMoving toward fully integrated synergistic research models. In Tashakkori

Teddie

(Eds.), Handbook of mixed methods in social & behavioral research (pp. 305–338). SAGE Publications.

63.

Oderberg

D. S.

(2012). Disembodied communication and religious experience: the online model. Philosophy & Technology, 25(3), 381–397. Retrieved from https://link.gale.com/apps/doc/A373886488/AONE?u=anon∼cfffe340&sid=googleScholar&xid=266d9fac

64.

Orgad

(2005). The transformative potential of online communication. Feminist Media Studies, 5(2), 141–161. https://doi.org/10.1080/14680770500111980

65.

Peroff

D. M.

Morais

D. B.

Seekamp

Sills

Wallace

(2020). Assessing residents’ place attachment to the Guatemalan Maya landscape through mixed methods photo elicitation. Journal of Mixed Methods Research, 14(3), 379–402. https://doi.org/10.1177/1558689819845800

66.

Qureshi

. (2020, November 17). Tackling the inequality pan- demic: Is there a cure? Brookings Institution. Retrieved from https://www.brookings.edu/research/tackling-the-inequality-pandemic-is-there-a-cure

67.

Ratislovavá

Ratislav

(2014). Asynchronous email interview as a qualitative research method in the humanities. Human Affairs, 24(3), 452–460. https://doi.org/10.2478/s13374-014-0240-y

68.

Riha

Abreu Lopes

Abdi Ibrahim

Srinivasan

(2021). Media and digital technologies for mixed methods research in public health emergencies such as COVID-19: Lessons learned from using interactive radio-SMS for social research in Somalia. Journal of Mixed Methods Research, 15(3), 304–326. https://doi.org/10.1177/1558689820986748

69.

Roberts

L. D.

(2015). Ethical issues in conducting qualitative research in online communities. Research in Psychology, 12(3), 314–325. https://doi.org/10.1080/14780887.2015.1008909

70.

Rogers

. (2015). How online qualitative research can drive richer results. Marketing Insights, 27(4), 32–37. ISSN:1040–8460.

71.

Ruihley

B. J.

Hardin

R. L.

(2011). Beyond touchdowns, homeruns, and three-pointers: An examination of fantasy sport participation motivation. International Journal of Sport Management and Marketing, 10(3/4), 232–256. https://doi.org/10.1504/IJSMM.2011.044792

72.

Sanchez-Rivera

(2020). The legacies of ‘race’ science, anti-Chinese racism, and COVID-19 in Mexico. Bulletin of Latin American Research, 39(suppl 1), 35–38. https://doi.org/10.1111/blar.13173

73.

Seymour

W. S.

(2001). In the flesh or online? Exploring qualitative research methodologies. Qualitative Research, 1(2), 147–168. https://doi.org/10.1177/146879410100100203

74.

Shannon-Baker

(2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. https://doi.org/10.1177/1558689815575861

75.

Sharifimonfared

Hammersley

(2020). Harm reduction and quitting techniques used by heavy MDMA (ecstasy) users. Addiction Research & Theory, 28(3), 222–230. https://doi.org/10.1080/16066359.2019.1622684

76.

Smith

M. L.

(1997). Mixing and matching: Methods and models. In Greene

J. C.

Caracelli

V. J.

(Eds), Advances in mixed-methods evaluation: The challenges and benefits of integrating diverse paradigms (pp. 73–85). Jossey-Bass.

77.

South

Stansfield

Amloˆt

Weston

(2020). Sustaining and strengthening community resilience throughout the COVID-19 pandemic and beyond. Perspectives in Public Health, 140(6), 305–308. https://doi.org/10.1177/1757913920949582

78.

Synnot

Hill

Summers

Taylor

(2014). Comparing face-to-face and online qualitative research with people with multiple sclerosis. Qualitative Health Research, 24(3), 431–438. https://doi.org/10.1177/1049732314523840

79.

Vogels

E. A.

Perrin

Rainie

Anderson

(2020, April 30). 53% of Americans say the internet has been essential during the COVID-19 outbreak. Pew Research Center. Retrieved from https://www.pewresearch.org/internet/2020/04/30/53-of-americans-say-the-internet-has-been-essential-during-the-covid-19-outbreak

80.

Watson

(2020). Methods braiding: A technique for arts-based and mixed-methods research. Sociological Research Online, 25(1), 66–83. https://doi.org/10.1177/1360780419849437

81.

P. E.

Styra

Gold

W. L.

(2020). Mitigating the psychological effects of COVID-19 on health care workers. Canadian Medical Association Journal, 192(17), E459–E460. https://doi.org/10.1503/cmaj.200519