What Would Be Really Helpful but Nobody Ever Tells You: Five Key Recommendations Derived From Lessons Learned During a Qualitative Study in Clinical Settings

Overview of project

Ours was a multidisciplinary study by researchers from four academic institutions and three clinical settings located within two health-care institutions (see Davies et al., 2017 for details). We learned from previous research and clinical practice that improving our understanding about positive interactions between HCPs and parents would contribute to optimizing pediatric palliative care for children and families. Thus, we aimed to develop a grounded theoretical conceptualization of best practices in HCP–parent interactions. With ethics approval, we interviewed 80 HCPs (across disciplines) and 34 parents in oncology, neonatal intensive care (NICU), and a children’s hospice, along with scheduled and ethnographic observations and informal conversations.

Using QSR NVivo 10 to manage our data, we followed grounded theory procedures for analysis. HCP and parent interviews were analyzed separately, initially by three researchers experienced in qualitative research achieving consensus on the codes, their relationships, and the emerging conceptualization, and with input from all other team members. Analysis was iterative from the start, as was our research journey—the ongoing reflections about our process that we recorded in our meeting notes enabled us to identify and describe five key recommendations that we share in this paper to help others as they develop and conduct qualitative research.

Recommendation 1: Conduct a pilot study

Pilot studies tend to be associated with positivist methodological approaches to research. They are seldom discussed or reported in qualitative approaches though they may be used in an informal, non-systematic way (Sampson, 2004). Morse (1997) argued that pilots are really the beginning of the project itself and as such are not true pilots. Yet, conducting a pilot study can be relatively inexpensive ($4000 for our project) and offer opportunities for adjustments and revisions prior to undertaking the main study (Kim, 2011; Sampson, 2004; Van Teijlingen & Hundley, 2002), thus contributing to the project’s success. Particularly when conducting studies in health-care settings and about sensitive topics, the value of obtaining buy-in from key stakeholders, presenting to research committees for input early to avoid mistakes, and establishing relationships within the clinical settings cannot be overemphasized. A pilot can also aid in refining a proposal and may convince the funder that achieving the goal is in fact feasible.

Our study was prospective. We knew that gaining permission at an administrative level for access to health-care settings does not necessarily translate into cooperation on the ground. So before approaching the “powers that be” (and, importantly, to find out who these powers actually were), we realized that it was expedient to contact people who could get things done. We also understood that from an organizational and logistical perspective, this person may be the unit physician. We had already worked with a physician from one site on previous research projects and he helped us identify a physician in each of the other settings who had the ability to influence and engage colleagues and would give the project credibility. These three physicians then became co-investigators (Co-Is) and facilitated entry to the field and access to resources within their respective departments. As part of the pilot, they toured the principal investigator (PI) (BD) and the eventual research analyst (GK) through their respective settings. At that time, we outlined the study’s purpose, proposed a recruiting and consenting process, discussed analytical features and ethics, and suggested preliminary logistics and workflow ideas. During the full study, these Co-Is successfully kept the doors and the lines of communication open.

The pilot helped us learn about each setting’s research environment and key people, for example, research coordinators (RCs). It enabled us to question and reflect on the project design, particularly recruitment procedures, and alerted us to potential challenges or requirements in each setting. Over a few days, we observed the physician Co-Is in interactions with parents and we chatted with these same parents. We also met other clinicians and managers, and the RCs, in all three settings. Everyone was enthusiastic and offered considerable feedback on how we might proceed, whom else to involve, and how best to shape and frame our design. They connected us to family advisory groups that we subsequently included in our study design, and they contributed to our more detailed and systematic understanding of the issues involved in conducting the project. For example, both NICU and pediatric oncology had numerous ongoing research projects; as part of that larger group, we would be required to fulfill certain additional obligations. Without the pilot, we would not have known to build in the time and other resources that were needed.

A pilot can facilitate both formal and informal access for conducting the larger study but does not usually result in finalized procedures because change is frequent within health-care environments. So, some of what we initially learned was different from what happened when we started the project. For example, we knew that only a research nurse (RN) could recruit families in the NICU, but we had not known that the unit also required that only an RN, not a research assistant (RA), collect data. Consequently, we had to hire an RN, which resulted in additional costs not originally in our budget. Although it was frustrating to be suddenly faced with unexpected challenges and revisions, our collaboration with clinicians facilitated a workable revised plan. Researchers must be ready to adapt and fit into the setting, particularly for large university-affiliated teaching institutions. However, while flexibility is essential, any changes must link back to the goals of the study.

Finally, adequate funding for pilot studies is likely to be available as many universities, and even some research centres based in clinical agencies, have funding sources available. We obtained the funds through the Research Facilitation Office at the University (XX) where faculty can apply for small facilitation grants to produce a research proposal for a national funding agency. Researchers may need to search for similar funds, but the value of a pilot makes the search worth the time and effort.

Recommendation 2: Hire a research analyst

The practice of hiring a statistician for a quantitative study is unquestioned. It is commonly agreed that a statistician’s specialized knowledge can supplement the researcher’s expertise, enhance the integrity and validity of the study (which in qualitative work translates into such issues as auditability, credibility, and fittingness), and is especially helpful for large volumes of data. Having a comparable person for our qualitative study made sense to us for similar reasons, especially given the anticipated complexity and volume of data. In the end, our project generated a large volume of data that created challenges in administrative, logistical, organizational, and analytical domains. We analyzed approximately 1780 pages of interviews and then later added follow-up verification interviews. The project generated close to 700 pages of field notes and almost 1000 pages of aggregated quotes as we completed analysis.

Hiring a research analyst may be one solution to successfully managing the challenges of a large qualitative data set. Data management software is essential and requires that the data analyst have expertise in using it. Too often qualitative researchers learn how to use software during the course of the research project itself. This concurrent approach is unsatisfactory as too much time is spent on learning how to manage the data rather than concentrating on the more analytical thinking that is needed for in-depth analyses.

A research analyst does not need a background in the substantive aspects of the study; such knowledge is the domain of the PI who is responsible for engaging in the conceptual analysis and interpretation of the data. But a research analyst must have a specialized body of knowledge and experience; the role is not one for an RA. Our research analyst (GK) had considerable expertise with NVivo, both as a trainer and as a researcher with a variety of projects. To our knowledge, this is the first time a qualitative project has used a research analyst in this way. Thus, until the role becomes more common, it may be difficult to find a qualified person to fill the role. One suggestion is to contact software program developers to find individuals who teach the software.

Managing a large data set requires considerable discussion within a team. An expert research analyst will facilitate the processes of developing identification codes; a system for labelling transcripts, field notes, and other data; and a tracking system for ongoing work. She or he can quickly develop a workflow to establish continual enrolment, data collection, analysis, and feedback to everyone concerned in a smooth rhythm that could itself be continually evaluated. For example, at each weekly meeting we discussed progress with enrolment, where to sample and target participants, what had been accomplished, and any issues or concerns, along with an update on emerging analysis and issues.

While the PI focuses on start-up issues, an analyst can begin the coding process immediately an interview is completed. Ideally, the analyst should transcribe the interviews to keep close to the data because hearing the voices of the informants provides an immediate and authentic connection to the original content. However, the number of interviews may outpace the analyst’s ability to transcribe and code the interviews; a transcriptionist may also be needed. Given that a hallmark of qualitative research is the iterative cycle of inquiry (Marshall & Rossman, 2010), continuing to conduct interviews when prior interviews have not been transcribed and analyzed puts the researcher at risk of gathering redundant material. Although we are aware that the reality of many qualitative research projects does not allow for concurrent data collection and analysis, we strongly support this approach.

The analyst can also develop a specific workflow for interview transcripts. Our interviewers de-identified study transcripts and the PI and Co-PI (RS) then reviewed them. On all but the latter third of every interviewer’s work, the PI wrote comments on the transcript about the content, as well as the strategies used during the interview, as a way of mentoring the interviewers. The research analysis entered each de-identified interview and the transcript with comments into NVivo for analysis. He then began coding and made related comments in a separate memo for each participant.

Coding structures can become extensive in qualitative studies and need careful management, so we added a regular agenda item to our weekly meetings that we called “evolution of codes.” The analyst led discussions about the evolving code hierarchy. As we continued with the coding structure, the agenda items became “analytical queries” that served to develop modifications and new directions for interview questions, some of which were formalized as written prompts. This analyst-led process allowed data collection and analysis to quickly develop into an iterative cycle that was extremely useful in refining and clarifying thematic content. We believe that other researchers would benefit from such a process and that analysis would then more closely follow the recommended concurrent approach.

Finally, an analyst’s extensive tracking can produce a comprehensive audit trail that adds rigour to the overall process in a way that is not generally possible. Our research analyst entered all aspects of our data, as well as key points of all discussions, into NVivo. His detailed and comprehensive database allowed us to check and re-check our decisions so that we could be confident that our final analysis truly reflected what participants were telling us.

Recommendation 3: Engage a compatible or “the right” team

Researchers frequently describe their team as “multidisciplinary,” yet the exact purpose of each team member is not always clear. We recommend that it is important to select individuals who are not only competent in their role but also compatible as team players and “fit” with the team. People who work well together contribute to a project’s success. Therefore, although a screening process to identify how well a potential team member might fit in may seem unusual, it could make a difference to the success, or not, of the project. While it is important to carefully make the team as strong as possible, at the same time all team members must be recognized in ways that help them too, for example, including the study on their resume for promotion or application for doctoral study.

In qualitative research, a researcher is often advised to seek out key players and get them on one’s side. Similarly, we recommend that researchers take the same approach when developing their research team. It is important to look beyond one’s own field and to be open to what others can bring to the study. The various members of our team brought experience from nursing, medicine, social work, counselling, anthropology, sociology, and/or public health. This diversity resulted in team members contributing valuable disciplinary insights, as well as theoretical and clinical expertise, that enhanced and expanded data analysis in ways that would not have been possible from a unitary disciplinary perspective. The different backgrounds and areas of expertise of the team members not only added depth and breadth to our purpose and analysis but also fostered learning for each of us. Importantly, similar to the best practice HCPs in our study, we all shared a common, broad worldview that enabled each of us to value individuals, nurture our curiosity, and remain committed to our goal. Therefore, we were able to engage in thoughtful, collegial discussion to clarify differences in perspectives and arrive at a common understanding despite perceived differences in power within the health-care setting.

Regardless of role, for example, being a student, an RN, a physician, or PI, every member of the team was actively encouraged to share his or her insights and every contribution was appreciated. We worked from the premise that everyone brought different strengths and skills that were equally valuable in their own way. Our way of being as a research team fit with the findings from our study that best practice HCPs treat every person as being worthy of respect and are committed to authentic engagement (Davies et al., 2017).

Varying expertise with research methods provided additional insights, for example, in identifying and understanding individual ways of thinking about the data either logically or conceptually, that allowed each of us to do what we did best. The ages of team members spanned five decades and allowed us to identify how age and experience played a role in teamwork and in our analysis. Our student ensured that what we said was clear and understandable at that level and all our clinical team members offered practical input based on their direct day-to-day experiences in the settings.

We had planned to include Punjabi- and Mandarin-speaking parents in our study not only because they represented the most significant South Asian and Chinese minorities in the geographical area but also because we had learned in our pilot that about one third of the families whose children are treated in the participating facilities speak Punjabi or Mandarin (not Cantonese, which is another major dialect group in China). Therefore, we deliberately sought RAs who could help bridge the potential cultural gap between some team members and the parents and also interview the parents in their own language. Our RN spoke Punjabi and self-identified as a member of her South Asian culture though she was raised in Canada. Our bilingual, bicultural RA was raised in China and retained family connections in the country, but had lived in Canada for some time. Although, in the end, we recruited only four (15%) Mandarin-speaking parents and one South Asian family who preferred to speak English, the parents valued having the opportunity to participate in their first language. During data collection, the RN and RA were able to explore parents’ comments to ensure they accurately identified and explained the nuances of meaning that may have been present. Further, during data analysis, the RN and RA were able to contextualize parental comments for the rest of the team and offer well-grounded cultural insights.

Having practicing clinicians on the team (physicians as Co-Is and a nurse as RN) who were all known in their particular settings and research environments offered logistical and gate-keeping advantages. The clinical team members helped pave the way for us to enter clinical settings. For example, when meeting new HCPs in a setting, we found that stating who was on our team seemed to provide some credibility because the HCP recognized a familiar person. The physician Co-Is all had at least some understanding of qualitative methods, which made them more able to appreciate the time involved in each step of the study and the importance of simultaneous data collection and analysis. In turn, the work could move along more smoothly because we did not always have to be explaining or defending our qualitative approach.

Clinician team members may also introduce some limitations. Two of our three physician Co-Is used qualitative approaches in doctoral dissertations, but more typically clinicians may lack knowledge of qualitative research needs, as they often work in a more quantitative research environment (e.g., clinical trials). Clinicians may not understand or appreciate how time-consuming qualitative methods can be and, because of their busy clinical responsibilities, it can be very difficult for them to fully participate in the research process despite their eagerness to do so. For example, our physician Co-Is were unable to attend the regular core team meetings, but participated actively in all full team meetings and offered assistance and input between meetings. Eager to make a difference in practice, there may also be premature attempts to arrive at policy development, so being aware of such possibilities is essential.

Finally, researchers may need to consider which roles are formal and institutionalized and which ones are perfunctory and administrative. Anticipation of these formal and informal networks avoids potential pitfalls and problems as the project unfolds. In clinical institutions, it is our experience that hospitals have rules about who can be a PI on an ethics application. Typically, an employee within the institution must be identified as the onsite PI, yet the funded researcher is unlikely to be an employee of the clinical site (hospital). This situation can lead to having to name an onsite PI who may not have a primary role as a researcher and/or may not hold a PhD. As a consequence of clinical site requirements, there is a potential for conflict about who leads and who is ultimately responsible for the research. Further, it could mean that the person who holds the funding and is ultimately responsible for the project may not be clearly identified as such on important documents, for example, consent forms. Having the “right” team members can help one navigate the potential challenges with this approach. In our situation, the site PI was actually a Co-I on the project and we had worked together on previous projects. Therefore, it was not difficult for us to have conversations about the specific roles of each member and particularly about who had the primary responsibility for how the study was conducted. It is important to note that our situation worked well because we had already cultivated a positive research relationship; however, not all situations may be as “easy.” Early and clear communication is crucial, which can be facilitated if a pilot study is undertaken. Roles and expectations should be navigated in a respectful way and researchers may want to consider drawing up a Memorandum of Understanding (MOU) to ensure maximal clarity. Particularly with newer partnerships, we advocate that important items, for example, the funded PI’s name and his or her role in the project being clearly articulated in documents such as ethics review and consent forms, are included in an MOU that is signed by both the researcher(s) and the clinician partner(s).

Recommendation 4: Attend to team cohesion

Development of a compatible, or “right,” team is crucial to the success of a project, but maintaining team cohesion is equally important. Each member has to feel valued as a full member of the team so we recommend that the PI in particular develops and sustains a plan to maintain team cohesion. A researcher must also recognize that team cohesion might be threatened if team members need to change during a project, for example, due to a maternity leave. Again, it is better to look for a new member with the right fit than to simply accept the first person who meets the minimal requirements.

Effective research teams build interpersonal cohesion as a deliberate strategy. For example, starting each meeting with an informal check-in on recent personal events, sharing chocolate and other treats, consoling in times of grief, and celebrating times of joy (marriages, graduations, new babies), all contribute to the active involvement of everyone and foster an esprit de corps that maintains team cohesion. Regular contact allows for understanding of a common language, makes time available to clarify potential misinterpretations that might arise because team members come from different backgrounds and have different perspectives on issues, and can facilitate engagement in mutual learning and in supporting one another. It also allows for quick and effective reaction to changes. We cannot overemphasize the value of weekly meetings and how expectations of attendance and active participation are instrumental to a project’s success.

Meetings help keep everyone on track, provide a structure for working together, and set the “rhythm” or pace of a project, which is crucial to prevent larger problems from developing and then spiralling out of control. We recommend pre-planning and circulating an agenda for all meetings, as well as setting goals for the work to be accomplished during the next week and over time to facilitate constant re-evaluation of timelines and monitoring of procedures, data collection, and analytical progress that allows for timely modifications as required to strengthen the project. We also recommend laying the foundation for team cohesion during an orientation meeting to ensure that everyone is on the same page and to set the stage for fruitful collaboration. Thereafter, schedule core team meetings at the same time each week for 1–2 h, preferably in-person but by conference call if needed. As much as possible, we recommend additional, full 1–2 day in-person meetings where the Co-Is can also join the core team. We found our in-person meetings to be exceptionally beneficial and we strongly recommend that budgets for qualitative projects include financial support for face-to-face meetings.

Recommendation 5: Conduct conceptual analysis through a process of “node expansion”

Qualitative research literature places considerable emphasis on data collection and coding or organizing of the material, usually into thematic or descriptive groupings that can then be reviewed and elevated to a conceptual level. However, we concur with Morse, Hupcey, Mitcham, and Lenz (1996) who suggested that concept development is the weakest link in the qualitative research process. There is a dearth of literature about how to elevate aggregated and conceptualized material into usable models. Most authors move from data analysis (aggregation, coding, thematic descriptions, hierarchies, splitting and merging of coded material) through to dissemination (e.g., findings, discussion) and pay only limited attention to the difficult in-between phase of producing findings that are credible, transferable, confirmable, and rigorous.

Many textbook discussions of analysis give short thrift to the step between data aggregation (coding) and dissemination of the findings. But it is the step in-between the conceptual aspect that is a labour-intensive and extremely time-consuming process, especially with a large data set. While software helps in organizing data, developing thematic codes, and aggregating these codes into hierarchical structures, it does not really help with the aspect of elevating the analysis to any kind of conceptual level. In other words, the thematic categories remain original raw data in the form of quotations from the transcripts even though they are now ordered and categorized. Additionally, the use of maps and diagrams, while facilitating the process of organizing coded data into meaningful segments, does not necessarily facilitate the in-depth analysis that is typically the goal of qualitative researchers. We developed a system whereby we opened our nodes as “expansions” to develop new insights and a comprehensive model of what was “best practice”—the goal of our study. We recommend that other researchers try out this process as a new way of conducting their conceptual analysis (see Table 1 for a step-by-step summary).

OPEN IN VIEWER

Table 1.

Summary of step-by-step process of node expansion.

Step	Actions	Use of NVivo software
1	Coded data were organized and categorized as “nodes” into NVivo. Nodes are central to understanding and working with NVivo—they let the researcher gather related material in one place so that she or he can look for emerging patterns and ideas (themes).	√
2	PI wrote descriptions about and reflected on the nodes, read excerpts that comprised the nodes to ensure understanding of the raw data, and discussed each node in team meetings.
3	Printed out each theme captured in a node.	√
4	All printed material reviewed separately by research analyst and PI who, using paper and pencil, marked up the documents with thoughts and ideas that arose while reading the material.
5	Marked comments by PI and research analyst were consolidated and conceptualized (pondering, brainstorming, drawing diagrams, conceiving metaphors) by the PI.
6	PI and research analyst rewrote concepts, themes, and comments as narrative using pertinent quotes to support the empirical evidence.	√
7	Node expansions were discussed in-depth at core team meetings with off-site Co-PI and other research team members who had read the interviews, and in less detail, with Co-Is during full team meetings.

After the initial coding, we grouped the data into “nodes.” NVivo facilitated our ability to describe and discuss each node. We found that going back to paper and pencil worked best for the next step of the process. We printed out all aggregated quotes for each thematic node and then began a review. The research analyst wrote his thoughts or ideas about each node, referring back to the original aggregated quotes for that particular node. The PI did the same and then both sets of notes were compared and combined, resulting in over 300 pages. We then explored the combined nodes during the analysis discussion in team meetings.

A significant aspect of expanding nodes was in the process of writing by hand. Neuropsychology research shows that the physical act of handwriting compared to writing on a computer engages different parts of the brain. The former increases a person’s ability to process information and to reframe that information in his or her own words to facilitate learning and retention, whereas writing on a computer results in shallower cognitive processing (Mueller & Oppenheimer, 2014). Thus, through writing by hand a researcher may be able to gain clarity and insights that are not in the themes themselves. It requires time, however, to be actively passive, to be open to inception, which is the grasping of the meaning of something, so researchers can put things in language that is understood by others.

Expanding nodes by commenting on and annotating the aggregated thematic quotes facilitated the process of building a conceptual model that was then refined, modified, structured, and continually expanded until it seemed fully formed. Technology only helped this process by storing the final written results; it certainly did not simplify the conceptual process or reduce the time involved. In fact, developing node expansions took time, which may be a problem for academics, particularly in practice disciplines, and for full-time clinicians, both of whom feel increasing pressure to do research. Yet, the process of node expansion facilitated comprehensive and exceptionally in-depth analysis that allowed us to develop a conceptual model that characterizes participants’ experience of how best practices of interaction occurs. Indeed, in our verification or “member-checking” interviews, the participants were unanimous in commenting, “Ah, I see it…yes, that’s what we do!”

Our process of expanding nodes allowed for the kind of analysis that is the gold standard in qualitative research. But the time required raises questions about the viability of qualitative research in academic and clinical settings unless ways are found to facilitate the thinking time that is needed. The incorporation of a research analyst into qualitative research projects may be one part of the solution, but clearly the issue is multi-faceted and requires continued discussion to ensure that qualitative research maintains the rigour that makes it so credible and useful.

Conclusion

We wanted to share our insights because we believe in engaging with others to enhance research practice and as experienced qualitative researchers we recognize that we continue to learn. Our experience reinforces the importance of being open to modifying a project as it unfolds to keep things on track and to refine procedures and administrative needs according to how they present themselves during the life of the project. In keeping with good qualitative practices, our experience also emphasizes the significance of maintaining a detailed audit trail of changes and modifications.

We acknowledge that conceptualization is a poorly articulated area in the qualitative research literature and that while our way is not the only way to overcome and deal with this gap, we consider it important for researchers to acknowledge, understand, and overcome the lack of a template for this important step. We believe that it is conceptualization that takes qualitative research out of the realm of “journalism” or mere description and elevates it to results that are transferable to other settings, something that should be the goal of every researcher who aspires to influence practice. Conceptualization also adds credibility because though many people can summarize and synthesize raw data, only skilled researchers can create results that are more widely applicable. We hope that the recommendations that arose from our own work will assist other researchers, as well as educators of upcoming researchers, to learn from our experiences in their own studies.