A CBPR-Enhanced Delphi Method: The Measurement Approaches to Partnership Success Case Study

The Measurement Approaches to Partnership Success (MAPS) Study

We used the Delphi method as part of the Measurement Approaches to Partnership Success (MAPS) study, a 5-year National Institutes of Health (NIH)–funded project whose overall aim is to define and assess success in long-standing CBPR partnerships. Long-standing in the MAPS study was defined as those partnerships in existence for at least 6 years, which we rationalized as continuing beyond a typical 5-year federal funding cycle (Israel et al., 2020). MAPS is being carried out by the Detroit Community-Academic Urban Research Center (Detroit URC), established in 1995, to foster CBPR partnerships aimed at understanding and addressing social determinants of health toward eliminating health inequities in Detroit (Israel et al., 2001). The Detroit URC is guided by a Board composed of members representing eight community-based organizations, two health and human service agencies, and three schools in an academic institution (see “Acknowledgments” section). Building on the Detroit URC’s long-standing CBPR approach, MAPS’s three aims are to (a) develop a clear definition of success in long-standing CBPR partnerships, (b) identify factors that contribute to success, and (c) design, test, and validate a practical measurement tool for partnerships to assess and strengthen their efforts to achieve health equity.

Delphi Stages 1 and 2

As depicted in Figure 1, we then used a mixed-methods approach to conduct a three-round Delphi process with the Expert Panel to assess the face and content validity of the instrument items. The first two rounds followed the traditional Delphi process with an online survey asking panelist members to utilize specific criteria in evaluating each of the 96 instrument items. In Round 1, panelists individually and anonymously ranked the importance, appropriateness, and clarity of the instrument items as they related to CBPR partnership success on a 5-point Likert-type scale (from very important to not very important) and assessed whether an item should be retained, modified, or removed. Items were retained if 75% or more of the Panel ranked items as very important or important for assessing each dimension of partnership success. Panelists’ qualitative comments provided a basis for considering changes such as modifying or rewording an item to improve clarity or removing items deemed redundant (Coombe et al., 2020). After reviewing Round 1 quantitative and qualitative responses, 79 of the original 96 items were retained and 33 of the 79 items were reworded. In Round 2, 1 month later, panelists were asked to rank the 79 items as “reflective” or “not reflective” of dimensions of CBPR partnership success on a 3-point Likert-type scale. After analyzing the responses from Round 2, we did not delete any of the items. Instead, we flagged items that showed variability in responses (20 items) among Expert Panelists as well as items that had qualitative comments (10 items) warranting collective discussion. Four items were also reworded.

OPEN IN VIEWER

Figure 1.

The CBPR-enhanced Delphi process with expert panel.

An Example From the MAPS Delphi Face-to-Face Meeting

In addition to helping finalize the MAPS items by looking at those items that had variability in responses from the first two rounds, the face-to-face meeting also gave the MAPS research team the opportunity to engage with the Expert Panelists in a collaborative way that included back-and-forth dialogue facilitated by a research team member with expertise in group process. At the request of the Expert Panel, we engaged in a broad conversation about equity and CBPR and the extent to which issues around equity were captured in the MAPS instrument. Described below, this unplanned discussion revealed new issues and insights that would not have been possible in a traditional Delphi process format.

Based on our key informant interviews and literature review, our research team defined Equity as “an environment has been created that: (a) enhances open, equitable, collaborative, and authentic relationships (i.e., relational equity); and (b) sharing of power and resources within the partnership (i.e., structural equity).” Panelists felt the original definition was too strongly focused on relationships and that equity should also be about power and resource sharing. One academic panelist noted,

Equity is both relational and structural. We created items that are both relational and structural. Many items are relational and soft items and [there are] fewer structural items. Perhaps, maybe separate into two constructs? Naming them these things makes sure that the structural is embedded. Structural items include percentage of dollars shared.

Another panelist, also an academic expert, responded to the comment by pointing out that the structural aspect was partly reflected in another dimension of partnership success, Reciprocity.

Those comments prompted a community panelist to challenge the possibility of equity in CBPR partnerships at all:

Equitable relationships are not possible between communities and academics—I would start there and say that I don’t think it makes sense to ask about equitable relationships . . . You’ve got to change structure. You can’t ask about [equitable relationships] because they don’t exist. There can’t be equity between me and [a university], between the power and money that [a university] has and how the media sees that, and how external forces see that.

Other panelists questioned whether equity was really about a 50-50 balance, whether equitable relationships were a function of who was involved (individuals vis-à-vis institutions), and whether equity could be achieved by allocating resources to where they were needed, rather than distributing them evenly among partners. This led to the question, “Who has the power to make resources equitable and to change institutional practices and policies?” A community panelist shared an example to demonstrate that changing institutional policies was possible. By acting collectively, her partnership was able to change a university’s institutional review board (IRB) process and alter the way the university’s cancer center examined and assessed disparities in cancer. She stressed, “We did it collectively. We strived for it as you can’t do it by yourself.”

This example led several panelists to comment on the importance of balancing the structural and relational components (or the “analytical” and “people” sides) of equity. A panelist suggested,

Maybe it would help if Section A [of the instrument] became Equitable Partnerships and under that are the process dimensions, the structural (sharing resources), but in a way that conveys the connection to power. As for the relationship issues, it would help if they were a little more and take into account some partners. In our work, the health department, who is an important partner, noted that they would come back after the touchy-feely part is over. (Academic expert)

Another academic panelist followed:

When I think of equitable relationships, I think of my responsibility to be at the table and bring structural elements to the table. It is almost tied to cultural humility. It comes to: are we answering as a whole or as an individual? I think we are in agreement with these items. Am I answering this as [name of panelist] with my partners? We are all in agreement in the definitions but getting to the more concrete ideas to get to later.

Both comments helped the MAPS research team realize that some of the Equity items needed to be clarified and revised. Subsequent discussions also led to questions about the potential use of the instrument—specifically, whether it is a scale for assessing partnership success or a tool that helps CBPR partnerships evaluate their state of development and identify changes that are needed to ensure long-term success.

Lessons Learned

The traditional Delphi method is limited in the extent to which diverse opinions can be expressed and discussed in depth and considered by the group as a collective. Instead, expert opinions are largely individualized, with little or no opportunity for give and take dialogue around complex topics. The MAPS approach demonstrates how the Delphi process can be enriched by incorporating CBPR principles to achieve a more in-depth, nuanced, and collective understanding of phenomena. Below, we highlight the lessons learned and practice implications for a CBPR-enhanced Delphi method.

The MAPS approach used a set of established criteria to guide the Delphi process and analyzed both quantitative and qualitative responses to eliminate redundancies and ensure items adequately captured dimensions of partnership success from the perspective of CBPR experts, thus assessing the face and content validity of the item pool. The integration of qualitative and quantitative data, without prioritizing one form of data over the other (i.e., not relying solely on a quantitative Delphi cutoff), reflected our use of CBPR principles—specifically, co-learning—in practice.

Our efforts to reach consensus through the initial anonymous rounds (Rounds 1 and 2) were designed strategically to lay the groundwork for deeper in-person discussions with the MAPS Expert Panel in Round 3. The face-to-face meeting created opportunities for diverse opinions to be expressed and led to an atmosphere of listening and curiosity that allowed the uncovering of multiple perspectives and deeper levels of understanding. In addition, as all MAPS Expert Panelists were experienced academic and community partners with background and skills using CBPR principles, the process was understood to be inclusive. Our process also benefited by having a heterogeneous group by race, ethnicity, geography, and research interest rather than a more uniform group of experts typically represented in a Delphi approach. The intentional selection of equal numbers of community and academic experts was also essential and methodologically beneficial. This suggests that the selection of panelists and the broader interpretation of “expert” is an essential element when using this approach and that integrating a community-academic panel for Delphi methods facilitates greater understanding of complex problems, improves bidirectional learning in informing research and practice outcomes, and may lead to new research questions and identification of gaps in knowledge.

Another critical aspect of the CBPR-enhanced Delphi process was the attention to facilitating the face-to-face meeting. It was critical to have an individual experienced with Delphi methods and familiar with CBPR principles and group dynamics facilitate the discussions. The facilitator invited differing opinions and handled them in ways that led to respectful and fruitful discussions in keeping with CBPR principles. Moreover, the ability to work with both the quantitative and qualitative results of the Delphi rounds to allow for discussions that went beyond simple agreements and disagreements led to deeper understanding of the constructs and clear wording of items included in the final MAPS instrument.

Conclusion

The Delphi method has been shown to be useful in generating opinions and reactions to information that will inform decision-making in groups. A strength of the Delphi method is the opportunity to view the reflections of experts on a given subject area and rank opinions without the influence of any one expert dominating the process. Adding a face-to-face component to the Delphi process creates an opportunity to reflect at a deeper level and facilitate discussion that allows for debate and work toward a shared understanding or “collective truth.” The use of the Delphi method in groups that have knowledge and skills related to CBPR principles, as demonstrated with the MAPS approach, adds two critical components to the process. First, the CBPR process values qualitative information and the use of qualitative and quantitative information together. When qualitative information is integrated into the Delphi rounds, it allows for deeper understanding of rankings and adds to a more nuanced interpretation of the content. Second, the integration of CBPR principles and processes in face-to-face meetings creates an atmosphere of shared respect for diverse opinions and sets the stage for all voices to get into the mix. This in turn allows for the creation of a “third view” (Coombe et al., 2020, p. 8) or the creation of new insights or revelations that might not have been revealed at earlier stages of the process. Using a CBPR-enhanced Delphi method facilitates a synergistic process that moves above and beyond consensus to enhance the Delphi technique and the development of the items of the MAPS instrument.