User experiences of structured stakeholder engagement to consider transferability: The TRANSFER approach

Barriers and facilitators to using systematic review findings in decision making

Methods that support the use of findings from systematic reviews in decision making have evolved significantly over the past decades. Databases where reviews are published have become more physically accessible (e.g., through sources like the Cochrane Library; www.cochranelibrary.com), publishers have developed more user-friendly formats to present findings (e.g., Supporting Policy-relevant Reviews and Trial (SUPPORT) summaries; Rosenbaum et al., 2011) and researchers have developed robust frameworks that help decision making groups, such as guideline panels, consider the most important decision criteria and related research evidence in their decision making processes (Evidence-to-Decision Framework (EtD) (Lavis et al., 2009; Moberg et al., 2018; Rosenbaum et al., 2018). However, a number of real and perceived barriers to the use of evidence in decision making remain, including people's lack of skills to find and interpret evidence, available time, and knowledge among decision makers regarding the function of systematic reviews (Innvaer et al., 2002; Oliver et al., 2014; Tricco et al., 2016). Studies have also identified activities that may facilitate the use of evidence, including improved consideration of context, and improved collaboration between stakeholders and researchers (Oliver et al., 2014; Rosenbaum et al., 2011; Tricco et al., 2016). For the purpose of this study, the term stakeholders refers to anyone with an interest in the findings from a systematic review, including patients and service users, patients and the public, health care practitioners, policymakers and other decision makers.

Considering context and the transferability of review findings

An intervention's implementation context may influence how it works (Shepperd et al., 2009). Context is therefore an important consideration when evaluating, or exploring experiences and perceptions, of an intervention in a systematic review. For the purpose of this study, context refers to ‘the multi-level environment (not just the physical setting) in which an intervention is developed, implemented and assessed: the circumstances that interact, influence and even modify the implementation of an intervention and its effects’ (Munthe-Kaas et al., 2020). Most systematic reviews include studies from different contexts that may vary according to geography, politics and/or culture (Lewin et al., 2017). Synthesized findings, based on data from studies that may have been conducted in a variety of contexts, may pose problems when people begin to consider the transferability (sometimes referred to as applicability, directness, relevance, etc.) of review findings to a local decision-making context. Decisions makers may question whether findings from a systematic review are relevant for their context. There is general consensus in the literature that ‘the real or perceived failure of reviews to address relevant questions, contextualize the findings, or make actionable policy recommendations’ can inhibit the use of systematic reviews in evidence-informed decision making (Chambers et al., 2011; Innvaer et al., 2002; Lavis et al., 2005; Petticrew, 2004; Sheldon, 2005). This is perhaps particularly relevant for interventions that are more complex and which are comprised of ‘a number of separate elements which seem essential to the proper functioning of the interventions although the “active ingredient” of the intervention that is effective is difficult to specify’ (BUK Medical Research Council). Improved collaboration with stakeholders, including decision makers, may be one way by which to address questions about context and transferability of review findings.

Collaboration between stakeholders and review authors in conducting reviews

Stakeholder engagement in systematic reviews has long been advocated by researchers and stakeholders alike (Cottrell et al., 2014; Lavis et al., 2005; Sakala et al., 2001; Wale et al., 2010). There is a perceived ‘gulf between decision makers and researchers, which prevent[s] the production of research from feeding into decision making processes’ (Orton et al., 2011). Contact and collaboration between researchers and decision makers has been touted as both a barrier when absent, and a facilitator when present, to the use of review findings in decision-making. Numerous studies have extolled the potential benefits of stakeholder engagement, including production of more useful and relevant systematic reviews (Cottrell et al., 2014; Sakala et al., 2001; Wale et al., 2010) (Aloe et al., 2020; Pollock et al., 2018). Engaging with stakeholders is considered critical for defining and ensuring a common understanding of the review question and what it seeks to answer, understanding the review contexts and improving stakeholder buy-in, as they may sit with a great deal of expertise which may otherwise be unavailable or difficult to obtain (Cottrell et al., 2014; Esmail et al., 2015). Despite this enthusiasm, there is little published literature evaluating stakeholder engagement in research activities generally (Esmail et al., 2015). There are however, numerous studies calling for training materials, and examples of how to support stakeholder engagement in systematic reviews (Pollock et al., 2018).

Specifically, there is currently a lack of practical guidance on how researchers can effectively collaborate with stakeholders to systematically and transparently consider context and the transferability of review findings (Burchett et al., 2018; Burford et al., 2013; Gruen et al., 2005).

The TRANSFER approach

TRANSFER approach aims to ‘support … collaboration between review authors and stakeholders from the beginning of the review process to systematically and transparently consider factors that may influence the transferability of systematic review findings’ (Munthe-Kaas et al., 2020). The main tenets of the TRANSFER framework are: (a) close collaboration between review authors and stakeholders, starting with the development of the review question, and (b) recognition that various stakeholders' expertise and contributions in the development of the review question are necessary for improving our consideration of context in systematic reviews. The TRANSFER approach consists of guidance for review authors, including guidance on: -

how to organize a meeting with stakeholders to define the population, intervention, comparison and outcomes to be examined in the review (using the PICO template)

The TRANSFER approach is intended to help review authors to identify factors that may influence the transferability of a review finding to a local decision making context. Identified transferability factors should be considered to be hypotheses, and can include factors related to geopolitical climate or social or cultural values. Review authors can use the identified transferability factors to inform subgroup analyses later in the review to assess whether a review finding is likely to be transferable to the review context. See Munthe-Kaas et al. (2019, 2020) for more details on the TRANSFER approach and how it was developed (Munthe-Kaas et al., 2019, 2020). See Table 1 for an example illustrating how the TRANSFER approach could support review authors to systematically consider how contextual factors could influence the transferability of review findings to the context of interest.

User testing to explore user experience of the TRANSFER approach

User testing (also referred to as usability testing) refers to a research method where the goal is to expose areas of improvement in the design of a service, system or product to improve the products next iteration (Rosenbaum et al., 2008). Part of this method includes exploring users' experience of using a product, in this case the TRANSFER approach. The term ‘user experience’ refers to ‘all the aspects of how people use an interactive product’ (Alben, 1996). It is ‘a consequence of a user's internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g., complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g. organizational/social setting, meaningfulness of the activity, voluntariness of use, etc.)’ (Hassenzahl & Tractinsky, 2006). By exploring user experience of the TRANSFER approach, we hope to better understand user needs and contexts, make appropriate revisions, and attempt to optimize the methods described in the approach.

User-centred development

We employed a user-centred design approach in this study. User-centred design, sometimes called human-centred design, is an approach to developing a product, service or system that puts the users' and other stakeholders' needs at the centre of the development (International Organization for Standardization, 2010). Two key characteristics of this approach are: (1) development that is based on an explicit understanding of users' needs and (2) an iterative process that entails creating or improving a prototype, exploring peoples' experiences of that prototype while carrying out tasks, identifying areas for improvements, and repeating the cycle.

User testing method

To understand users' needs related to TRANSFER, we explored their user experiences of the tool by combining several methods. At the core we drew on a method called ‘user testing’—a study design commonly applied in design development (Chisnell & Rubin, 2008) to understand people's ability to carry out desired tasks with a product or service.

We employed a ‘constructive interaction with think-aloud’ method (Als et al., 2005; Kahler et al., 2000; Van den Haak et al., 2004). This involved participants working together using the TRANSFER approach while we observed and captured their concurrent think-aloud reflections of what they were thinking and feeling. From now on, we will refer to these sessions as ‘group user tests’. We followed up these group user tests immediately with individual semi-structured interviews employing retrospective think-aloud and written participant feedback.

Participant selection

We used a purposive sampling method to identify participants. For each group user test we aimed to include at least: (a) two participants who had experience conducting systematic reviews (effectiveness reviews in three user tests, and qualitative evidence synthesis in one user test), and (b) two participants who were considered to be stakeholders in a review process (including policy makers, guideline panel members, other decision makers) and who were familiar with using the findings from or commissioning systematic reviews. We sent email invitations to systematic review authors from research institutes in Norway and policy makers from the Norwegian directorates of Health, Integration and Diversity, and Children, Youth and Family Affairs. Test participants were not rewarded financially but did receive reimbursement for travel costs when requested.

Data collection

Each group user test was made up of a meeting scenario, feedback forms and semi-structured interviews with review authors.

Analysis

We transcribed the audio-recordings of interviews. A third party transcribed the audio-recordings of two of the meetings and one researcher checked them for accuracy. We analysed data using the framework analysis method (Ritchie & Lewis, 2003). We used an a priori template of codes based on an adapted version of Morville's honeycomb model of user experience (minus the facets accessibility and value, which we deemed to be premature at this stage in the development of TRANSFER; see Table 3) as the basis for analyzing the user experience of the conversation guide (Boyatzis, 1998; Crabtree & Miller, 1999) (Rosenbaum, 2010; Morville, 2004; Fereday & Muir-Cochrane, 2006). Through our analysis we identified many findings related to stakeholder engagement and other aspects of the meeting scenario that did not fit into the honeycomb model. We decided to use thematic analysis to organize these findings into themes (Boyatzis, 1998).

Two researchers coded each of the documents. We discussed all of the codes and themes until consensus was reached. We did not need to bring in a third researcher to settle any disagreements. We actively looked for examples of contrasting and discrepant participant opinions/experiences. All of the data were coded and analysed in their original language. We translated quotations for use in this article.

Data privacy and ethical consideration

According to the Norwegian Centre for Research Data (nsd.no) online portal for determining registration of studies, this study did not warrant application for review by the data privacy committee https://nsd.no/personvernombud/en/notify/index.html (i.e. participants could not be identified in the data, and we did not save, analyse or use and sensitive or personal information). In Norway, this type of study falls outside of the remit of the Regional Committee for Medical and Health Research Ethics. In this case, it remains up to each institution to follow best ethical research practices. We followed ethical guidelines as outlined by The Norwegian National Committees for Research Ethics (Guidelines drawn up by The Norwegian National Research Ethics Committee for medical and health research).

Participant profiles

We included 17 participants in four group user tests conducted in Norway in 2018 and 2020. See Table 3 for an overview of relevant participant characteristics. For the remainder of this paper, participants refers to either stakeholders or review authors. In three of the group user tests the stakeholders were decision makers from Norwegian governmental departments that often commission systematic reviews and/or are involved in guideline processes health and social services. In one group user test, we were unable to engage real decision makers, so three review authors played the role of stakeholders. We will still refer to these participants as stakeholders throughout the findings section.

One of the review authors was familiar with the concept of the TRANSFER approach, and two review authors had been involved in the development of the approach, but did not have experience using the conversation guide. Three of the eight stakeholders had taken part in an earlier iteration of the TRANSFER approach but indicated to the research team that their experience with or knowledge of TRANSFER felt insignificant.

Findings related to users' experience of the TRANSFER conversation guide

In this section, we describe the users' experience of the TRANSFER conversation guide. We begin by describing general impressions and summarizing participants' feedbacks with a selection of quotes to illustrate their experiences. We then present a table that organizes participants' feedback into themes using a modified version of Morville's honecomb model (see Table 4) (Rosenbaum, 2010).

The participants appeared to have a generally positive response to the TRANSFER conversation guide:

[The TRANSFER conversation guide] would have been helpful in previous reviews (review author, user test 1 interview).

[The conversation guide] brought forth many relevant discussions and covered what was necessary (and maybe even more than that) (stakeholder, user test 3 feedback form).

Generally, we observed great variation in how easily participants appeared to be able to use the conversation guide document. One participant speculated that it would be easier to use with practice:

To a certain extent, one feels unprepared for this. It is good to be tuned in. It was good. It becomes a training exercise. If you have done this a few times with a specific problem, it will become easier (stakeholder, user test 2 meeting scenario).

However, participants expressed practical concerns regarding the usability of the conversation guide such as how much time it may take to use it, potential difficulty of assessing identified transferability factors, and whether all factors are equally relevant for each review.

Additionally, some participants found the format challenging. For example, several participants expressed concerns that the list style of the conversation guide could lead to overly detailed discussions. Participants suggested that the conversation guide should be simplified, that there should be more space for notes, and that it should be clearer that the list is meant as ‘prompts’ and not a checklist.

Several participants struggled with the terminology, or research jargon, used in the conversation guide. One participant commented that it would be helpful to have a translation of the conversation guide available in their own language.

Participants' suggestions for improving the guide included offering more examples and explanations for each potential factor, giving more detailed instructions, making the guide more flexible, adding a glossary of terms and including a more in-depth discussion of the review question before embarking on the conversation guide.

In Table 4 we present an overview of the findings related to users' experience of the TRANSFER conversation guide organized in accordance with a modified version of Morville's honeycomb model (Rosenbaum, 2010).

How participants experienced stakeholder collaboration

In this section, we describe our observations of, and participants' own expressed experiences with, stakeholder collaboration during the simulated meeting. The findings described in this section are based on both the authors' observations of the user test, participants' voiced thoughts (using the think-aloud technique) and feedback gathered from the semi-structured interviews with review authors following the simulated meeting. There is therefore an over-representation of feedback from review authors. We begin by presenting findings related to participants' experience of structured stakeholder collaboration and then present findings related to how the participants understood and perceived the goal of the TRANSFER approach in particular.

Participants' experiences unpacking context and transferability

One of the aims of the TRANSFER approach is to provide guidance to review authors on how to systematically and transparently consider contextual factors that may influence the transferability of review findings. Our data showed how TRANSFER could support this process, while uncovering some of the areas where people struggled.

Summarizing the meeting scenarios

In this section we describe two issues related to the results of the meeting scenarios. First, we discuss challenges we observed related to the hypothetical nature of the simulated meetings. Next we present and compare the results of the individual meeting scenarios from each user test.

Observation of application of TRANSFER in a systematic review process

As previously described, we also applied TRANSFER to a real-life setting. When we compare the findings from the user tests with the real-life example, we notice a number of key differences: -

The lead author recruited participants who had personal and/or professional experience with the phenomena of interest (the intervention) as opposed to participants who were unfamiliar with the topic

The above questions were based on specific transferability factors but were worded in a way as to encourage discussion that could perhaps include topics related to other transferability factors as well. The review authors only participated in the discussion to ask clarifying questions or move the discussion forward. They did not participate in the actual discussion of transferability factors. Furthermore, all participants actively engaged in the discussion and we even observed interactions between participants that were not facilitated by the review author (e.g., one participant asking another participant how the issue was in their context).

Considering context and transferability

When provided with a list of transferability factors to consider, review authors and stakeholders are able to discuss context in a more thoughtful and nuanced manner, moving beyond country level to actually defining what it is that may make studies from a certain context less transferable to another context. Such a considered approach to context would hopefully facilitate the use of review findings by decision makers by producing reviews that are more useful and relevant for a decision makers' context (Innvaer et al., 2002; Oliver et al., 2014; Tricco et al., 2016). Additionally, such a systematic and transparent consideration of context may be able to inform assessments of the GRADE/-CERQual domains of indirectness and relevance. Some papers have suggested using a Delphi approach as a means to collaborate with stakeholders and collect information on transferability from stakeholders (Nasser et al., 2012; Wang et al., 2006).

The user tests also revealed a fundamental issue related to the TRANSFER approach related to transferability factors. Are transferability factors merely a subset of effect modifiers? In this case, should we consider expanding the remit of TRANSFER from only considering transferability factors, to identifying all potentially important effect modifiers related to the review question? This concept needs further investigation and thought.

Stakeholder collaboration

Stakeholder collaboration in itself is widely regarded as an important way to ensure the use of systematic review findings in informed decision making (Innvaer et al., 2002; Oliver et al., 2014; Tricco et al., 2016). Findings from this study indicate that collaboration between review authors and stakeholders could lead to a more clearly defined review question and to a more structured discussion of potential transferability factors. Furthermore, a structured approach to stakeholder collaboration may encourage more active stakeholder participation by creating a space where it becomes evident to the stakeholder that their expertise is valuable, appreciated and has the potential to improve the relevance of the systematic review findings to their decision-making context. Such stakeholder participation may not only affect the actual relevance of the review findings, but may also potentially influence the stakeholder's perception of the relevance of the review findings.

Improving user experience of the TRANSFER conversation guide

Although participants were generally positive towards the use of the TRANSFER approach, the group user tests uncovered several format and content issues that need to be addressed to improve the TRANSFER conversation guide. In particular, we need to find ways to convey the usefulness of the TRANSFER approach for people who are unfamiliar with it. Although people indicated that use would become easier with experience, it is important to resolve challenges facing first time users, such their experience of length and density of the conversation guide, need for more examples/explanations, and facilitate easy meeting preparation. Furthermore, given the complex nature of discussions on context and transferability, translations of the conversation guide (and guidance for review authors) to languages other than English would likely be helpful. Finally, since only one review author read the guidance for review authors that we sent before the user test, we must explore whether the length or density of the guidance for review authors is a hindrance to its use, or whether it was difficult for participants to find time to prepare for a user test.

TRANSFER and qualitative evidence syntheses

The findings from this study indicate that review authors were positive toward the use of TRANSFER in qualitative evidence syntheses. We acknowledge that there is a wider debate on the appropriateness of assessing transferability (generalizability) of qualitative research (Finfgeld-Connett, 2010). However, given the underlying beliefs of the authors (and participants in this study) that qualitative research should be synthesized to inform decision making, the debate regarding generalizability is immaterial. Within the ‘subtle realist’ position (social reality is not completely constructed), synthesis can provide a greater understanding of a phenomenon than a single primary study (Hammersley, 1992). However, as with reviews of effectiveness, it is important to consider and assess factors that could influence the transferability of review findings to a decision-making context.

Moreover, the participants proposed that identified transferability factors could also be used to inform inclusion criteria and the sampling strategy in a synthesis. There appear, however, to be three important issues to explore before incorporating TRANSFER into reviews of qualitative evidence: First, how does the TRANSFER approach fits within the GRADE-CERQual Approach (Lewin et al., 2015). Second, how could TRANSFER fit with qualitative syntheses that use more interpretive synthesis methods. Third, how can the language in the conversation guide be adapted for use with qualitative research.

Challenges related to group user testing

The advantage of involving multiple participants in each group user test was that it more closely resembled a meeting between review authors and stakeholders. The ‘think aloud’ protocol was also considered particularly useful as it allowed us more insight into the individual thoughts of each participant within the group, and not just the overall interaction between participants.

A drawback of involving multiple participants in a group user test, however, is that it was difficult and time consuming to transcribe the data, and analyse the findings. Furthermore, it was very difficult to compare and contrast between group user tests with different participants because the discussions were very connected to the dynamics of the specific group of participants.

We also encountered challenges related to the nature of group user tests more generally. It was difficult to recruit participants; therefore in one session review authors took on the role of stakeholders, rendering that data less representational and therefore weaker. During the group user tests, participants were often frustrated or confused due to the unfamiliarity of the review question being discussed, and participants did not prepare in the way that (we hope) they would for a real meeting with stakeholders.

Observing application of TRANSFER in a systematic review process

Few of the challenges related to the TRANSFER conversation guide that we uncovered during the group user tests were present when we observed the application of TRANSFER In a systematic review process. This indicates that many of the issues that arose during group user tests may be addressed by better preparation of the review authors adapting the conversation guide to the topic and participants, and discussing a topic that is relevant for the participants.

Future research

The findings from these user tests resulted in substantial changes to the format and language used in the TRANSFER Conversation guide (see above). We also realized the importance of including a discussion of the review question at the beginning of the meeting. We would like to test the refined version of the conversation guide, well as the PICO template with a wider range of review authors and stakeholders (e.g., different languages, settings, topical background, and experience levels). We would also like to continue observing the use of the TRANSFER approach in systematic review processes.

Some of the findings from this study can be used as evaluation measures in future studies to evaluate the experience and/or effect of stakeholder participation. For example, future studies could explore:

The perceived benefits and challenges of a structured approach to stakeholder collaboration

Researcher's role

The data for the study was collected and analysed by two researchers (the two first authors) who developed the TRANSFER approach. The researchers are experienced systematic review authors, familiar with conducting both reviews of effectiveness and qualitative research. At the time of the user tests, the researchers were colleagues of, or had previously engaged with most of the participants (review authors and stakeholders) at some point before the user test.

Our involvement in developing TRANSFER and familiarity with some of the participants may have influenced both the feedback given by participants, and how we analysed the data. On the other hand, our intimate knowledge of the TRANSFER approach put us in a unique position to understand: (a) where the conversation guide fell short of its intended aims, and (b) the nature of the participants' misunderstandings or problems. Likewise, participants' familiarity with us may have reduced the stress they felt while using a tool for the first time under observation.

Strengths and limitations

The strengths of this study are the use of multiple methods of data collection to triangulate data, the meeting scenario was organized to resemble a real meeting between review authors and stakeholders, and the use of the honeycomb model of user experience to interpret the findings. Furthermore, by including findings from our observation of a real-life application of the TRANSFER approach in this study, we are able to explore which of the challenges identified in our analysis may be related to user testing rather than the TRANSFER approach. However, this study has the following limitations: -

A small number of user tests.

Implications for practice

The findings from this study indicate that review authors and stakeholders may find a structured approach to collaboration beneficial when considering context and transferability of review findings. The results of such structured discussions with stakeholders may be included in a systematic review to:

provide rationale for inclusion criteria and subgroup analyses;

The goal of the TRANSFER approach, as well as the concepts context and transferability need to be clarified for review authors and stakeholders intending to use the TRANSFER approach. The TRANSFER conversation guide needs to be modified with respect to format and language to improve user experience. Furthermore, meetings with stakeholders are likely to be more productive when review authors are familiar with the conversation guide, and all participants are adequately prepared for the meeting.

Using a conversation guide when collaborating with stakeholders may lead better collaboration between review authors and clients, better review questions, clearer inclusion criteria and improved consideration of context. The TRANSFER approach conversation guide appears to be equally relevant for review teams undertaking qualitative evidence synthesis and reviews of effectiveness.