The impact of participatory dementia research on researchers: A systematic review

Abstract

Participatory approaches are increasingly required and used in research. In this review, we examined the impact (benefits and disadvantages) of participatory dementia research on researchers as potential key actors to sustainably implement the participatory approach. Our aim was to provide information on how the participatory process could be improved. We conducted a systematic literature review covering the Cochrane Library, PsycNet, PubMed, Scopus, and Google Scholar. The inclusion criteria for publications were as follows: research (1) involving people with dementia as co-researchers, (2) including a description or discussion of the impact of participatory research for researchers, (3) published between 2000 and 2020, and (4) in English. We performed a quality assessment of the included publications. Our final review included nine publications; three categorized as high quality, five as medium quality, and one as low quality. Four of the publications categorized as high or medium quality implemented participation at the level of partnership, and two publications implemented participation at the level of delegated power. The beneficial impact of participatory dementia research on researchers consisted of an increased understanding of people with dementia that widened researchers’ theoretical knowledge and perspectives and increased their competence in working with people with dementia. Disadvantages comprised the required additional effort and time as well as the difficulty of establishing a balanced relationship between researchers and co-researchers. Disadvantages may result from the lack of a definition and structure of participatory research and a lack of training on both sides. More well-designed and transparent methods of evaluating participatory research projects are needed.

Keywords

participatory research patient and public involvement co-research dementia neurodegenerative disorders neurocognitive disorders impact systematic review

Introduction

Participatory approaches involving different stakeholders in research processes are increasingly used internationally. People with lived experience provide insights into their lives and living circumstances, thereby taking on the position of co-researchers as experts of a specific condition or experience (Bergold & Thomas, 2012; Bethell et al., 2018; Crocker et al., 2017; Staley & Minogue, 2006). The definition of different terms, such as participatory research, research engagement, or patient and public involvement (Miah et al., 2019), is not uniform, entailing different roles and participation degrees of the stakeholders involved. Public involvement is defined as “research being carried out ‘with’ or ‘by’ members of the public rather than ‘to’, ‘about’, or ‘for’ them” (INVOLVE, 2021, para. 1). For this review, we used the definition of participatory research as “collaborative co-governance of research, involving researchers and those affected by issues under study or who are in positions to act on the knowledge generated by research (e.g., end-users including participants of an intervention, clinicians, health managers, and policy makers)” (Green et al., 1995 cited by Jagosh et al., 2011, p. 2). In this review, we refer to collaborative research with shared decision-making of parties involved with people with dementia. Furthermore, different terms are used in the literature to refer to people engaging in research (Crocker et al., 2017; Ocloo et al., 2021), such as research partners or non-academic researchers. In this paper, we will use the term co-researchers for people with dementia who have no prior scientific training in the field of research in which they are involved, as distinct from researchers with a professional background. By sharing their experience-based knowledge with researchers, co-researchers contribute to the interaction between practical and theoretical knowledge (Bergold & Thomas, 2012).

For researchers, adopting a participatory approach implies changing usual working processes and practices. The expected benefits of participatory research include a better understanding of those affected by a condition (Staley, 2009, p. 64). Furthermore, researchers gain experience in conducting research projects with co-researchers, which is now often required by funders (Staley, 2009, p. 8). Nevertheless, with regard to the impact for all involved Staley (2015, p. 1) discussed the question whether patient and public involvement is worth doing. The author concluded that evidence on this topic is still lacking but a positive effect is to be expected, especially for researchers (Staley, 2015).

The question of whether it is worth doing is even more relevant in dementia research, where collaboration with those with the condition requires additional effort due to cognitive impairment, communication restrictions, and behavioral changes. Several studies have successfully included people with cognitive decline (e.g., Miah et al., 2019), but symptoms like agitation, apathy or hallucinations (e.g., Cerejeira et al., 2012) as well as disease progression render the participation of those affected more difficult and time-consuming. However, the prevalence of dementia (e.g., Fiest et al., 2016; Livingston et al., 2020) accompanied by social and economic consequences of cognitive decline highlights the importance of participatory dementia research to allow the necessary broadening of scientific perspectives (Staley & Minogue, 2006; Watson et al., 2019).

Further research is needed on the impact of participatory dementia research on research outcomes and parties involved, including people with dementia and researchers (Bethell et al., 2018; Miah et al., 2019). Evidence exists regarding the effects of participatory research on people with dementia, such as their empowerment and an increase in positive feelings (Span et al., 2018; Suijkerbuijk et al., 2019), as well as on the project outcomes in dementia research, such as the better quality of generated data (Clarke et al., 2018; Stevenson & Taylor, 2019). In this review, we wanted to expand the existing knowledge on the impact of participatory research with the experience of researchers. The experience of researchers has been the focus of only a few studies (Staley, 2017), although they represent one important group for impact evaluation (Morgan et al., 2018, p. 3). A lack of knowledge about the positive impact of participatory research is one reason researchers do not include different perspectives (Evans et al., 2013 cited by de Wit et al., 2018). A better understanding of the impact on researchers may be one of several motivating factors to conduct participatory research. Most importantly, a stronger evidence base could aid in the sustainable implementation of participatory approaches in dementia research (Miah et al., 2019). For this reason, in this systematic review, we focused on the beneficial and disadvantageous impact of participatory research as experienced by researchers. Our research questions were:

(1) What are the benefits of participatory research including people with dementia for researchers?

(2) What are the disadvantages of participatory research including people with dementia for researchers?

We have examined these features of participatory research to outline resources needed as well as limitations and to support next steps.

Methods

We conducted a systematic literature review. Details are reported complying with the requirements of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2009). The review protocol has not been preregistered. The search was carried out in April 2020 with the following databases: Cochrane Library, PsycNet, PubMed, and Scopus. These databases cover an interdisciplinary range of literature on the subject of dementia. In addition, we searched articles’ references based on the snowball method, and we searched using Google Scholar to capture gray literature.

Search strategy with inclusion and exclusion criteria

The search strategy was based on four keywords: “dementia”, “research”, “participation” and “impact”. For each keyword, synonyms were identified to find relevant Medical Subject Headings (MeSH) terms and to develop a search string. We focused on the concepts of participation and participatory research. Starting in PubMed, the string syntax was adapted to other databases. The search string in PubMed was as follows:

(((participation [All Fields] OR participatory [All Fields]) AND (“research”[MeSH Terms] OR “research” [All Fields])) AND (“dementia” [MeSH Terms] OR “dementia” [All Fields])) AND (“Impact (Am Coll Physicians)” [Journal] OR “impact” [All Fields]) AND (”2000/01/01” [PDAT]: “2020/12/31” [PDAT])

All search strings used in the different databases are shown in Supplement Material 1. The inclusion and exclusion criteria for the literature search are shown in Table 1. We decided to include non-impacted and non–peer-reviewed material.

Table 1.

Inclusion and exclusion criteria for the systematic review.

Inclusion criterion	Exclusion criterion (EC)
English language	Language other than English (EC1)
	Published before 2000 (EC2)
Published between 2000 and 2020	Does not describe research involving people with dementia or MCI (EC3)
Describes research directly involving people with dementia or mild cognitive impairment (MCI)	Does not report impact of participatory research for researchers (EC4)
Includes a description/discussion of the impact of participatory research for researchers	Review articles (EC5) *
* publications mentioned in reviews and fulfilling our inclusion criteria were included in our review. Abbreviations: MCI: mild cognitive impairment and EC: exclusion criterion

One researcher (AK) searched each database and Google Scholar using the inclusion criteria listed in Table 1 and created a publication list based on the title and abstract of the articles. Duplicates were removed, and the titles relevant to our review were marked. The search in Google Scholar yielded more than 50,000 results, so only the first 20 pages of the results were viewed.

The second researcher (HP) replicated the search on the same dataset with the same strings as well as inclusion criteria and marked the relevant publications independently. Conflicts in selection were resolved by discussion between the two researchers. Subsequently, the relevant articles were stored in EndNote X9 (Version X9.2, Clarivate Analytics, Philadelphia).

Quality assessment

We appraised the methodological quality of the articles using a checklist that we developed based on the Mixed Methods Appraisal Tool (MMAT), which is suitable for all study designs with the exception of non-empirical data (Hong et al., 2018, p. 1). Since studies with a participatory approach are often descriptive, we have created a checklist based on the MMAT due to the lack of critical appraisal tools for these study designs. This checklist is shown in Supplement Material 2.

One researcher (AK) rated the papers individually. The rating results were discussed with a second researcher (HP). We ranked the papers according to the number of questions answered in the affirmative on a scale from 0 to 7. The quality assessment was intended to enable a classification into high quality (6–7), medium quality (3–5), and low quality (0–2).

Data analysis and extraction

We analyzed the data based on Mayring’s (2014, pp. 104–106) qualitative content analysis using a mixed approach. First, main categories were formed inductively based on publications’ content for presentation of study characteristics. The codings of the inductively formed categories were used to produce a descriptive table including the relevant study features. One study feature in our review is the degree of participation, which we operationalized by using Arnstein’s ladder (1969, p. 217), as shown in Table 2. According to Harris et al. (2018), existing models of participation can be applied to decide when participation is achieved in projects. Arnstein’s model (1969) was originally used to describe (political) civic participation. The power structures described there between the political decision-makers involved and the citizens at different levels of participation can be applied to the relationship between researchers and co-researchers in the process of research participation. Arnstein’s model (1969, p. 217) consists of three successive levels of participation: Nonparticipation is the lowest level with the lowest degree of participation. Sublevels manipulation and therapy belong to this level. This is followed by the sublevels of tokenism, namely, informing, consultation, and placation, with an increasing degree of participation. The highest level of participation, according to the model, is citizen control divided into partnership, delegated power, and citizen control.

Table 2.

Degree of participation according to Arnstein (1969) and adapted to participatory dementia research.

Degree of participation	Sublevel	Arnstein’s description (1969, pp. 217–224)	Application to dementia research
Citizen control	Citizen control	Citizens are “demanding that degree of power (or control) which guarantees that participants or residents can govern a program or an institution, be in full charge of policy and managerial aspects, […]” (p. 223)	People with dementia have decision-making power (e.g., DEEP, 2016)
	Delegated power	Citizens may achieve “dominant decision-making authority over a particular plan or program. […] Another model of delegated power is separate and parallel groups of citizens and powerholders, with provision for citizen veto if differences of opinion cannot be resolved through negotiation” (p. 222)	Researchers and people with dementia are equal in decision-making processes, including development of alternatives and identification of preferred solutions
	Partnership	The power is “redistributed through negotiation between citizens and powerholders” (p. 221)	People with dementia are involved in negotiations and decision-making processes as well as engaged in discussion with researchers
Tokenism	Placation	Citizens “begin to have some degree of influence” and to advise powerholders (e.g., in communities); the powerholders keep the power to decide (p. 220)	People with dementia are allowed to give advice, but the right to decide is in the responsibility of the researchers (e.g., involvement in advisory boards of research or care facilities)
	Consultation	Important first step to legitimate participation: Citizens are “heard” but have no power for negotiation; participation might be measured by how many people answered a questionnaire or joined a meeting (p. 219)	Researchers listen to and consult people with dementia. The power to decide remains under the control of the researchers
	Informing	Most important first step to legitimate participation with “a one-way flow of information”: Citizens “hear” but have no power for negotiation (p. 219)	Researchers inform people with dementia (“it is for your benefit”). There is a one-way communication (e.g., providing flyers, and brochures)
Nonparticipation	Therapy	Nonparticipation: The objective is to “enable powerholders to […] ‘cure’ the participants” (p. 217)	Researchers cure people with dementia. The power remains within the researchers
	Manipulation	Nonparticipation: The objective is to “enable powerholders to ‘educate’ […] the participants” (p. 217)	Researchers educate people with dementia. The power remains within the researchers

Second, we deductively formed main categories based on our research questions to identify the benefits and disadvantages of participatory research on researchers. The categories were defined, anchor examples were selected, and coding rules were formulated. The coding rules for the deductively formed categories are presented in Supplement Material 3. We assigned the text passages from the publications to the main categories. We summarized text passages reporting analogous content in the main categories as a subcategory. The text analysis was performed with MAXQDA 2020 software (Version 20.2.2, VERBI GmbH, Berlin).

Results

Study selection

The search generated 859 results, and through the systematic process, we identified nine eligible papers, as shown in the figure (Figure 1). We screened the titles and abstracts of 786 publications. We aimed to include 46 publications in full-text screening. Six publications were not available as full texts and were therefore excluded. Thirty-one articles were excluded because they did not meet our inclusion criteria.

Figure 1.

PRISMA flowchart (Moher et al., 2009, p. 3) of the review search results. Exclusion criterion (EC, Table 1): EC1: language other than English, EC2: published before 2000, EC3: publication does not describe research involving people with dementia or mild cognitive impairment (MCI), EC4: publication does not report impact of participatory research for researchers, and EC5: review articles.

Quality assessment

According to our checklist, three papers were of high quality (Armstrong et al., 2018; Tanner, 2012; Yates et al., 2015). Five papers showed medium study quality because of an unclear research aim (Littlechild et al., 2015), an unclear methodological approach, especially with regard to data collection (Litherland et al., 2018; Littlechild et al., 2015; Morgan et al., 2018), intransparency in the presentation of results (Stevenson & Taylor, 2019), intransparency in the interpretation of findings (Litherland et al., 2018; Parveen et al., 2018), or incoherence between data sources, collection, analysis, and interpretation (Litherland et al., 2018; Littlechild et al., 2015; Morgan et al., 2018; Stevenson & Taylor, 2019; Parveen et al., 2018).

According to our checklist, one study had low quality mainly due to insufficient description of methods, including data collection and analysis (Hendriks et al., 2014). The quality evaluation of the papers is shown in Table 3.

Table 3.

Quality assessment.

Paper	Screening questions		Detailed questions					Score
	S1	S2	D1	D2	D3	D4	D5
Armstrong et al., 2018	Yes	Yes	Yes	Yes	Yes	Yes	Yes	High (7)
Hendriks et al., 2014	Yes	Cannot tell	Cannot tell	Cannot tell	Yes	Cannot tell	No	Low (2)
Litherland et al., 2018	Yes	Cannot tell	Yes	Yes	No	Yes	Cannot tell	Medium (4)
Littlechild et al., 2015	No	Yes	Cannot tell	Yes	Yes	Yes	Cannot tell	Medium (4)
Morgan et al., 2018	Yes	Cannot tell	Yes	Cannot tell	Yes	Yes	Cannot tell	Medium (4)
Parveen et al., 2018	Yes	Yes	Yes	Yes	Yes	Cannot tell	Cannot tell	Medium (5)
Stevenson & Taylor, 2019	Yes	Yes	Yes	Yes	Cannot tell	Yes	Cannot tell	Medium (5)
Tanner, 2012	Yes	Yes	Yes	Yes	Yes	Yes	Cannot tell	High (6)
Yates et al., 2015	Yes	Yes	Yes	Yes	Yes	Yes	Yes	High (7)
S1. Are there clear research questions or a clear aim? S2. Do the data address the research questions or aim? D1. Is the methodological approach appropriate? D2. Are the data collection methods adequate to address the research question/aim? D3. Are the findings adequately derived from the data? D4. Is the interpretation of results sufficiently substantiated by the data? D5. Is there coherence between data sources, collection, analysis, and interpretation? Answer possibilities: Yes (1), no (0), and cannot tell (0). Classification into high quality (6–7), medium quality (3–5), and low quality (0–2)

Descriptive characteristics

Descriptive data of the papers included are provided in Table 4. Seven papers refer to another study or project. Both papers by Littlechild et al. (2015) and Tanner (2012) refer to the care transitions project. Seven of the included papers were from the United Kingdom (UK), one from Belgium and one from the United States United States of America (USA). They were all published between 2012 and 2019. The scientific background and the number of researchers were not reported in the majority of papers. Six papers lacked a definition of the participatory approach used. Three publications used a definition addressing the relationship between researchers and co-researchers and describing collaboration on an equal footing (Littlechild et al., 2015; Parveen et al., 2018; Stevenson & Taylor, 2019), without details about the research phase in which co-researchers were involved or the planning of the collaboration. According to the adapted Arnstein’s ladder of citizen participation for participatory dementia research, four of the papers having high to medium quality described research involvement on the partnership level, and two publications described delegated power in research, as shown in Supplement 4.

Table 4.

Descriptive table of included papers.

Author(s)	Year of publication	Country	Research topic	Design	Reference to a study/project	Researchers involved (n)	Co-researchers involved (n)	Definition of involvement and/or participation	Degree of participation (see Table 2, supplement 4)	Monetary compensation of co-researchers	Training for participatory collaboration
Armstrong et al.	2018	USA	Evaluation of the impact of patient involvement on clinical practice guideline development for positron emission tomography (PET)	Qualitative with parallel group study design	Original study	Involvement of physicians in experimental and control group, number not reported	One person with MCI	Not reported	Placation: Affected person in a meeting group to develop a clinical practice guideline	Not reported	Training for all participants
Hendriks et al.	2014	Belgium	Challenges in conducting participatory research in studies developing technological devices for people with dementia	Descriptive	ATOM, Dementia Lab	Information on the impact of participatory research with people with dementia on professional designers, number not reported	People with dementia involved, number of co-researchers not reported	Not reported	Unclear	Not reported	Not reported
Litherland et al.	2018	UK	Change of life during progression of dementia	Descriptive	IDEAL study	Number and background of researchers not reported	Research involvement group with four people with dementia and five caregivers	Not reported	Partnership: Regular meetings during the study with low-threshold methods for continuous research involvement of group members, focus group asked about attitudes toward PPI.	Yes	Not reported
Littlechild et al.	2015	UK	Evaluation of the impact of co-research with older people	Qualitative design with 30 interviews and five focus groups	Care transitions project	Six researchers with unknown scientific background	16 co-researchers including people with dementia and older people from a black and ethnic minority	“Co-research in this paper refers to research that is ‘done with’ or ‘by’ older people through active involvement in the research process rather than ‘to’, ‘about’ or ‘for’ them as research subjects (Fudge et al., 2007).” (p. 19)	Delegated power: Involvement of co-researchers in all research phases (methods design, identification of research priorities and data analysis, and dissemination of results)	Yes	Training for co-researchers
Morgan et al.	2018	UK	Evaluation of the impact of Alzheimer’s Society Research Network	Mixed methods design with a survey to review monitor roles, qualitative interviews and a survey of advisory board members	Alzheimer’s Society Research Network	23 responses from researchers with unknown scientific background in the survey to review monitor roles, five dementia researchers in qualitative interviews, and unknown number of researchers in the survey of advisory board members	40 responses from members of the research network including people with dementia and persons indirectly affected by dementia in the survey to review monitor roles, two people with dementia were involved in qualitative interviews, and no information about the role of people with dementia in the survey of advisory board members	Not reported	Partnership: People with dementia as research network volunteers can choose to be involved in several ways and give as much time as they wish. For the study, people with dementia were asked about their attitudes toward PPI in surveys and interviews, and data analysis and interpretation was done by researchers	Not reported	Not reported
Parveen et al.	2018	UK	PPI in a dementia care study about cultural obligation and differences	Descriptive	Caregiving HOPE study	Number and background of researchers not reported	One person with dementia in a research advisory team	“The following definition of PPI was adopted: Carers, people living with dementia, members of the public and carer support workers are experts by experience with a wealth of knowledge and experience […]. All those involved with the research were considered members of the study team regardless of personal or professional background.” (p. 993)	Partnership: Involvement of co-researchers in all research phases (prestudy conception, study design and implementation, and dissemination)	Yes	Training for lead researcher
Stevenson and Taylor	2019	UK	Involvement of people with dementia as co-researchers in data analysis	Qualitative design	Risk communication in dementia care study	Number and background of researchers were not reported	Four people with dementia as co-researchers	“PPI in research involves relevant groups and individuals, for example, service users or carers, as partners rather than as participants (HSC Public Health Agency, Research & Development Division, 2014; Taylor, Killick, O’Brien, Begley, & Carter-Anand, 2014; Trivedi & Wykes, 2002)”. (p. 701)	Partnership: Participating as a co-researcher in the research phase of data analysis, data analysis planned and prepared by researchers with decision-making power	Not reported	Not reported
Tanner	2012	UK	Implications of involvement in research as co-researchers	Descriptive	Care transitions project	One researcher who also wrote the publication	Three people with dementia involved as co-researchers	Not reported	Delegated power: Involvement of co-researchers in the research phase of data collection, co-researchers conducted interviews	Not reported	Training for co-researchers
Yates et al.	2015	UK	Involvement of service users in therapy development for dementia	Qualitative design with six focus groups and ten interviews	Original study	Two researchers in the focus groups (one as facilitator and one for documentation) with unknown scientific background	A total of 28 people: 18 people with dementia in focus groups and ten people with dementia in interviews	Not reported	Consultation: Researchers interviewed people with dementia in focus groups and individual interviews	Not reported	Not reported

In three papers, the co-researchers received financial compensation for their research engagement (Litherland et al., 2018; Littlechild et al., 2015; Parveen et al., 2018). In four papers, special training for participatory research was reported: in two publications, a training course for co-researchers was offered (Littlechild et al., 2015; Tanner, 2012); in one paper, training was mentioned for the lead researcher (Parveen et al., 2018); and in one paper, training was offered for all people involved (Armstrong et al., 2018).

Benefits of participatory dementia research for researchers

In the qualitative analysis, 58 codings were assigned to the category of benefits of participatory dementia research for researchers, as shown in Supplement 5. In the majority of publications, researchers evaluated participatory research as overall helpful and beneficial. The main benefit was the widening of the researchers’ perspectives through the involvement of co-researchers (Littlechild et al., 2015; Morgan et al., 2018, pp. 14-19; Stevenson & Taylor, 2019; Tanner, 2012; Yates et al., 2015). Furthermore, interaction with people with dementia increased researchers’ competence (Littlechild et al., 2015; Morgan et al., 2018, pp. 14-17; Stevenson & Taylor, 2019) by improving communication skills for a lay audience (Littlechild et al., 2015; Morgan et al., 2018) and expanding the knowledge of the researchers (Armstrong et al., 2018; Hendriks et al., 2014; Morgan et al., 2018, p. 13; Yates et al., 2015). Participatory dementia research minimized the concerns and insecurity of researchers to behave inappropriately toward the target group (Litherland et al., 2018) because of an in-depth understanding of living with dementia (Morgan et al., 2018, pp. 13-14) and of the perspectives of those affected (Stevenson & Taylor, 2019).

With regard to the benefits for the researchers’ work, participatory research enabled to increase the quality of generated data (Armstrong et al., 2018; Morgan et al., 2018, p. 18; Yates et al., 2015) and facilitated the dissemination of results to stakeholders in dementia care settings (Littlechild et al., 2015; Stevenson & Taylor, 2019). Littlechild et al. (2015) described the relationship between researchers and co-researchers as a powerful combination allowing strengths of both groups to synergize. Participatory research enabled a “sense of familiarity and trust” (Tanner, 2012, p. 298) between researchers and co-researchers. The interaction with people with dementia increased the overall person-centeredness in the research context (Armstrong et al., 2018); researchers felt more motivated and reminded of the original motivation for their work (Morgan et al., 2018, p. 13). Overall, researchers viewed participation as an important strength of their scientific work (Littlechild et al., 2015; Morgan et al., 2018, p. 13; Parveen et al., 2018).

One publication (Littlechild et al., 2015, p. 28) pointed out that researchers did not see data collected by co-researchers as “better” or “worse” compared to the data they would collect independently: Instead, the process was merely “different” (Littlechild et al., 2015, p. 28).

Disadvantages of participatory dementia research for researchers

In the qualitative analysis, 27 codings were assigned to the category of disadvantages of participatory dementia research for researchers, as shown in Supplement 5. Participatory research with people with dementia meant additional effort for researchers (Hendriks et al., 2014; Litherland et al., 2018; Parveen et al., 2018; Tanner, 2012) and additional time (Armstrong et al., 2018; Morgan et al., 2018, p. 17; Tanner, 2012), because research methods had to be adapted to the specific needs and cognitive skills of the co-researchers (Hendriks et al., 2014). Data quality decreased with regard to some interesting aspects because co-researchers did not have the methodological skills required to further explore the aspects during data collection (Littlechild et al., 2015; Tanner, 2012).

Furthermore, researchers reported an emotional burden (Hendriks et al., 2014; Littlechild et al., 2015) deriving from co-researchers’ seeking for advice falling outside of the research projects (Hendriks et al., 2014) as well as uncertainties about their role in participatory research (Littlechild et al., 2015). Some researchers described the need to accommodate such emotional impact with the required neutrality of their role as challenging (Hendriks et al., 2014).

In their work, Hendriks et al. (2014) concluded that participatory research approaches did not create equality between researchers and co-researchers. Although researchers were able to learn much from co-researchers, a hierarchy remained between them (Hendriks et al., 2014). In addition, some researchers stated that they placed too high demands on the co-researchers regarding the workload and tasks in the research process (Hendriks et al., 2014).

Discussion

In this systematic review, we explored the overall benefits (e.g., increased understanding of dementia and the perspectives of people with dementia; new competencies in conducting research; and an increased depth of data) and overall disadvantages (e.g., additional effort and time required; decreased quality in data generation; and added emotional burden) of participatory dementia research on researchers.

In the majority of publications identified in our review, researchers considered participatory research with people with dementia to be beneficial overall. The most frequently stated advantages were the widening of researchers’ perspectives (Littlechild et al., 2015; Morgan et al., 2018, pp. 14-19; Stevenson & Taylor, 2019; Tanner, 2012; Yates et al., 2015), the gain of competencies in working with people with dementia and conducting participatory research (Littlechild et al., 2015; Morgan et al., 2018, pp. 14-17; Stevenson & Taylor, 2019), valuable content gain in research projects (Armstrong et al., 2018; Hendriks et al., 2014; Morgan et al., 2018, pp. 17-18; Yates et al., 2015), and improved relationships between researchers and co-researchers (Litherland et al., 2018; Littlechild et al., 2015; Parveen et al., 2018; Tanner, 2012). These findings correspond with the results of previous studies (Brett et al., 2014; Staley, 2009). In our review, we found that researchers had fewer reservations about participatory research with people with dementia after the implementation of the research approach (Litherland et al., 2018). This finding may be particularly relevant for participatory research with people with dementia because it may be related to the condition’s associated cognitive impairment and stigma. This finding deserves further exploration.

Although researchers thought positively about participatory dementia research in most of the publications identified, disadvantages were also outlined, mainly consisting of additional effort (Hendriks et al., 2014; Litherland et al., 2018; Parveen et al., 2018; Tanner, 2012) and additional time (Armstrong et al., 2018; Morgan et al., 2018, p. 17; Tanner, 2012) required by participatory approaches. These results correspond to previous findings (Bethell et al., 2018; Brett et al., 2014). The additional effort and time for researchers seem particularly constant in participatory research with people with dementia, since working with people with cognitive impairment involves additional ethical and practical issues such as legal requirements concerning informed consent (Gove et al., 2018) compared to working with people who do not have cognitive problems.

Researchers must be aware that successful implementation of participatory research requires not only additional time and resources but also skills that need to be developed through training. Two papers included in our review reported the provision of training for co-researchers (Littlechild et al., 2015; Tanner, 2012). In one paper, training was conducted for the lead researcher (Parveen et al., 2018), and in one publication, all people involved received training (Armstrong et al., 2018). Lack of training in the involvement of laypeople may overwhelm both researchers and co-researchers. These factors need to be considered when planning research studies since they add to workload.

Additional effort for researchers may also arise because of their perceived responsibilities toward co-researchers: In addition to their traditional research tasks, such as planning and conducting research, disseminating results, and networking (Kyvik, 2013), the moral obligation to protect co-researchers with cognitive impairment and to support them beyond project time (Hendriks et al., 2014) leads to a dual responsibility. The legal and ethical responsibility of researchers can influence their relationship with co-researchers (Gove et al., 2018). One paper included in this review stated that a power imbalance between researchers and co-researchers remained, despite the attempt to implement equal cooperation through participation (Hendriks et al., 2014). Co-governance, which underlies the participatory approach (Green et al., 1995 cited by Jagosh et al., 2011, p. 2), was usually not achieved in the publications included in our review. In only two papers (Littlechild et al., 2015; Tanner, 2012), we identified delegated power as participation degree indicating collaboration at eye level between researchers and co-researchers. Four papers described participatory dementia research at the level of partnership (Litherland et al., 2018; Morgan et al., 2018; Parveen et al., 2018; Stevenson & Taylor, 2019).

Considering the effort associated with participatory research, the question arises: A great deal of effort but for what? Do the positive effects on researchers balance the disadvantages? The question of whether and how participatory research should occur remains to be negotiated between the parties involved. Existing frameworks for the implementation of participatory research, such as in the field of community-based participatory research (e.g., Kwon et al., 2018), could provide researchers with useful recommendations for participatory dementia research, accounting for the special requirements of working with people with dementia, which we have discussed in this review.

To date, only a few studies have investigated the impact of participatory dementia research on researchers. However, taking the effects of participatory research into consideration could lead to an increasing willingness of researchers to involve those affected in research (Miah et al., 2019). Reservations of researchers about the research involvement of laypeople may result in tokenism for apparent participation, which only serves the fulfillment of political demands (Thompson et al., 2009).

The design and impact of participatory research are often documented anecdotally in studies (Fudge et al., 2007; Miah et al., 2019). Although the overall quality of the majority of articles was high to medium, most publications included in this review did not define participation and did not provide a transparent representation of the collaboration with the co-researchers. The impact of participatory research was not comprehensively and systematically evaluated in the papers, as this assessment was mostly not the main objective of the publication. In future studies and to ensure an independent evaluation of participatory research, the evaluation should already be taken into account in the planning stage of studies and carried out as an external process evaluation. Evaluating participatory research can be understood as a systematic collection of information about the activities, characteristics, and outcomes of the conducted co-research. This will help gaining insights into its facilitating and hindering factors, on how to improve participatory research, and how to conduct future co-research projects.

Limitations

Our study had several limitations that must be taken into account when interpreting the results. We selected participation and participatory research as search terms. In view of the lack of uniformity in terminology in the literature, similar approaches, such as research involvement and co-design, may have been missed.

By only including English language publications, relevant studies in other languages were excluded. Google Scholar was included in the systematic research to capture gray literature and to reduce publication bias. This search generated a large number of hits. Not all of them were screened so that relevant publications may have been overlooked.

Regarding the assessment of the papers included, we were bound to use the information available to us in the descriptions of the articles. Two publications (Littlechild et al., 2015; Tanner, 2012) referred to the same project. The two papers contained varying levels of detail, which made it difficult to unanimously determine the level of participation. It was important for us to assess the quality of the papers independently of the participation level, as the participatory approach was rarely the main focus of the papers; thus, a comparison of the publications was only possible using other evaluation features. We evaluated the quality of the papers and included all papers in the analysis of the benefits and disadvantages of participatory dementia research. One study focused particularly on the challenges of participatory research (Hendriks et al., 2014). Therefore, Hendriks et al. (2014) described disadvantages of the research approach. These results, however, need to be interpreted carefully since they are taken from a publication in which detailed information on the methods were missing.

The papers that we identified in the systematic literature review did not explore long-term effects. The long-term impact of participatory research on research projects, on co-researchers and on researchers has yet to be determined in future research projects. At the same time, we suggest that researchers should not be considered a homogeneous group. For example, junior researchers and senior researchers differ greatly in their roles and responsibilities in projects (de Wit et al., 2018). Differences also exist based on the scientific background of researchers and the current field of research. Therefore, it is worthwhile to investigate whether participatory research affects researchers differently.

Conclusions

Participatory research involving the views of those affected by the research into the research process itself has both ethical and practical justifications. As shown in this systematic review, participatory approaches are already integrated into dementia research, but there is still room for improvement with regard to the level of participation of people with dementia: a concrete objective should be to achieve progressively higher levels of Arnstein’s model.

Researchers act as a key group for implementing sustainable participatory research. Current evidence on the benefits and disadvantages of participatory research for researchers themselves is very limited. This systematic review sheds light on the little researched impact of participatory dementia research on researchers themselves. Regarding the benefits, researchers have identified that participatory research has helped them gain a better understanding of the daily effects of dementia as well as the perspectives of people with dementia. In addition, they acquired important new competencies and skills in working with co-researchers. Future research should focus on how to maximize this expansion of perspectives and knowledge, especially with regard to the potential of participatory research to decrease stigma against dementia and reduce researchers’ reservations about people with dementia as co-researchers.

This systematic review also revealed the disadvantages of participatory dementia research, particularly the additional effort and time required. These requirements for the implementation of participatory research (e.g., resources for the training of co-researchers and researchers) must be considered in the planning phase of future projects. Currently, many funders make participatory approaches mandatory, and often, without necessary adaptations to their funding scheme. Therefore, researchers may simply not have the means and resources to conduct a participatory project.

Last, future participatory projects should involve some independent process evaluation, which should also include evaluating the impact of the research approach not only for co-researchers but also for researchers. This would help to identify and evaluate additional factors to maximize benefits and minimize disadvantages for all those involved.

Supplemental Material

sj-pdf-1-dem-10.1177_14713012211067020 – Supplemental Material for The impact of participatory dementia research on researchers: A systematic review

Supplemental Material, sj-pdf-1-dem-10.1177_14713012211067020 for The impact of participatory dementia research on researchers: A systematic review by Antonia Kowe, Hentry Panjaitan, Olga A Klein, Marina Boccardi, Martina Roes, Sonja Teupen, Stefan Teipel in Dementia.

Footnotes

Declaration of conflicting interests

Stefan Teipel has done the listed works below (all in Germany). MSD Sharp & Dohme GmbH, Lindenplatz 1, 85540 Haar: 11/09/2018 - Quality circle for physicians in Kühlungsborn, Talk: ‘Dementia and Diabetes - current report’. 14/11/2018 - MSD Expert-forum: NAB Alzheimer in Munich, participator as consultant. 13/08/2019 - Event ‘Diabetes and Dementia’ in Rostock, Talk: ‘Dementia and Diabetes - current report’. ROCHE Pharma AG, Emil-Barell-Str. 1, 79639 Grenzach-Wyhlen: 12/09/2019 - 3. Nationales Advisory-Board in Frankfurt (Main), participator as consultant. 27/09/2019 - ROCHE Symposium at the DGN Congress in Stuttgart, Talk: ‘Amyloid as target for diagnosis and treatment in Alzheimer’s disease’. Biogen GmbH, Riedenburger Straβe 7, 81677 München: 23/04/2020 ‐ Biogen Advisory Board Session. 28/04/2020 - Biogen Advisory Board Session. 18/09/2020 - Biogen Advisory Board Session. Dr Willmar Schwabe GmbH & Co. KG, Ottostr. 24, 76227 Karlsruhe: 22/01/2021 - Virtual Advisory Board. The other authors declare that there is no conflict of interest. A language editing service was used.

Funding

The present study was funded in part by a grant from the German Research Foundation (TE 621/3-1) and the European Social Fund (DigiCare, ESF/14-BM-A55-0021/19).

ORCID iDs

Antonia Kowe

Martina Roes

Sonja Teupen

Supplemental material

Supplemental material for this article is available online.

Antonia Kowe is scientific coordinator of the Department “Ageing of Individuals and Society” (AGIS) at the Interdisciplinary Faculty of the University of Rostock. She is Master of Science of Public Health and has a focus on participatory research in dementia, involving patients, caregivers as well as medical experts, care providers, and other stakeholders into the research process.

Hentry Panjaitan is a research fellow at the German Center for Neurodegenerative Diseases (DZNE) Site Rostock/Greifswald and has a Bachelor of Science in Cognitive Science. She works at the project Digital Training of Nursing and Health Management (DigiCare) which explores strategies for case-based learning to improve teaching in care and health management courses. She has experience in involving patients and caregivers in the research process.

Olga A. Klein is a research fellow at the German Center for Neurodegenerative Diseases (DZNE) Site Rostock/Greifswald. Her background is in Neuro- and Health Psychology with a focus on the management and treatment of neurodegenerative diseases and neuropsychological rehabilitation. At the DZNE, she is part of the clinical dementia research group and her work focuses on the effectiveness of digitally supported healthcare services for family caregivers of people with dementia combining randomized controlled trials and participatory research approaches.

Marina Boccardi, Experimental Psychologist, PhD in Neuroscience, leads the Late-Translational Dementia Research working group to identify hurdles in the translation of scientific knowledge to the care of patients with neurocognitive disorders. Her field of research entails examining the methodological issues preventing the definition of clinical or policy evidence-based guidelines based on the available literature, and devise procedures to prevent or overcome such hurdles and facilitate implementation. This approach requires a variety of methods, ranging from formal evidence-to-decision procedures (e.g., Cochrane, GRADE) to consensus procedures (e.g., Delphi methods) to the use of participatory approaches with different stakeholders (e.g., clinicians, patients, and caregivers).

Martina Roes is the site speaker of the German Center for Neurodegenerative Diseases (DZNE) Site Witten and working group leader of the research groups Implementation and Dissemination Research and Dementia Care Interventions. Recently Prof. Roes leads various projects: PELI-D - Preferences for everyday living inventory of the elderly; BENi - Implementing structural and environmental preventive interventions to promote physical activity for residents in nursing homes; DEMacute - DEMfriendlyHospital -developing a framework for dementia friendly hospitals in Germany. Furthermore, Prof. Roes leads a team which is developing, applying and evaluating research designs and methods, instruments, and processes of public-patient-involvement (PPI) and participatory research in the field of dementia care research.

Sonja Teupen is interim group leader of the research group “Methods in Health Care Research” at the German Center for Neurodegenerative Diseases (DZNE) Site Witten, Germany. With a background in Social Sciences and a doctorate in Sociology, her current research interests are in subjectivity in dementia, qualitative research methods and methodology and participatory research approaches. She has lectured at various universities on the subjective experience of dementia, self and identity, qualitative research, and social differentiation in organizations.

Stefan Teipel is full professor for Clinical Dementia Research at the Rostock University Medical Center and senior group leader for clinical dementia research at the DZNE Site Rostock/Greifswald, Germany. His research interests include the development and evaluation of novel diagnostic markers and interventions for neurodegenerative diseases in clinical research and routine care. His group also develops technical assistive systems for people with dementia. The participation of patients and caregivers and other non-academic stakeholders has become an important focus over the last 3 years. The topic of the current article is an important contribution to this research focus for the group.

References

Armstrong

M. J.

Mullins

C. D.

Gronseth

G. S.

Gagliardi

A. R.

(2018). Impact of patient involvement on clinical practice guideline development: A parallel group study. Implementation Science, 13(1), 55–13. https://doi.org/10.1186/s13012-018-0745-6

Arnstein

S. R.

(1969). A ladder of citizen participation. Journal of the American Institute of Planners, 35(4), 216–224. https://doi.org/10.1080/01944366908977225

Bergold

Thomas

(2012). Participatory research methods. A Methodological Approach in Motion. Historical Social Research/Historische Sozialforschung, 37(4), 191–222. https://doi.org/10.12759/hsr.37.2012.4.191-222

Bethell

Commisso

Rostad

H. M.

Puts

Babineau

Grinbergs-Saull

Wighton

Hammel

Doyle

Nadeau

McGilton

(2018). Patient engagement in research related to dementia: A scoping review. Dementia, 17(8), 944–975. https://doi.org/10.1177/1471301218789292

Brett

Staniszewska

Mockford

Herron-Marx

Hughes

Tysall

Suleman

(2014). A systematic review of the impact of patient and public involvement on service users, researchers and communities. The Patient-Patient-Centered Outcomes Research, 7(4), 387–395. https://doi.org/10.1007/s40271-014-0065-0

Cerejeira

Lagarto

Mukaetova-Ladinska

E. B.

(2012). Behavioral and psychological symptoms of dementia. Frontiers in Neurology, 3(73), 73–21. https://doi.org/10.3389/fneur.2012.00073.

Clarke

C. L.

Wilkinson

Watson

Wilcockson

Kinnaird

Williamson

(2018). A seat around the table: Participatory data analysis with people living with dementia. Qualitative Health Research, 28(9), 1421–1433. https://doi.org/10.1177/1049732318774768.

Crocker

J. C.

Boylan

A. M.

Bostock

Locock

(2017). Is it worth it? Patient and public views on the impact of their involvement in health research and its assessment: A UK-based qualitative interview study. Health Expectations, 20(3), 519–528. https://doi.org/10.1111/hex.12479

de Wit

Beurskens

Piškur

Stoffers

Moser

(2018). Preparing researchers for patient and public involvement in scientific research: Development of a hands‐on learning approach through action research. Health Expectations, 21(4), 752–763. https://doi.org/10.1111/hex.12671

10.

DEEP (2016). Setting up an influencing group. https://www.dementiavoices.org.uk/wp-content/uploads/2016/01/DEEPGuide-Setting-up-an-influencing-group.pdf

11.

Fiest

K. M.

Jetté

Roberts

J. I.

Maxwell

Smith

Black

Blaikie

Cohen

Day

Holroyd-Leduc

Kirk

Pearson

Pringsheim

Venegas-Torres

Hogan

(2016). The prevalence and incidence of dementia: a systematic review and meta-analysis. The Canadian Journal of Neurological Sciences, 43(Suppl 1), S3–S50. https://doi.org/10.1017/cjn.2016.18

12.

Fudge

Wolfe

C. D.

McKevitt

(2007). Involving older people in health research. Age and Ageing, 36(5), 492–500. https://doi.org/10.1093/ageing/afm029

13.

Gove

Diaz-Ponce

Georges

Moniz-Cook

Mountain

Chattat

Øksnebjerg

(2018). European working group of people with dementiaalzheimer Europe’s position on involving people with dementia in research through PPI (patient and public involvement). Aging & Mental Health, 22(6), 723–729. https://doi.org/10.1080/13607863.2017.1317334

14.

Harris

Cook

Gibbs

Oetzel

Salsberg

Shinn

Springett

Wallerstein

Wright

(2018). Searching for the impact of participation in health and health research: challenges and methods. Biomed Research International, 2018(1), 1–12. https://doi.org/10.1155/2018/9427452

15.

Hong

Q. N.

Pluye

Fàbregues

Bartlett

Boardman

Cargo

Dagenais

Gagnon

M. P.

Griffiths

Nicolau

O'Cathain

Rousseau

M. C.

Vedel

(2018). Mixed methods appraisal tool (MMAT) version 2018: User guide. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf

16.

INVOLVE (2021). What is public involvement in research? https://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/

17.

Jagosh

Pluye

Macaulay

A. C.

Salsberg

Henderson

Sirett

Bush

P. L.

Seller

Wong

Greenhalgh

Cargo

Herbert

C. P.

Seifer

S. D.

Green

L. W.

(2011). Assessing the outcomes of participatory research: protocol for identifying, selecting, appraising and synthesizing the literature for realist review. Implementation Science, 6(24), 1–8. https://doi.org/10.1186/1748-5908-6-24

18.

Kwon

S. C.

Tandon

S. D.

Islam

Riley

Trinh-Shevrin

(2018). Applying a community-based participatory research framework to patient and family engagement in the development of patient-centered outcomes research and practice. Translational Behavioral Medicine, 8(5), 683–691. https://doi.org/10.1093/tbm/ibx026

19.

Kyvik

(2013). The academic researcher role: Enhancing expectations and improved performance. Higher Education, 65(4), 525–538. https://doi.org/10.1007/s10734-012-9561-0

20.

Litherland

Burton

Cheeseman

Campbell

Hawkins

Oliver

Scott

Ward

Nelis

S. M.

Quinn

Victor

Clare

(2018). Reflections on PPI from the ‘action on living well: asking you’ advisory network of people with dementia and carers as part of the IDEAL study. Dementia, 17(8), 1035–1044. https://doi.org/10.1177/1471301218789309

21.

Littlechild

Tanner

Hall

(2015). Co-research with older people: Perspectives on impact. Qualitative Social Work, 14(1), 18–35. https://doi.org/10.1177/1473325014556791.

22.

Livingston

Huntley

Sommerlad

Ames

Ballard

Banerjee

Brayne

Burns

Cohen-Mansfield

Cooper

Costafreda

S. G.

Dias

Fox

Gitlin

L. N.

Howard

Kales

H. C.

Kivimäki

Larson

E. B.

Ogunniyi

Mukadam

(2020). Dementia prevention, intervention, and care: 2020 report of the lancet commission. Lancet, 396(10248), 413–446. https://doi.org/10.1016/S0140-6736(20)30367-6

23.

Mayring

(2014). Qualitative content analysis: Theoretical foundation, basic procedures and software solution. https://www.ssoar.info/ssoar/bitstream/handle/document/39517/ssoar-2014-mayring-Qualitative_content_analysis_theoretical_foundation.pdf

24.

Miah

Dawes

Edwards

Leroi

Starling

Parsons

(2019). Patient and public involvement in dementia research in the European union: a scoping review. BMC Geriatrics, 19(1), 1–20. https://doi.org/10.1186/s12877-019-1217-9

25.

Moher

Liberati

Tetzlaff

Altman

D. G.

& the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine, 151(4), 264–269. https://doi.org/10.1371/journal.pmed.1000097

26.

Morgan

Grinbergs-Saull

Murray

(2018). We can make our research meaningful: The impact of the alzheimer’s society research network. Alzheimer’s Society. https://www.alzheimers.org.uk/sites/default/files/2018-04/Research%20Network%20Report%20low-res.pdf?_ga=2.87732184.1369384992.1531215596-224747478.1531215596

27.

Ocloo

Garfield

Franklin

B. D.

Dawson

(2021). Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Research Policy and Systems, 19(8), 1–21. https://doi.org/10.1186/s12961-020-00644-3

28.

Parveen

Barker

Kaur

Kerry

Mitchell

Happs

Fry

Morrison

Fortinsky

Oyebode

J. R.

(2018). Involving minority ethnic communities and diverse experts by experience in dementia research: The caregiving HOPE study. Dementia, 17(8), 990–1000. https://doi.org/10.1177/1471301218789558

29.

Span

Hettinga

Groen-van de Ven

Jukema

Janssen

Vernooij-Dassen

Eefsting

Smits

(2018). Involving people with dementia in developing an interactive web tool for shared decision-making: Experiences with a participatory design approach. Disability and Rehabilitation, 40(12), 1410–1420. https://doi.org/10.1080/09638288.2017.1298162

30.

Staley

(2009). Exploring impact: Public involvement in NHS, public health and social care research. https://www.invo.org.uk/wp-content/uploads/2011/11/Involve_Exploring_Impactfinal28.10.09.pdf

31.

Staley

(2015). ‘Is it worth doing?’Measuring the impact of patient and public involvement in research. Research Involvement and Engagement, 1(6), 1–10. https://doi.org/10.1186/s40900-015-0008-5

32.

Staley

(2017). Changing what researchers' think and do’: Is this how involvement impacts on research? Research for All, 1(1), 158–167. https://doi.org/10.18546/RFA.01.1.13

33.

Staley

Minogue

(2006). User involvement leads to more ethically sound research. Clinical Ethics, 1(2), 95–100. https://doi.org/10.1258/147775006777254489

34.

Stevenson

Taylor

B. J.

(2019). Involving individuals with dementia as co-researchers in analysis of findings from a qualitative study. Dementia, 18(2), 701–712. https://doi.org/10.1177/1471301217690904

35.

Suijkerbuijk

Nap

H. H.

Cornelisse

IJsselsteijn

W. A.

de Kort

Minkman

(2019). Active involvement of people with dementia: a systematic review of studies developing supportive technologies. Journal of Alzheimer's Disease, 69(4), 1041–1065. https://doi.org/10.3233/JAD-190050

36.

Tanner

(2012). Co-research with older people with dementia: Experience and reflections. Journal of Mental Health, 21(3), 296–306. https://doi.org/10.3109/09638237.2011.651658

37.

Thompson

Barber

Ward

P. R.

Boote

J. D.

Cooper

C. L.

Armitage

C. J.

Jones

(2009). Health researchers’ attitudes towards public involvement in health research. Health Expectations, 12(2), 209–220. https://doi.org/10.1111/j.1369-7625.2009.00532.x

38.

Watson

Saunders

Muniz Terrera

Ritchie

Evans

Luz

Clarke

(2019). What matters to people with memory problems, healthy volunteers and health and social care professionals in the context of developing treatment to prevent alzheimer’s dementia? A qualitative study. Health Expectations, 22(3), 504–517. https://doi.org/10.1111/hex.12876

39.

Yates

L. A.

Orrell

Spector

Orgeta

(2015). Service users’ involvement in the development of individual cognitive stimulation therapy (iCST) for dementia: A qualitative study. BMC Geriatrics, 15(4), 1–10. https://doi.org/10.1186/s12877-015-0004-5

40.

Hendriks

Huybrechts

Wilkinson

Slegers

(2014, October 6th-10th). Challenges in doing participatory design with people with dementia. In Proceedings of the 13th Participatory Design Conference: Short Papers, Industry Cases, Workshop Descriptions, Doctoral Consortium papers, and Keynote abstracts. Volume 2 (pp. 33–36), Windhoek, Namibia.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.44 MB