Bridging the Gap in Implementation Science: Evaluating a Capacity-Building Program in Data Management,Analysis,Utilization,and Dissemination in Low- and Middle-Income Countries

Abstract

Building capacity in implementation science within health programs is dependent on training in theory and practice of epidemiology, statistics, and research in addition to high self-efficacy toward application of training. This article describes a training program providing technical assistance to more than 300 health facilities in Kenya and Tanzania, its evaluation results, and its ability to improve participants' knowledge, competencies, and self-efficacy on data management, analysis, and dissemination among health care professionals. Two months prior to the training, participants (n = 98) were emailed a pre-course survey including 19 questions using a Likert-type response for planning the content of the workshop. Six to 12 weeks after the training, a post-course survey was emailed to all participants. Five different trainings were conducted indicating 5 participant cohorts. The questions posed involved course satisfaction, course impact on knowledge and skills, and self-efficacy in data analysis and utilization. Post-course survey results revealed that the participants had confidence in data analysis, which was significantly different from the pre-test results (0.05 α). Qualitative commentary complemented the findings of the impact of the workshop. Four manuscripts and 13 abstracts have been submitted post training. Results suggest that a short-term training program can achieve immediate gains in data and research self-efficacy among health care professionals. Although increasing self-efficacy is a necessary first step in developing skills, educators should engage in continuing education for sustainable dissemination practices. There is an urgent need to determine the current infrastructure to promote scientific dissemination. This will assist countries to produce better evidence to support their programs, policies, and overall health programs.

Introduction

Although African researchers doubled their production of publications between the years 2003 and 2012, the continent produces a mere 1% of total world research.¹ The underrepresentation of the countries in Africa among other low- and middle-income countries (LMICs) has been attributed to 5 major areas: Poor research production, poor preparation of manuscripts, poor access to scientific literature, poor participation in publication-related decision-making processes, and bias of journals.² Not only are LMICs underrepresented, but their research capacity is also subpar to high-income countries because of “few qualified researchers, less funding, poor infrastructure, and lack of expertise in preparing manuscripts for publication.”³ Capacity-building, or “the development and strengthening of human and institutional resources” is a critical component in solidifying LMIC participation and representation in the research and publication arena.^4,5

HIV remains a long-term development challenge in sub-Saharan Africa, which accounts for 70% of all people living with HIV.⁶ Therefore, capacity-building research for HIV – particularly to increase evidence-based practices to improve treatment and regulations regarding HIV – is of critical importance. Great progress has been made in the broadening of HIV/AIDS services available in LMICs, but it is still not enough.

LMICs bear the highest burden of disease and would benefit immensely if research was conducted to combat its high prevalence. Health care professionals are overburdened with the overwhelming task of the medical needs of the patients under their charge coupled with the need for medical advancement in equipment, care, and facilities. Additionally, programs are in place to enhance research capacity, but they fall short in delivery because many are aimed at undergraduate or graduate training opportunities for students and are considered inadequate to meet the needs of LMICs.^7,8 The 10/90 gap in research – only 10% of worldwide resources devoted to health care research is conducted in those countries where 90% of preventable deaths occur – applies to most, if not all, LMICs.^9,10 Because of the failure of established research capacity-building programs, an effective new model is desperately needed to promote and sustain research capacity-building.¹¹ New models and funding were provided once the need was realized and has resulted in many improvements in this field. These emergent models have attended to issues such as: career trajectories; institutional infrastructure and mentorship support for research; tailoring research capacity efforts to a wide range of disciplines in diverse fields of science and scholarship; and the inclusion of clinical, community, and policy partners in the process.^12
–14 Health service delivery in these settings is still overwhelmed with long-term HIV treatment, care, and support needs that have increased the human resource crisis.¹⁵ One strategy to address this gap is to develop research capacity among locally-based HIV staff and clinical providers in order to “task shift” research activities from more academic clinical researchers to those with less research experience. This model has not always been considered effective but can be utilized in the research field to increase production of evidence-based findings that can be translated into practice.¹⁶

This study's capacity-building approach was directly relevant to those in HIV programs; however, the principles are applicable to a much wider audience. The training methodology was focused on empowering health care professionals to understand what entails a research study and how to write abstracts and manuscripts based on the analysis of the data they had been collecting. Through a detailed selection process, frontline health care teams who deliver services to more than 300 health care facilities in Kenya and Tanzania were identified. Over a period of 12 days, participants attended 3 staggered workshops on concept note development, interpreting and understanding data, and manuscript (and abstract) writing. Before the commencement of the training, each participant was emailed a pre-course survey to form a baseline from which to draw results. After the training concluded, a post-course survey identical to the pre-course survey was given to participants, and the pre- and post-course results were analyzed for statistical significance. The training approach was based on principles of adult learning with a focus on peer review during all the steps of planning, organizing, and conducting a training/learning event. The course modeled a variety of effective training methodologies that included experiential and participatory learning. These methodologies were demonstration, practice, discussion, brainstorming, case studies, role play, games, and presentation. The process of delivery of training included plenary sessions, group work exercises, group discussions, and review sessions. This article describes the training program and its evaluation results and the program's ability to improve participants' skills and competencies in data management, analysis, interpretation, and dissemination.

Methods

Study design

These trainings were carried out between March 2015 and February 2017. Pre- and post-course comparison cross-sectional surveys were employed to evaluate health care providers' knowledge, competency, and skills related to data management, analysis, utilization, and dissemination before and after the training program. Faculty from The University of West Florida (USA), The University of British Columbia (Canada), and a researcher from Kenya Medical Research Institute (KEMRI) designed the pre- and post-course surveys that were piloted among selected health care providers before being emailed to the course participants for completion.

Study participants

The study participants (n = 98) consisted of staff working for 4 nongovernmental organizations (NGOs) in Kenya and Tanzania. The NGOs received funds from various development partners to improve access to and availability of comprehensive HIV services for more than 300 supported health facilities providing HIV comprehensive services distributed throughout the countries. Study participants were all trained clinicians (medical doctors, nurses, pharmacists, laboratory specialists, community specialists, and data management specialists) who are all members of interdisciplinary technical assistance teams that proactively and strategically respond to the gaps in knowledge, skills, and competency of HIV care and prevention in staff working in facilities at the district, regional, and national level. Five trainings were conducted separately indicating altogether 5 different cohorts of between 12 to 20 attendees. Follow-up was done using a structured mentorship approach wherein participants were assigned to faculty and updates on the milestones were provided on a monthly basis. Three NGOs were based in Kenya and 1 was based in Tanzania (an affiliate of an American University registered locally as an NGO in Tanzania).

Ethical considerations

This paper describes a training program that delivers a structured curriculum aimed at producing novice researchers skilled in data management, analysis/interpretation, and utilization. As part of the evaluation plan for the program, a survey was developed as a tool to assess both process and outcomes of the program. The survey was administered online on a voluntary basis to participants, all of whom were provided with full information on the evaluation, including the purpose and nature of the evaluation. Prior to obtaining consent, the participants were assured that their personal information would be kept confidential and that their identity would not be revealed. Participants were further assured that there would be no negative consequences for not participating. Questionnaires were designed to take not more than 15 min to complete.

The pre- and post-course survey instrument

The pre- and post-course surveys were created and administered online through Google Survey. The pre-course survey was sent to course participants prior to attending the course to examine their personal aims and objectives for attending the training and their baseline level of knowledge and confidence in the areas covered by the training. In addition, the pre- and post-course surveys focused on major themes regarding the course: knowledge, skills, practical applicability, and further capacity-building needs. The survey comprised 19 questions; each question in the survey tool asked participants to rank their expectations or experiences. Questions involved 3 components: (1) knowledge – using true or false questions, (2) level of skill (and competency) – based on a 5-point Likert scale ranging from Never heard of it to Confident, and (3) understanding of key concepts and tools using a 5-point Likert scale ranging from Never heard of it to I am an expert on the concept. In addition, demographic information was collected regarding age, sex, discipline, and number of years at current job.

Participants were then asked to complete a post-course survey (that was similar to the pre-course survey) 6 to 12 weeks after they had completed their training. These pre- and post-course surveys aimed to examine participants' opinions of the training (with the intention of improving it for future learners) and to investigate the impact of the training on their level of knowledge and confidence in the subject as well as the impact of the training on their practice and attitudes toward data management, analysis, utilization, and dissemination.

Training approach

The training curriculum was developed by faculty from The University of West Florida and The University of British Columbia in addition to a biostatistician and instructor from KEMRI, all of whom are well published and have extensive experience in HIV service delivery in LMICs and represent diverse areas of expertise in public health research, quantitative and qualitative research, ethics, quality management, and education.

The initial training was organized in February 2015 as a pilot course conducted at an NGO supporting HIV care and treatment facilities in Nairobi, Kenya. It was implemented as a proof of concept to: (1) test course material and the selected teaching approach, (2) revise the teaching material based on the participants' feedback and facilitators' experience, and (3) verify a potential demand in data management, analysis, utilization, and dissemination. The course was attended by the HIV technical assistance team comprising 17 health care providers (medical doctors, nurses, laboratory technologists, pharmacists, community outreach specialists, and data management personnel). The course was conducted by 3 facilitators who had participated in the development of the course materials and methods. Data collected from the course feedback were reduced to 3 major categories: (1) course content and materials, (2) teaching methods and learning experiences, and (3) course organization and duration. For each category, the data were further broken into the following subcategories: (1) what participants liked about the course, (2) what participants disliked or thought should be improved, and (3) participants' practical suggestions and considerations. Recommendations from this pilot course suggested that the selection of participants should be based on participants who have shown interest in abstract writing, implementing quality improvement, or implementing a research project to ensure that the course is more effective and the teaching material more relevant. Hence, participants of the 4 NGOs presented in this paper were selected on this basis; selected teams worked on their proposed studies throughout the course (Fig. 1).

FIG. 1.

The training approach.

The training aimed to demystify scientific writing, analysis, interpretation, and the publication process with a focus on teaching the fundamentals of effective dissemination. Instruction was focused primarily on the process of writing and publishing scientific manuscripts; however, research methodology and grant writing were also addressed. The training was organized into 3 staggered, sequential workshops delivered in approximately 12 days. Workshop 1, Concept Note Development, lasted 2 days; participants were taught how to effectively organize thoughts and ideas into concept. Participants had a concept sheet, which is a brief summary of their larger project (eg, an intervention, thesis, research report, performance, service project), that concisely described the content and scope of the project and identified the project's objective, its proposed methodology, ethical considerations, and intended results. Participants presented their draft concept idea and received peer feedback. Workshop 2, Interpreting and Understanding Your Data, lasted 5 days; participants were taught to understand statistical and monitoring & evaluation concepts in data analysis, build skills in basic data analysis, enhance data interpretation, and understand the peer review process. Participants were tasked with drafting complete methods and results sections of their manuscripts by end of the workshop. Workshop 3, Manuscript and Abstract Writing, lasted 5 days; participants were taught how to communicate new information with the intention of improving health care. It was stressed that this information must be organized and presented clearly and logically. Tips on how to organize an evidence-based manuscript were provided. Participants were taught how to construct each of the elements learned so as to logically communicate the author's message; participants were also instructed on strategies in conducting direct referencing. Participants developed draft abstracts and manuscripts by the end of the training. Throughout the process, participants continuously received a one-on-one critique of their abstract and/or manuscript.

During the course, participants applied concepts and principles to their abstract or manuscript, allowing them to learn by “doing” and “reflecting.” Short, theoretical sessions were followed by extensive, practical sessions during which participants worked in small groups on their own research projects. In subsequent plenary sessions, each individual shared his or her work and feedback for the benefit of all participants. Participants were assigned an experienced research mentor to assist and guide them in completing their projects.

Data analysis

Responses were uploaded into the Google Survey Excel template and into a STATA version 12 (StataCorp LP, College Station, TX) database and analyzed using descriptive statistics. Differences in trainees' mean levels of knowledge of each of the subject areas before and after the training were analyzed using paired samples t tests. Chi-square tests (McNemar paired-sample tests) were used to assess the statistical significance between the pre- and post-course survey answers. A P value of <0.05 was considered significant.

Results

Sample characteristics

As shown in Table 1, a total of 106 trainees attended all the sessions of the training, 98 (92%) of whom completed both the pre- and post-course surveys. The majority of participants were female and most were between 30–39 years of age. More than 70% of participants had been working with their respective NGO for more than 5 years. The largest proportion of the participants (65%, n = 66) were clinicians (ie, medical doctors, nurses).

Table 1.

Characteristics of the Participants

Sociodemographics (n = 98)	N (%)
Sex
Female	81 (83%)
Age group
20–29 years	12 (12%)
30–39 years	75 (77%)
40–49 years	10 (10%)
≥50 years	1 (1%)
Years of service
<1 year	4 (4%)
1–2 years	11 (11%)
2–5 years	12 (12%)
≥5 years	71 (73%)
Profession
Medical Doctor	29 (30%)
Nurse	34 (35%)
Pharmacist	2 (2%)
Laboratory Technician	2 (2%)
Community-Based Personnel	13 (13%)
Data Management Personnel	18 (18%)

Participant knowledge in the subject areas

Table 2 indicates a significant difference (P < 0.05) between the pre- and post-scores for the subject areas that assessed knowledge. Different statements requiring recognition and comprehension were asked in the pre- and post-course surveys, including prevalence, strength of association (odds ratio), study designs, descriptive versus inferential statistics, and validity versus reliability.

Table 2.

Knowledge in Different Subject Areas

Subject Areas	Pre-survey	Post-survey	X ²	Significance (P value)
Prevalence has to do with existing cases of a disease but incidence has to do with new cases developed over time.	54	84	19.68	<0.001
The odds ratio is very similar to the risk ratio, particularly if a disease is rare. However, the odds ratio is usually higher than relative risk.	76	81	3.28	0.07
Cross-sectional studies make comparisons at a single point in time whereas longitudinal studies make comparisons over time.	38	83	44.8	<0.001
Randomized clinical trials are usually stronger than case-control studies to establish causality.	44	91	53.2	<0.001
lf your significance level is (P value) 0.05, the corresponding confidence level is 95%.	30	98	57.8	<0.001
Main difference between inferential statistics and descriptive statistics is: Descriptive statistics are used to describe a sample, and inferential statistics determine if those statistics are likely to describe the population of interest.	49	96	55.9	<0.001
The validity defines the extent to which a tool measures what it is supposed to measure, while reliability defines the degree to which a tool produces consistent or similar results on the same phenomenon at different times or when used by different people.	61	97	24.7	<0.001

Table 3 summarizes pre- and post-course skills competency capacity on the 19 training and research skills taught in the course. Pre-training skills and competencies scores ranged from 1.52 to 2.02 on the scale of 0 to 5, with 5 being extremely confident. Participants reported lower confidence in collecting and presenting data in their field of study, reading and critiquing journals, quantitative and qualitative research, and preparing abstracts to submit to a conference before the course began. The difference in improvement between the pre and post surveys was similar across the different domains. Explaining the difference between quantitative and qualitative research demonstrated the largest improvement (mean difference = 2.9) as a result of the course.

Table 3.

Skills and Competencies

Skills and Competencies	Mean pre-survey score (SD)	Mean post-survey score (SD)	t test statistic	Significance (P value)
If I was tasked with collecting and presenting data to an audience on HIV prevalence within my community, I feel that I have the knowledge to successfully and accurately complete the project.	1.97 (1.0)	4.71 (0.7)	−22.6	<0.001
I understand measures of central tendency: mean, median, mode, interquartile range, standard deviation	2.02 (0.9)	4.92 (0.8)	−23.1	<0.001
When collecting variables for a data set I know the best sources from which to obtain this information.	1.52 (0.7)	4.25 (0.8)	−23.8	<0.001
I have confidence in reading and critiquing journals in my field.	1.87 (0.9)	4.63 (0.9)	−20.7	<0.001
I have confidence in identifying appropriate data analysis methods for routine program data.	1.80 (0.8)	4.58 (1.2)	−18.5	<0.001
I am able to describe at least 3 different research study designs.	1.67 (0.8)	4.52 (1.3)	−17.9	<0.001
I can explain the difference between quantitative and qualitative designs.	1.64 (0.9)	4.50 (1.3)	−17.1	<0.001
I understand the process toward preparing an abstract to submit to a conference.	1.56 (0.8)	4.15 (1.4)	−15.6	<0.001

SD, standard deviation.

Key comments from the post-survey

Most participants found the training material very useful, specifically with regard to understanding the scientific writing process. Teaching was valued as the well-structured and step-by-step presentation of material with clear objectives for each session and practical exercises that ensured the proper absorption of material. One participant stated,

“The practical approach of the course was very useful, the materials were good, and the examples were thought-provoking. I learned how to use and maybe teach abstract writing to other staff in new ways, and how to work more effectively. Wonderful program!”

Working on their own data throughout the course, including allocations of time for necessary revisions and improvements, was considered an extremely practical exercise. One participant noted:

“The whole process I found really interesting and engaging, I have never had any data utilization training previously and the amount of help and confidence this has given me can only help me going forward in my career…From the experience I can only say how helpful and informative I found the experience. My only advice to anybody doing it in the future would be to go in with an open mind, be prepared to listen and you will get… [sucked] in!!!!”

The post-training evaluation demonstrated that, in addition to general improved performance of routine responsibilities, participants applied the knowledge and skills obtained in the course for a wide range of purposes. Another participant noted,

“I have an MPH yet the concepts were framed in a way I could understand. I was impressed with the simplicity of language in understanding data analysis. You covered a lot in a short time. Will definitely be putting into practice all I've learnt [sic] into my proposal.”

Participants appreciated the interactive, engaging, and friendly learning environment as well as the facilitators' capabilities and encouraging support during the course. The opportunity to learn from a group of international “experts with real-life experience” was seen as a unique opportunity. One participant explained,

“Really well paced with a good balance of class and exercises. The models and concepts were all explained and demonstrated clearly. The team was always clear and concise and never patronizing either! The activities were really well tied into the class work and really helped us to write up our work.”

Another participant noted that,

“Excellent theories, models and training…the best part was how tailored it was to our particular situation. [Instructor name] is a living embodiment of how to do it. He pushed us all to exactly the right levels and got the results we needed. Can't wait to see my name as a lead author in Pubmed.”

Discussion

These findings suggest that an intensive, short-term training program can achieve large, immediate gains in data and research self-efficacy among NGO staff. In addition, the results demonstrate initial evidence of behavior change related to use of data and research skills to support their community work – namely 4 manuscripts and 13 abstracts have been published by health care workers who attended the training. Providing data and research capacity-building opportunities and resources to NGOs and health care teams working in service delivery is both an essential piece of building infrastructure for these organizations' growing data needs and a facilitator of more equitable and effective partnerships with researchers.^15,17 Although the purpose of this 12-day course was not to create researchers, studies have shown organizations that are better “consumers” of health data and empowered participants in research processes are more effective in assessing their community's health needs, advocating and planning for needed programs and services, and employing evidence-based practices.¹⁸

It is necessary to build capacity of staff in NGOs, research institutes, and health policy organizations so higher quality operational research can be conducted.¹⁹ The quality and effectiveness of research training needs improvement. To be useful, training must have a practical focus whereby people receive hands-on experience (not just theory) in doing practical elements of research that can be used immediately in their work.²⁰ There is also a need for more skills in concept development, study designs, abstract, and manuscript writing. Organized sessions during which health care providers are prepared to present their work and findings to potential users of research, such as health sector policy makers and strategic planners, is a mechanism to build health care providers' skills to translate evidence into action and thereby improve implementation.

There is much discussion surrounding the importance of implementation science to support evidence-based practices.²¹ However, this thinking is often in the abstract. To illustrate why evidence-based or practice-based research on health issues is vital for programs, one should have concrete ideas of what needs to be done, what can be done, and how practice-based research can support these ideas.^22,23 For example: What practical steps does a program need to take to build a system for health research that meets its national needs – in terms of such areas as building skills, creation of areas of expertise, and improvement or creation of specific institutions? Another example is: On what priority health areas and problems in society does your country or region need to focus research to improve health? And most importantly, what can you do now – with the resources and skills available today – and where should you aim to be in several years as your dissemination of evidence-based practices progresses?

HIV programs that have a strong implementation science component are badly needed throughout Africa.^24,25 Perhaps using strategies discussed in this manuscript would increase research in this field of study to better improve knowledge of HIV and promote the development of such programs. The University of West Florida (UWF), in collaboration with other partners, has developed a practical-based training known as Reaching Engaging Advancing Research (REAR), aimed at improving abstract and manuscript writing among health care teams. The training helps health care leaders process how to address problems and needs that programs face. It can provide tools and advice based on its past experience. This will inform the authors' policy and planning work and help countries decide how they can organize themselves to produce better evidence to support their policies and national health programs. It is expected that this information and these experiences will give a broader indication of what is needed in the East African region. A proposal to conduct an assessment among Ministry of Health management-level staff was identified as an opportunity to expand human resource capacity through a UWF-led REAR project, which is an initial formative study using key informant interviews, focus groups, direct observations, and short surveys to determine feasibility and perceptions of manuscript and abstract development among Ministry of Health Management Teams in Kenya, Tanzania, and Rwanda.

Footnotes

Acknowledgments

The authors of this paper alone are responsible for the views expressed in this publication, which do not necessarily represent the decisions or policies of the University of West Florida, The University of British Columbia, and the Kenya Medical Research Institute. We want to thank Kennedy Muthoka, Dr. Abubakar Maghimbi, Dr. Linah Digolo, Dr. Lyndon Marani, Susan Kimathi, Robert Biswalo, Dr. Tulli Tuhuma, and Dr. Gabriel Mahasi for their input toward the success of this training initiative. The training workshops were made possible through support from Christian Health Association of Kenya, LVCT Health, Maryland Global Initiatives in Tanzania, Palladium Group, Centre for Health Solutions, and Health Strat Kenya.

Author Disclosure Statement

The authors declare that there are no conflicts of interest. The authors received no financial support for the research, authorship, and/or publication of this article.

References

The World Bank. Improving the quality and quantity of scientific research in Africa. 2014. www.worldbank.org/en/region/afr/publication/improving-the-quality-and-quantity-of-scientific-research-in-africa Accessed January 15, 2017 .

Langer

, Díaz-Olavarrieta

, Berdichevsky

, Villar

. Why is research from developing countries underrepresented in international health literature, and what can be done about it?. Bull World Health Organ, 2004; 82:802–803.

Chu

, Jayaraman

, Kyamanywa

, Ntakiyiruta

, Umubyeyi

. Building research capacity in Africa: equity and global health collaborations. PLoS Med, 2014; 11:e1001612.

World Health Organization. Tobacco Free Initiative (TFI). Capacity building and initiatives. 2013. www.who.int/tobacco/control/capacity_building/background/en Accessed January 15, 2017 .

Enoch

Health research capacity strengthening: a UKCDS mapping. www.ukcds.org.uk/sites/default/files/content/resources/UKCDS_Health_Research_Capacity_Strengthening_Mapping.pdf Accessed February 24, 2017 .

UNAIDS. THE GAP REPORT. 2014. www.unaids.org/sites/default/files/media_asset/UNAIDS_Gap_report_en.pdf Accessed February 24, 2017 .

Frenk

, Chen

. Overcoming gaps to advance global health equity: a symposium on new directions for research. Health Res Policy Syst, 2011; 9:11.

Gadsby

EW.

Research capacity strengthening: donor approaches to improving and assessing its impact in low- and middle-income countries. Int J Health Plann Manage, 2011; 26:89–106.

Global Forum for Health Research. The 10/90 report on health research: 2003–2004. 2004. http://announcementsfiles.cohred.org/gfhr_pub/assoc/s14789e/s14789e.pdf Accessed February 24, 2017 .

10.

Ghaffar

, Ijsselmuiden

, Zicker

Changing mindsets: research capacity strengthening in low- and middle-income countries. www.who.int/tdr/publications/documents/changing_mindsets.pdf Accessed February 24, 2017 .

11.

Whitworth

JAG

, Kokwaro

, Kinyanjui

, et al. Strengthening capacity for health research in Africa. Lancet, 2008; 372:1590–1593.

12.

Bates

, Taegtmeyer

, Squire

, et al. Indicators of sustainable capacity building for health research: analysis of four African case studies. Health Res Policy Syst, 2011; 9:14.

13.

Hoffman

, Røttingen

J-A

, Bennett

, Lavis

, Edge

, Frenk

. Background paper on conceptual issues related to health systems research to inform a WHO global strategy on health systems research. 2012. www.who.int/alliance-hpsr/alliancehpsr_backgroundpaperhsrstrat1.pdf Accessed February 24, 2017 .

14.

ESSENCE on Health Research. Seven principles for strengthening research capacity in low- and middle-income countries: simple ideas in a complex world. 2015. www.who.int/tdr/publications/seven-principles/en Accessed February 24, 2017 .

15.

Stein

, Lewin

, Fairall

, et al. Building capacity for antiretroviral delivery in South Africa: a qualitative evaluation of the PALSA PLUS nurse training programme. BMC Health Serv Res, 2008; 8:240.

16.

Cooke

A framework to evaluate research capacity building in health care. BMC Fam Pract, 2005; 6:44.

17.

Lansang

, Dennis

. Building capacity in health research in the developing world. Bull World Health Organ, 2004; 82:764–770.

18.

Memiah

, Muhula

, Oruko

, et al. Adaptive scenarios in quality improvement (QI): a case study of an international health development and research organization embracing change. Int J Heal Sci Res, 2015; 5:285–291.

19.

Durlak

, DuPre

. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol, 2008; 41:327–350.

20.

Marjanovic

, Hanlin

, Diepeveen

, Chataway

. Research capacity-building in Africa: networks, institutions, and local ownership. J Int Dev, 2013; 25:936–946.

21.

UNAIDS. Global report: UNAIDS report on the Global AIDS epidemic 2010. 2010. www.unaids.org/globalreport/documents/20101123_GlobalReport_full_en.pdf Accessed February 24, 2017 .

22.

Cole

, Boyd

, Aslanyan

, Bates

. Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis. Health Res Policy Syst, 2014; 12:17.

23.

Vogel

Research capacity strengthening: learning from experience. www.ukcds.org.uk/sites/default/files/content/resources/UKCDS_Capacity_Building_Report_July_2012.pdf Accessed February 24, 2017 .

24.

Padian

, Holmes

, McCoy

, Lyerla

, Bouey

, Goosby

. Implementation science for the US president's emergency plan for AIDS relief (PEPFAR). J Acquir Immune Defic Syndr, 2011; 56:199–203.

25.

Johnson

, Kennedy

, Harris

, Lincoln

, Neace

, Collins

. Strengthening the HIV/AIDS service delivery system in Liberia: an international research capacity-building strategy. J Eval Clin Pract, 2005; 11:257–273.