Use of community engagement interventions to improve child immunisation in low- and middle-income countries: A systematic review and meta-analysis

Characteristics of included studies

The search returned over 46,000 papers, out of which 61 impact evaluation met the criteria for being included in the review, along with an additional five on-going studies. For the included IE, authors undertook a targeted search and identified 47 qualitative papers and 69 project reports that were used to strengthen understanding of the design, context and implementation of the programmes.

Though mostly concentrated in South Asia (32) and Sub-Saharan Africa (25), the studies included in this review are spread across 19 LMICs. Within these two regions, the majority of the studies were from India (21) and Nigeria (8). The majority of the studies largely fell into two engagement classifications, engagement as the intervention (engagement is embedded) (27) and engagement in design (16). There were 15 studies that fell into more than one engagement classification and were classified as having multiple engagement types. The remaining five studies were classified as engagement in implementation autonomy of interventions. Full immunisation coverage or FIC was the most commonly reported outcome (33) followed by DPT3 (27), measles (21) and Bacille Calmette–Guerin (BCG) (16). About 52 per cent of the studies included in the review were randomised controlled trials (RCTs). Of the remaining studies, the majority either employed a difference-in-difference estimation strategy (33%) or controlled-before after study design (8%).

Regarding the characteristics of cost evidence, 22 of the 61 evaluations selected for inclusion in the systematic review reported some type of cost analysis. Of these, 18 used an experimental design and the remaining four used quasi-experimental methods to identify the impact of treatment. Of the 22 studies: studies reported a total cost of the intervention; 12 included a cost-effectiveness analysis; three included cost-efficiency analysis; two studies included a cost–benefit analysis and two analysed cost per quality-adjusted life year (QALY) or cost per disability-adjusted life year (DALY).

Quality of evidence base

The authors appraised the quality of both quantitative and qualitative as well as cost evidence base. The quantitative evidence was mostly low quality, though the randomised studies were generally of higher quality than quasi-experimental studies. The randomised studies for the most part followed recommended allocation sequence methods ensuring comparability of intervention and control groups and that they have the same prognosis before the start of intervention. A majority of the quasi-experimental studies were controlled before after studies and most of them either did not provide the required information or provided incomplete information on baseline balance between control and treatment arms or there was a high degree of imbalance at baseline leading to high risk or some concerns of selection bias. For most of the randomised and quasi-experimental studies, we identified concerns related to the way vaccination coverage outcomes were measured. This was mainly due to the use of caregiver reported (self-reported) measures of vaccinations received by a child in the absence and/or incompleteness of immunisation cards, which is common in LMICs and is often influenced by the intervention itself and subject to recall or social desirability biases.

Risk of bias assessments were conducted on the 47 included qualitative papers and their quality was generally high. Papers were scored as absent, weak, or present on 12 key elements and received corresponding scores of zero, one, or two for these ratings. Most (27) papers were missing at least some key elements, resulting in them receiving a zero value for these elements in their quality assessment scores. However, 18 of the 46 qualitative papers received quality assessment scores over 20, indicating that they received a value of two, or a ‘strong’ rating, for most key elements. The most common key elements to be missing were descriptions of sample characteristics and the analytic methods, with 17 studies failing to report on each of these.

The quality of the cost evidence of the 22 studies that included any kind of cost evidence was mixed. About half of the included evaluations appear to have carried out a planned, organised, cost analysis which included stating the form of economic evaluation, indicating the perspective from which the costing is carried out and describing the method of cost data collection. Indications of the quality of underlying cost data also are mixed—just over half (12) of the 22 studies reviewed in the cost analysis used expenditures (rather than budgets) to produce total cost estimates, whereas other studies simply did not report the source of cost data. About half of the included cost studies had high quality, detailed description of costs and cost analyses—which is essential for determining the comparability of total and average cost estimates.

Review questions 1 and 2

The evidence indicates that community engagement interventions had a small but significant positive effect on all the primary immunisation outcomes related to coverage and their timeliness. The findings are robust to exclusion of studies assessed as high risk of bias. The average pooled effect on full immunisation coverage in the random effects (RE) model was an increase of 0.14 standard deviations units (95% confidence interval [CI]: 0.06, 0.23) across all kinds of community engagement interventions. The community engagement interventions also had positive and mostly significant effect on the timeliness of vaccinations and led to an average pooled increase of 0.15 standard deviation units (95% CI: 0.07, 0. 24) in the timeliness of full immunisation.

We found that among the four types of community engagement interventions, it was the engagement as the intervention (engagement is embedded), which involves creation of community buy-in or development of new cadres of community-based structures, that had consistent positive effects on more primary vaccination coverage outcomes than the other engagement types. For example, it had a small but positive and significant increase in full immunisation coverage by 0.08 standard deviations (95% CI: 0.03, 0.13). The evidence base for sub-group analysis for female children was sparse (only 2 studies) and the effect on both full immunisation and DPT3 coverage for this group was insignificant.

Review question 3

The qualitative synthesis highlighted the importance of accounting for contextual factors which could act as barriers or facilitators to immunisation. Several studies note that limited availability of services, especially insufficient staff and vaccine supply, were dominant barriers to immunisation, affecting outcomes in the early portion of the causal chain. There was more variation in barriers to related to social norms, fear, and an understanding of the importance of immunisations by type of engagement. Poor quality of services, including uninviting attitudes of health workers, posed a barrier to immunisation in communities that received engagement as the intervention or were engaged in the design of the intervention. Interventions that even acknowledge or address these barriers are likely to be more effective in improving outcomes. Studies also note that these very constraints could become facilitators or enablers of favourable outcomes provided a study has adequately situated itself to leverage them. Across all engagement types, most studies associated favourable social norms, caregivers' awareness and perception of the benefits of vaccination and high maternal education rates to improved immunisation outcomes.

In terms of programme design, intervention features or characteristics, across all engagement types, were often associated with intervention success or failure of an intervention. When it came to success, certain aspects of community engagement itself such as conducting stakeholder consultations, holding community dialogues or involving community leaders were associated with better immunisation outcomes. Studies also attributed intervention success to nonengagement intervention features including incentives given to caregivers, leadership and supportive supervision, which improved overall health service delivery and health worker performance. Among the studies which attributed intervention failure to intervention characteristics, inadequate duration frequency or exposure to the intervention were the most notable reasons.

Implementation failures, such as low fidelity, were a common reason for intervention failure. Across all engagement types, most studies did not properly account for realities on the ground and were forced to change their implementation plans. These issues may have cropped due to exogenous or uncontrollable factors or may have been related to invalid theory of change assumptions. These issues, in turn, may affect the early portion of the causal chain such intervention uptake.

This synthesis also tried to tease out factors associated with mechanisms of change. Studies reported that certain uptake and fidelity challenges may have resulted in inefficient mechanisms for change. For example, administrative challenges, particularly related to technical issues and communication, were common. There may be other mechanisms of change underlying the causal chain of an intervention. However, an extensive analysis was not possible due to limited reporting by studies on mechanisms of change and the factors influencing them.

Review question 4

Among the studies for which we calculated cost-effectiveness, we find that the non-vaccine cost per dose of intervention to increase immunisation coverage by 1% averaged US $44.10 (all costs are reported in 2019 US dollars). This estimate of cost-effectiveness excludes an outlier observation from Carnell (2014) for which the cost estimate is unreliable. There are two other outlier observations that drive up average cost-effectiveness. The average cost per vaccine dose to increase immunisation coverage by 1% without the three outliers was US $3.97. The range of cost-effectiveness estimates varied from a minimum of $0.89 to a maximum of $29.98 (the excluded estimate for Carnell was $641.08).

The cost-efficiency of the interventions was measured by calculating the non-vaccine intervention cost per vaccine dose. Cost-efficiency ranged from a minimum of US $3.87 per vaccine dose to a maximum of US $5449 per vaccine dose. The average cost-efficiency was US $42.34 per vaccine dose (when the maximum value is excluded). The observed variation in cost-efficiency is at least partly explained by the inclusion of the most costly intervention which included health-system building activities (i.e., building and staffing a health post) and is several orders of magnitude larger than the nearest estimate of the non-vaccine intervention cost per vaccine dose.

Implications for policy makers and practitioners

Generally, community engagement interventions were successful in improving immunisation outcomes. The effects were robust to exclusion of high risk of bias studies. The effects were also uniform across geographies and baseline immunisation rates. However, some engagement approaches, like those with embedded community engagement, appear to be more effective than the others.

The results suggest it is important to pay particular attention when designing and implementing interventions in the following areas:

Implications for research

While we found significant heterogeneity across a multitude of immunisation-related outcomes, there were inconsistent effects related to moderators that might explain the variability among the studies. The risk of bias analysis has shown that, for the randomised studies, while the confounding bias is relatively low, researchers should rely less on self-reported outcome measures of vaccination coverage, which are more susceptible to biases. A majority of non-randomised studies were controlled before after designs and had a high risk of selection bias and confounding, due to incomplete or omission in reporting of baseline balance or having high baseline imbalance. There are therefore opportunities to conduct these studies with more rigorous evaluation designs. There are concerns related to reporting, in particular for quasi-experimental studies there is a lack of transparency with regard to how authors addressed potential contamination or how authors responded to implementation problems (e.g., attrition).

Researchers should consider the following when undertaking IE in this area:

Better reporting of interventions: In many cases, we had difficulties in identifying precisely what the impact evaluation was evaluating due to limited reporting of the intervention components and characteristics. In addition, most studies lacked or had inadequate description and discussion of the intervention theory of change. This limits the amount of learning that can take place from the studies. Authors should consider drawing on tools such as The Template for Intervention Description and Replication (TIDieR) reporting guidelines for health.

Consideration of equity: There is a lack of research on how community engagement interventions affect immunisation outcomes by gender, income or for hard to reach populations. Sub-group analysis needs to be incorporated right at the beginning of the evaluation and not as an afterthought. More disaggregated data by characteristics of interest, such as sex, socio-economics status, religion, etc. is needed to draw valid conclusions about potential differences in the impacts of community engagement interventions on immunisations by them.

Prioritisation of mixed-methods IE and greater focus on intermediate outcomes: Few studies incorporated qualitative research that would allow them to uncover the mechanisms that lead to the success or failure of the intervention. Drawing from qualitative work in mixed methods studies, evaluators should be sure to report on why they think their interventions worked, not just if the interventions worked. Moreover, greater focus on effect of interventions on intermediate outcomes, in addition to final immunisation outcomes, will improve the learnings on mechanisms of change. These crucial considerations will ensure the best use of their research in informing future policy and practice.

Improved and standardised reporting of cost data and analysis: Detailed cost data tables, with the clear descriptive information about the data, methods, unit costs of resources used, and cost adjustments that may have been performed on the cost data is not currently standard practice, but should be. Reporting of total costs, average costs, and marginal costs per impact, and the cost per vaccine dose per child all should be reported for evaluated immunisation programs. There is a wide range of available guidance to assist researchers in this reporting.

Donors play a key role in driving the demand and allocating needed budget for sub-group and cost analysis in IE: Research teams are more likely to plan and report on sub-group and cost analysis when the when the donor requires it —starting with a request for the relevant information in the Request For Proposal (RFP) and when donors enforce its reporting at program's end.

Engagement as the intervention (engagement is embedded)

In these interventions a serious attempt was made to gain community buy-in for activities or new cadres of community-based structures were established, such as village health committees or community health volunteers. While, in a few cases, the development of new cadres can be solely for the purpose of didactic teaching and one-way communication, it generally results in dynamic discussion and two-way communication. As such, we chose to include these interventions. Interventions which are themselves community engagement can motivate communities to take ownership of service delivery and address local problems with local solutions.

Engagement in the design of interventions

In these interventions community input or feedback was sought before the implementation of an intervention. Such feedback can take the form of a pilot, needs assessment, formative evaluation, or other outreach effort. This form of engagement must occur before the implementer undertakes an action. These interventions allow the community to influence the form of the ultimate action taken. Depending on the weight that is given to the feedback, these engagement activities can align to any of the IAP2 categories other than inform, which was not considered.

Engagement in implementation autonomy of interventions

In many interventions the community is not asked for input in their design, but is utilised in their implementation as health care workers, facilitators, or problem solvers. In the spirit of IAP2 framework to go beyond inform level interventions, we only included those interventions in this category where the community members involved in the implementation of the intervention had some opportunity to affect or influence its implementation. This broadly aligns with the involve, collaborate and empower levels of engagement. Due to the inclusion criteria of some autonomy in implementation, the interventions under this category generally involved an existing community led governance structure which weighed in on implementation decisions or the community providing resources without which the intervention could not be implemented. We excluded interventions in which community members, like community health workers or frontline health workers, were involved in implementation, but no new cadres of health workers were created and they could not influence its implementation. For example, community health workers supplying hygiene kits or doing home visits. In addition, interventions which built capacity of existing cadres of community members or provided supportive supervision were excluded. For example, m-health apps for community workers and training of peer facilitators or community health workers or frontline health workers that only allowed for a one-way transfer of knowledge.

Examples of engagement types

ENGAGEMENT AS THE INTERVENTION (ENGAGEMENT IS EMBEDDED)

Effect of health intervention integration within women's self-help groups on collectivisation and healthy practices around reproductive, maternal, neonatal and child health in rural India

Health-focused self-help groups were created for women of reproductive age in marginalised communities.

Effect of peer education on knowledge, attitude and completeness of childhood routine immunisation in a rural community of Plateau State

A new cadre of peer educators was created. Women were trained to provide women with information about routine childhood immunisation. This does not qualify as engagement in implementation because the intervention was the training of peer educators and not the subsequent actions of the peer educators.

Impacts of engaging communities through traditional and religious leaders on vaccination coverage in Cross River State, Nigeria

Traditional and religious leaders were trained to utilise their leadership role to support immunisation. After the training, these leaders then presented data at ward development committee meetings and engaged with the community to encourage immunsation.

ENGAGEMENT IN THE DESIGN OF AN INTERVENTION

Cognitive behaviour therapy-based intervention by community health workers for mothers with depression and their infants in rural Pakistan: a cluster-randomised controlled trial

Many pilots directly seek the feedback of community members. In this study, health workers and depressed mothers were asked about the relevance and usefulness of the intervention before scaling.

Mobile Phone Incentives for Childhood Immunisations in Rural India

Caregivers were given mobile phone credit incentives for completing immunisations. The amount of the incentive was decided upon through conversations that involved community members.

ENGAGEMENT IN THE IMPLEMENTATION OF AN INTERVENTION

The impact of an immunisation programme administered through the Growth Monitoring Programme Plus as an alternative way of implementing Integrated Management of Childhood Illnesses in urban-slum areas of Lusaka, Zambia

The GMP+ sessions were conducted by medical personnel from Public Health Centers. During these session, community volunteers provided some operational and managerial support to ensure the effective implementation of the sessions.

Effects of payment for performance on accountability mechanisms: Evidence from Pwani, Tanzania

The intervention is a performance-based financing mechanism. The community health committee was involved in decisions about how to spend the funds gained through the pay for performance mechanism.

Types of study designs

Types of participants

Rural, peri-urban and urban populations in LMICs were considered. Country's income status was determined by its World Bank classification at the time an intervention was carried out. Our review focused on interventions targeting community members, but interventions need not target the community as a whole. Studies that targeted subgroups were included.

Types of interventions

Interventions falling into one of the three types of community engagement outlined in Section 4.2 were included.

Types of outcomes

The primary outcomes considered in this review are coverage rates for (a)full immunisation, (b) third dose of DPT or pentavalent, (c) measles, or (d) the timeliness of any of these doses. Additional antigen-specific immunisation coverage outcomes and secondary outcomes reflecting upstream conditions (e.g., attitudes about vaccination and access to immunisation services) and downstream effects (e.g., morbidity and mortality) of the primary outcomes were also included. Some outcome measures differed in degree (full, partial or no routine immunisation) or were a subset of each other (dropout after DPT1/Penta 1 represents partial immunisation). These were included to account for differences in reporting. If heterogeneity in results was explored by subgroups, subgroup specific outcomes were considered.

Official health records and parent recall were considered acceptable measures of immunisation coverage. If a study reported both types of measures, we extracted data on both. However, we used the information from official health records (immunisation card) as the ‘true’ measure of the outcome.

Although there are standardised tools for measuring constructs like parental attitudes towards vaccination (e.g., Oladejo et al., 2016; Shapiro et al., 2018), these have not yet been universally adopted or validated. Therefore, we deferred to authors' definitions of these constructs and their instruments for measuring them. The list of outcomes along with their definitions is provided in Supporting Information: Appendix B.

Duration of follow up

Because the timeline for administering most childhood vaccinations is short (most should be administered in the first 14 weeks after birth), there were no restrictions on duration of follow-up.

Types of settings

Studies must have been conducted in a LMIC. There were no additional restrictions on types of settings.

Search strategy to respond to quantitative research questions 1, 2, and 4

The search strategy for this systematic review was combined with that of an evidence gap map on the state of evidence in the immunisation sector. The EGM had a broad scope and included IE and SR, both completed and ongoing, that report findings on effectiveness of any intervention on immunisation outcomes of children in LMICs. It was developed in consultation with an information specialist.

Search strategy to respond to qualitative research question 3

After identifying included IE, we undertook a targeted search for supporting documentation related to the interventions evaluated in the included studies (Supporting Information: Appendix D). We searched clinical trial registries and databases using the names of programmes from included studies. We conducted targeted searches of websites of implementing agencies and funder websites.

Search strategy to respond to cost analysis question 4

The evaluations included in the cost and cost-effectiveness evidence were drawn from the 61 evaluations that were identified for the systematic review. Since cost reporting and analysis is often not required by donors or included in IE of global development interventions, we undertook structured outreach to the authors of the 61 included studies to request any additional cost evidence. In case the lead or corresponding author email addresses were nonfunctional, we contacted one of the co-authors.

We successfully contacted 58 out of 61 study authors that reported intervention effectiveness only or intervention effectiveness and partial cost information and requested estimates of the intervention costs as well as any additional raw data files, summary tables, sensitivity analyses, cost evidence, published reports or notes, and non-published documentation, methods or analyses relating to cost-effectiveness. In the first two weeks of this request, our team obtained responses from a total of 25 study authors, of which 12 produced or pointed us in the direction of economic evidence for their respective interventions and 7 authors confirmed that any such costing analysis was not undertaken as part of their study. We excluded any studies that did not report estimates of both costs and effectiveness and identified 22 evaluations with both cost and effectiveness estimates.

Description of methods used in primary research

Quantitative methods were expected to rely heavily on cluster randomisation at the clinic level. Where randomisation was not possible, difference-in-difference and fixed effect estimation were likely to be used. Other methods, such as interrupted time series and synthetic control, were possible but expected to be less likely.

Criteria for determination of independent findings

Statistically dependent effect sizes occurred when several publications used the same underlying data, multiple treatment arms were compared to a single control, similar analyses were conducted at different times, or the different outcome measures were used to reflect a single underlying construct (Borenstein et al., 2009).

To avoid double-counting of evidence from different papers that focus on the same study, we linked these papers before analysis. We extracted data from the most recent publication when several publications reported on the same effect. After extracting the effect sizes from the main paper, we only extracted data on outcome measures and samples or subgroups from the linked papers that did not appear in the main paper.

Where studies reported data for the same outcome for an intervention over multiple time periods, we extracted data for each of the reported time periods. Selected time points that were most similar across papers for inclusion in the meta-analysis. Where authors reported the same outcome using more than one analytical model, we extracted the data from the authors' preferred model specification. Where the authors did not specify a preference, we extracted data from the model with the most controls.

Where studies reported an index of different outcomes and the effects on the individual factors that comprised the index, we extracted data for the overall index measure. Where studies reported outcomes or evidence according to subgroups of participants, we extracted data on both the full sample (where possible) and on the individual subgroups, to answer Review Question 3 regarding differential effects by population type.

Where studies reported outcomes related to multiple treatment arms and only one comparison group, we extracted the data and estimated an effect size for each of the treatment arms. Further discussion of selection of independent effects can be found in Section 5.3.2.

Selection of studies

Quantitative search results were screened on title/abstract by a team of trained reviewers. Two reviewers independently screened each abstract. After screening a ‘training set’ of 1000 random records, we activated EPPI-Reviewer's machine learning algorithm (O'Mara-Eves et al., 2015; Thomas et al., 2011), which ranked the remaining records according to their likelihood of inclusion, based on data from the training set. We then screened the prioritised list of records in order, with the machine learning function continually re-prioritising the list based on the additional screening data. We intended to cease screening after reaching a point where 500 consecutive records had been excluded. However, this never occurred and all abstracts were screened.

During title and abstract screening, weekly reconciliation meetings were held to discuss and resolve disagreements. Studies included at the title/abstract level were retrieved for full review. Full texts were also independently screened by two reviewers, with disagreements resolved through discussion.

Qualitative papers were screened as they were identified in the search and reviewed again before they were fully coded.

Quantitative data extraction

We used Microsoft Excel to extract descriptive information and effect sizes from included studies. This information was double-coded using procedures set out in a codebook developed as part of the review protocol. Coders reconciled their answers. If an agreement could not be reached, a core team member was consulted. A third reviewer undertook spot checks for quality assurance (Supporting Information: Appendix E1). Cost data was single coded and checked by a core team member. The following broad categories of data were extracted (Supporting Information: Appendix E2):

Descriptive data including authors, publication date and status, as well as other information to characterise the study including country, type of intervention and outcome, population, and cost study context including implementing partners, analytical perspective, intervention activities, intervention ingredients, intervention exposure, and cost data sources.

We extracted effects reported across different outcomes or subgroups within a study, and where information was collected on the same programme for different outcomes at the same or different periods of time, we extracted information on the full range of outcomes over time. Where studies reported effects from multiple model specifications, we used the author's preferred model specification. If this was not stated or was unclear, we used the specification with the most controls. Where studies reported multiple outcomes or evidence according to subgroups of participants, we recorded data on relevant subgroups separately.

Qualitative data extraction

All impact evaluation and additional documentation identified in the search were coded in Nvivo. CL developed an initial set of themes related to this study question with support from AB and MJ. Themes included barriers and facilitators, reasons for project success or failure, and uptake and fidelity challenges (Table 2). Other themes related to the data sources, research design and inclusion criteria were also included for potential future analyses (Supporting Information: Appendix E3). Four consultants were trained on the application of these thematic codes in Nvivo (various versions). Consultants selected all text related to the provided list of themes within the assigned documents. They were encouraged to add themes as they were identified, that is, deductive coding was applied. However, inductive coding was applied to the Reasons for project success or failure code because we did not have any a priori expectations and it contained more information than anticipated.

For the majority of the coding period, AB and CL reviewed the work of consultants on a weekly basis and provided written and/or verbal feedback. After coding was completed, CL and AB reviewed the coding for each theme that would be analysed to ensure consistency across consultants and over time. Reviews were conducted by reading the Nvivo files; therefore, they largely resulted in the un-coding or recoding of text that had been selected by consultants rather than the addition of new text. During the final review, redundant information related to the same impact evaluation, whether it was identified in the same paper or a different paper, was uncoded. Different pieces of information related to the same theme for the same evaluation were not uncoded. In some cases, uncoded text was added to the annotations of the coded text so that it could be retrieved later.

Assessment of risk of bias in IE

The risk of bias assessments for each included study were conducted by two independent reviewers, with disagreements resolved through discussion (and the core team member making final decisions on any contested cases). We used the Cochrane Non-Randomised Studies Group and procedures recommended by Waddington et al. (2017) to develop a standardised set of categories of bias (Supporting Information: Appendix F1). We coded papers as ‘Yes,’ ‘Probably Yes’, ‘Probably No’, ‘No’ and ‘No information’, reflecting if each category was free from the bias. Overall risk of bias was scored as low when all domains were assessed as yes or probably yes, high when any domain was no or probably no, and some concerns when one or more domains had no information, but the remaining domains were yes or probably yes. We included all studies, regardless of risk of bias rating, in the synthesis, but we conducted a sensitivity analysis wherever possible (e.g., where there were at least two studies not assessed as high risk of bias) leaving out studies assessed as high risk of bias. We report both results for all instances where sensitivity analyses were completed.

Bias arising from the randomisation process (for RCT's) or confounding bias (for quasi-experimental designs)

Critical appraisal of qualitative studies

We critically appraised qualitative and mixed-methods studies using an adaptation of the nine-item framework developed by the Critical Appraisal Skills Programme (CASP 2018). The critical appraisal tool was piloted by C. L. and Y. V. to assess inter-rater reliability of the tool. A. B., C. L., M. J., and Y. V. collectively refined the assessment tool pre and post pilot. Appraisals were done by Y. V. Detailed notes were kept while Y. V. conducted the appraisal and referred back to during two phases of self-checking.

The tool consisted of 22 questions arranged according to the order in which a standard academic article is arranged: Introduction, Methodology, Results, Discussion (Supporting Information: Appendix F2). For each question there were three possible responses ‘strong’, ‘weak’, or ‘none’ with slight variations in phrasing to suit each question. Ethical approvals were coded as present/absent. Of the 22 questions, 12 represented key elements which we expected to be present in all qualitative papers. Scores were developed by assigning 2, 1, or 0 for each key element, corresponding to ‘strong’, ‘weak’, and ‘none’, and summing these values.

Documents that provided supporting descriptive or quantitative information on the design, delivery, or context of interventions did not undergo critical appraisal because these documents were not expected to fulfil the criteria of many of the questions in the appraisal tool. The purpose of including them in our review was to ensure we had all possible information about the context and interventions included in our review. These documents helped triangulate the information extracted from the quantitative and qualitative documents.

Critical appraisal of cost evidence

For appraisal of cost evidence we assessed the quality of underlying cost data, reporting, and analysis using information that was included in the immunisation studies. We assessed risk of bias along six primary dimensions which were adapted from a combination of tools (Supporting Information: Appendix F3), including: Doocy and Tappis (2017); Campbell Collaboration Economic Methods Policy Brief (Shemilt et al., 2008); and Methods for the Economic Evaluation of Health Care Programmes (Drummond et al., 2015).

This risk of bias tool specifically assesses the bias that arises from the collection and reporting of cost analysis in conjunction with impact evaluation studies of global development interventions. These studies often do not incorporate cost analyses. Indeed recent estimates suggest just 15%–18% of impact analyses include any kind of cost analysis (Brown & Tanner, 2019). This under-reporting of cost in conjunction with IE leads to very small samples from which to draw inferences about the cost, and cost-effectiveness of development interventions. Moreover, often when estimates are included with IE, the quality of the underlying data is low. Cost may have been added as an ‘after-thought’, rather than planned for in advance or treated as a research endeavour. Data sources for cost information may not be well-documented or may have been estimated using ‘back-of-the-envelope’ techniques (this phrase literally is used in write-ups of cost methods). There is often insufficient detail in reporting of cost to assess the quality of estimates and data and basic robustness checks of the analysis are very infrequently performed.

Finally, this tool specifically considers the elements of cost and effectiveness that were extracted for the analysis of incremental cost effectiveness. Our particular measure estimates the non-vaccine cost per dose of interventions to increase absolute immunisation coverage by 1%, using available data extracted from the evaluations wherever possible, or by adapt the steps and calculations as outlined in Ozawa et al. (2018). The specific inputs to this analysis include estimates of: total intervention cost (net of vaccine cost); the number of vaccine doses provided to each child; the endline proportion of children that received immunisations in the treatment and control groups; and the cost per child immunised.

Review questions 1 and 2: Statistical meta-analysis

Once all effect sizes were calculated and converted to a SMD, we examined the data for outliers. We defined outliers as any effect sizes ±3.29 standard deviations from the mean (Tabachnick & Fidell, 2001). Outliers were confirmed to be sure they were not errors or spurious.

We conducted meta-analyses of all sets of two or more studies with sufficiently similar intervention, outcome, and comparison groupings. This approach was taken by Wilson and colleagues (2011). While we recognise the potential statistical bias in combining only two studies (namely that we cannot be certain about the dispersion of the effects), the conclusion we can draw from a statistical summary (albeit with limitations) is preferable to the unknown bias of an ad hoc summary that is less transparent and less valid (Borenstein et al., 2009; Ryan, 2016; Valentine et al., 2010). We used the metafor package (Viechtbauer, 2010) and/or the robumeta package (Fisher & Tipton, 2015) in R software to conduct the meta-analyses (R Core Team, 2020). At the conclusion of the results section, we also include a table of the summary effects that has been converted to the estimated percent increase in the intervention group compared to the control group. This conversion is akin to the What Works Clearinghouse ‘improvement index’, whose computation is described in their Procedures Handbook, Version 4.0 (What Works Clearinghouse, 2017).

We dealt with dependent effect sizes by using robust variance estimation (RVE: Fisher & Tipton, 2015; Hedges et al., 2010) or data processing and selection techniques. The RVE approach allows us to use all available data in our effect size estimates, even data that is statistically dependent. However, these analyses must have >4 degrees of freedom to make valid inferences. In all cases, we first report the average estimated effects from the RE models using independent effects for the outcome category, and then separately for subgroup analyses by intervention type (and other subgroups where data are available) as this is our preferred model specification. The dependencies in our data arise most often due to multiple intervention arms compared to a single control, or due to several different data sources being used for estimation (e.g., immunisation card and caregiver recall). We believe we have chosen criteria for selection that increases the reliability of the data in our independent RE models (i.e., preferring immunisation card data over recall data) as well as selections that are most salient for practitioners (i.e., choosing the effect from the intervention i.e. being scaled up or that has been determined to be the most cost effective). While we will present the estimates for both, we will only present plots and figures from the models using traditional meta-analysis with independent effects and will draw largely on the traditional meta-analysis models for our discussion and conclusions. Following the presentation of the RE models we present the robust variance estimation for each outcome category (if sufficiently powered), as a robustness check.

The following data processing approaches were used to ensure the independence of data:

Several publications reported on the same study: We used effect sizes from the most recent publication or used the one which contains more detailed reporting of results.

Review question 3: Qualitative synthesis

Qualitative analysis applied a logic model approach to comprehend underlying processes by which quantitative outcomes could have been achieved (Harden, 2017). This allowed for the qualitative research to directly relate to the quantitative work. Project context, design, and implementation can all affect outcomes along the causal chain. These factors are generally interrelated: project design should account for local context and implementation is influenced by context and design. To understand the interrelationship between these, we identified themes related to barriers to immunisation, facilitators of immunisation, reasons for project success, reasons for project failure, and uptake and fidelity challenges. In the discussion section, we then arranged these themes based on their relevance to context, design, and implementation.

Hierarchy charts related to barriers, facilitators, reasons for project success or failure, and uptake and fidelity challenges were created in Nvivo 12. They reflect the frequency at which a theme was reported by an impact evaluation or its supporting documents. The most frequently cited sub-themes were selected for reporting. Quotes illustrating these themes were selected based on (1) their representativeness of the other coded text, (2) their ability to stand alone, without context, and (3) brevity. A mixture of direct quotes from study participants and author experiences were selected. Although information from literature reviews was coded, this was generally not included in the selected quotes because it represents secondary data.

Review question 4: Cost analysis

We conducted an initial review and inventory of the cost and effectiveness evidence for each of the 22 evaluations using a CEA Data Inventory Tool. A single analyst conducted the inventory by reviewing the full-text of the published impact evaluation and any additional evidence of cost or cost-effectiveness that was collected. For each study, the analyst documented the evaluation outcomes, number of treated units, treatment arms; and created binary indicators (and documented the tables and page numbers) for any reported: cost-effectiveness ratios, quantitative discussions of cost (and by type), any cost claims, any cost data tables and their contents. The analyst additionally documented and inventoried any additional cost data files and publications of cost-evidence that were produced separately from the impact evaluation report.

The core team member reviewed all 22 of the evaluations selected for the cost and cost-effectiveness evidence synthesis and the data inventory assembled by the team. This initial review confirmed wide variation in the quality and type of cost evidence reported in the evaluation studies. The core team member developed a data extraction tool which was piloted for initial data extraction on several studies by two independent analysts. The team collectively reviewed the extracted information and resolved any differences in the interpretation of the evidence through discussion.

Two analysts split the sample of 22 evaluations and independently extracted the data. The core team member independently reviewed the extracted data from each evaluation. For each of the included evaluations, we extracted the author, publication year, and country in which the evaluation took place. We extracted information on the context of the costing study, including the targeted population, implementing partners, analytical perspective, intervention activities, intervention ingredients, intervention exposure, cost data source, methods, and type of efficiency analysis. We extracted information necessary to convert costs into common values, including the base year of the costing, currency of expenditure, exchange rate, exchange rate year, inflation rate, and discount rate if reported. We extracted total intervention cost (excluding the cost of vaccines), average costs and their denominators, cost per vaccine delivered, and average and marginal cost per immunised child. We extracted immunisation outcomes, including the types of vaccines, number of vaccine doses reported, baseline and final immunisation coverage, and the estimate program impacts on the immunisation status of participants in the treatment and control groups at baseline and endline.

To ensure the comparability of the extracted cost elements, we converted expenditures in foreign currency to $US dollars using the official exchange rate (LCU per US$, period average) from the International Financial Statistics of the International Monetary Fund. Historical exchange rates were retrieved from the World Bank's World Development Indicators database.

The goal of the cost and cost-effectiveness review was to compare the cost-effectiveness of the included studies. However, a chronic and persistent challenge with assembling comparable cost-effectiveness estimates is the inconsistency of reporting—both in terms of the elements of cost selected for reporting in each evaluation (e.g., do we report total costs, average cost or marginal costs) but also in terms of how those costs are analysed in tandem with impact estimates. Available guidance for the systematic review of economic evaluation is still under development. Chapter 20 of the Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020; Aluko et al., 2020) has yet to provide guidance for conducting an integrated full systematic review of economic evidence.

A few recent SR have begun to address the challenge of synthesising economic evidence. For example, Doocy and Tappis (2017) systematically review cash-based approaches in humanitarian emergencies reporting total cost, cost-effectiveness and cost-efficiency where those values are computed by the study authors. The challenge of course is that only a modicum of evaluations actually report the key information. This limits the ability to synthesise the cost-effectiveness evidence.

Several recent SR of immunisation evidence (Munk et al., 2019; Ozawa et al., 2012, 2018; Portnoy et al., 2021) begin to assemble cost evidence in ways that enable comparisons between evaluations. The analysis here adapts the approach applied in (Ozawa et al., 2018) to synthesise the cost-effectiveness of the 22 evaluations that included elements needed to estimate the cost-effectiveness of community participation interventions to improve child immunisation in LMICs.

We estimate the non-vaccine cost per dose of interventions to increase absolute immunisation coverage by 1% using available data extracted from the evaluations wherever possible, or by adapting  the steps and calculations as outlined in Ozawa et al. (2018).

Step 1: Extract total cost or estimate total intervention cost, if needed.

A majority of the included evaluations do report an estimate of total program costs, excluding the cost of vaccines. In cases where vaccine costs were included we refer to detailed tables to subtract vaccine cost from the total. In the case where the total intervention cost is not reported we estimate total cost by multiplying the average cost per beneficiary by the number of individuals in the treatment group. If cost data only are reported for a single intervention arm, we estimate the total cost of treatment in that arm and report in separate lines for more multiple treatment arms.

Step 2: Extract vaccine change in coverage or calculate change in vaccination coverage if needed

A majority of the included evaluations report the difference in immunisation coverage between treatment and control groups. In this case, we estimate the change in vaccination coverage as the difference in the proportion of children immunised in the treatment group—the proportion of children immunised in the control group at endline. If these differences also are reported by treatment arm, we break out the reporting of coverage change by treatment arm. Several studies do not have sufficient information to calculate this change, in which case they are dropped from the analysis. In the case where studies report a null or negative effect, we do not calculate vaccine coverage change. Using the method described in Ozawa et al. (2018), we take the average of the difference in vaccination coverage in cases where vaccine coverage changes are reported for each vaccine type (i.e., separately for measles and DPT3, two common childhood vaccines). However in cases where the corresponding cost information also is available (and we can calculate the cost per vaccine dose), we calculate the cost per dose for the individual vaccines, rather than take the average. In with more than one intervention and for which authors report intervention costs and coverage information separately, we reported the multiple interventions separately.

Step 3: Extract the number of vaccine doses delivered in each study and the cost per dose when reported, or calculate if needed.

We extract the number of vaccine doses delivered when it is reported, and the cost per dose. However, in many cases the number of vaccine doses is not reported. Using the method described in Ozawa et al. (2018), we estimate the number of doses that a child in the requisite age group should receive per the appropriate country-specific vaccination protocol that was in use at the time the study was conducted. We estimate number of vaccine doses per protocol (as described in the study itself or using standard protocols for time/country.

Using this information together with the coverage information, we estimate cost per dose (a measure of cost-efficiency) by using the following approach:

Cost per vaccine dose = ( C / n t ) * n t v n t / v d , $\text{Cost}\,\text{per}\,\text{vaccine}\,\text{dose}=(C/n_t)* \left(\frac{n_t^v}{n_t}\right)/v_d,$

where C is the total cost; n _t is the number treated;

n t v $n_t^v$

n t v n t $\left(\frac{n_t^v}{n_t}\right)$

Step 4: Calculate the non-vaccine cost per dose of interventions to increase absolute immunisation coverage by 1%.

Subgroup analysis and investigation of heterogeneity

Heterogeneity in findings by type of engagement was examined. In the meta-analysis, this was considered through subgroup analysis. In the qualitative analysis, this was done by developing hierarchy charts for each type of engagement and considering themes separately by type of engagement. For the meta-analysis, four types of community engagement were considered: engagement as the interventions, engagement in the design of the intervention, engagement in the implementation of the intervention, and multiple levels of engagement. In the qualitative analysis, studies that used multiple types of engagement were considered to fall into each of the categories of engagement they used. This was because there was insufficient data to consider this group separately. In addition, contextual information found through the qualitative work is likely to apply equally to each type of engagement that a study used. Where qualitative findings were similar across levels of engagement, these are reported together; where qualitative findings are meaningfully different by level of engagement, results are presented in separate sections.

Wherever there were at least four studies contributing effects (for RE models with independent effects) and there was evidence of heterogeneity, that heterogeneity was examined through moderator analysis. In RVE analyses, moderator analyses were only conducted when there were at least 4 degrees of freedom for the predictor (when degrees of freedom are less than 4, we cannot make valid inferences). Where there were too few studies, or included studies were too heterogeneous in terms of interventions or outcomes, we present a discussion of individual effect sizes along the causal chain. As heterogeneity exists in theory due to the variety of interventions and contexts included, we used inverse-variance weighted, RE meta-analytic models (Borenstein, Hedges, Higgins & Rothstein, 2009).

Sensitivity analysis of quantitative studies

We conducted sensitivity analysis to assess whether the results of the meta-analysis were sensitive to the removal of any single study. We did this by removing studies from the meta-analysis one-by-one and assessing changes in results. In the context of analyses using independent effects, studentized residuals and Cook's distances are used to examine whether studies may be outliers and/or influential in the context of the model (Viechtbauer & Cheung, 2010). Studies with a studentized residual larger than the

100 × ( 1 − 0.05 / ( 2 × k ) ) $100\times (1-0.05/(2\times k))$

α = 0.05 $\alpha =0.05$

k $k$

Results of the search

Due to the broad initial search strategy of the EGM, we retrieved 43,208 results through our database and website searches and citation tracking. After de-duplication, we identified 29,481 unique abstracts for screening. Of these, 1285 articles were screened at full-text. After eliminating the studies that did not meet our scope, we arrived at 61 IEs that met our inclusion criteria for this review. Figure 2 summarises how the included studies were selected from the search results.

OPEN IN VIEWER

Figure 2

PRISMA diagram of search and screening process. PRISMA, preferred reporting items for systematic reviews and meta-analyses.

We identified 116 additional papers through the qualitative search process. The distribution of these papers by document type is provided in Table 3.

Included studies

This section gives an overview of the characteristics of included studies, including geographic coverage, breakdown by community engagement type, outcomes and study design characteristics. A comprehensive overview of included studies is presented in Supporting Information: Appendix G: Table 1.

Geographic distribution of the impact evaluation evidence base

Though mostly concentrated in South Asia (32) and Sub-Saharan Africa (25), the studies included in this review are spread across 19 LMICs. Within these two regions, the majority of the studies were from India (21) and Nigeria (8). Figure 3 provides the geographic coverage of the included studies and number.

OPEN IN VIEWER

Figure 3

Geographic distribution of the studies and number of studies by country

Excluded studies

The main reason for exclusion at the title stage was irrelevance or not considering outcomes related to routine immunisation in LMICs (26,611). The most common reasons for exclusion at full text were not meeting the country (120), outcome (229), study design (338) or community engagement criteria (165).

Quantitative risk of bias

Risk of bias in RCTs

Of the included studies with experimental designs (k = 31) the majority (k = 23) have been identified as being at high risk of bias, and only two were rated as having a low risk of bias. Six studies were assessed as having some concerns. Of the six potential causes of bias that were analysed, outcome measurement bias and deviations from intended interventions were the most commonly documented issues. The most common pitfall related to outcome measurement bias was the use of caregiver reported (self-reported) measures of vaccinations received by a child in the absence and/or incompleteness of immunisation cards, which is common in LMICs and is often influenced by the intervention itself. One such study stated that ‘The calculation of FIC coverage was based on data from the child's immunisation card and/or the mother's recall. Mother's recall was taken in situations in which the card or specific data points on the card were missing’ (Gurley, 2020). Relatedly, Demilew (2021) reported that their outcome measures ‘…could be subject to recall bias or social desirability bias differentially by treatment group….’ Deviations from intended interventions had to do with issues such as potential spillover related to geography (e.g., in Siddiqi et al., 2020; centres were contiguously located) or transferable vouchers (e.g., Morris et al., 2004). Other sources of bias in included RCT studies are reporting bias, and bias in the randomisation process. Figure 4 illustrates the frequencies with which each cause of bias was identified across the 31 included RCTs.

OPEN IN VIEWER

Figure 4

Frequencies of Bias in RCTs. RCT, randomised controlled trial.

Bias in Quasi-Experimental designs

Of the 30 included quasi-experimental studies, the majority (k = 27) present a high risk for bias, two were assessed as low risk of bias and one study was assessed as having some concerns. Like the experimental studies the most common identified causes of bias amongst quasi-experimental designs were issues related to outcome measurement bias and deviations from intended interventions. This is no surprise as self-reporting was again a common concern (e.g., Admassie et al., 2009; Findley et al., 2013). Figure 5 illustrates the frequencies of other biases in our included quasi experimental designs (QEDs).

OPEN IN VIEWER

Figure 5

Frequencies of Bias in QEDs. QED, quasi experimental design.

Qualitative risk of bias

Risk of bias assessments were conducted on 47 qualitative papers. Papers were scored as absent, weak, or present on 12 key elements and received corresponding scores of zero, one, or two for these ratings. Most (27) papers were missing at least some key elements, resulting in them receiving a zero value for these elements in their quality assessment scores (Table 7; Figure 6). However, 17 received quality assessment scores over 20, indicating that they received a value of two, or a ‘strong’ rating, for most key elements (Figure 7). The most common key elements to be missing were descriptions of sample characteristics and the analytic methods, with 17 studies failing to report on each of these (Figure 6). The research aim was the most common key element to be stated strongly, with 39 papers clearly stating their aims.

OPEN IN VIEWER

Table 7

Distribution of papers by strength of key elements

	Number
One or more key elements absent	27
One or more key elements weak	15
All key elements strong	5

OPEN IN VIEWER

Figure 6

Distribution of papers by 12 key elements

OPEN IN VIEWER

Figure 7

Distribution of qualitative assessment scores. Papers received scores of 0, 1 or 2 reflecting the strength of reporting on 12 key elements in the risk of bias assessment tool (Supporting Information: Appendix F2).

Risk of bias in cost effectiveness estimates

The risk of bias analysis of the cost and cost-effectiveness analysis assessed the quality of underlying cost data, reporting, and analysis using information that was reported in the immunisation studies. We assessed risk of bias along six primary dimensions which were adapted from a combination of tools, including: Doocy and Tappis (2017); Campbell Collaboration Economic Methods Policy Brief (Shemilt et al., 2008); and Methods for the Economic Evaluation of Health Care Programmes (Drummond et al., 2015), that were adapted for cost analyses carried out in conjunction with impact evaluation studies of global development interventions.

Results

The risk of bias protocol was applied to the 22 immunisation evaluations that reported cost information. A summary of the six dimensions used to assess risk of bias is summarised below and results of this analysis are shown in Figure 8.

Planned, organised, cost analysis. A common challenge of cost analysis is a lack of planning which often can lead to poor underlying data quality.

Quality of data sources. The highest quality data for assessing the cost of interventions in low-and middle-income settings often are drawn from the expenditure reports or accounting statements of the program implementers, rather than from budgets or using back-of the envelope estimates.

Descriptive, detailed cost information. The quality of descriptive detail on the costs of intervention inputs allows judgement into cost components and whether they align with intervention activities.

We consider the quality of other data elements that were key inputs to the cost synthesis, for example we specifically review the quality of total cost and its components; vaccine costs; the cost per child immunised; and the number of vaccines.

The reporting of key cost details needed to adjust for time and currency differences.

The analytical assessment of cost parameter and cost model uncertainty.

Just over half of the included evaluations appear to have carried out a planned, organised, cost analysis. Thirteen of 22 evaluations clearly state the form of economic evaluation (i.e., cost-effectiveness analysis); 10 studies report the perspective of the costing, which is key for judging the correct inclusion and exclusion criteria for the components of a total cost estimate; and 11 of 22 evaluations describe the method of costing that was used to collect cost data (i.e., the ingredients method).

Indications of the quality of underlying cost data are mixed. Nine of the 22 evaluations used expenditure reports to generate cost estimates, two used budgets and the remaining 11 evaluations provided no information on the provenience of the underlying cost data that was used in the analysis.

The quality of the descriptive detail on reported costs was mixed. Just over half of all evaluations (14 of 22) provided thorough, descriptive information on costs; two evaluations provided some descriptive information, for example, a breakdown of key unit costs; and six evaluations gave very minimal or no descriptive information on costs.

In 17 of 22 evaluations, we have high confidence that total cost excludes vaccines costs. A majority of evaluations reported an estimate of total cost (20 of 22 evaluations). In 16 of the 20 total cost estimates, there were clear indications that vaccine costs had been excluded from the estimate, and in one case, vaccine costs were included but reported separately so that we could subtract vaccine cost from total cost to derive the comparable total cost. It was not possible to tell if vaccines were excluded from the total cost estimates of five evaluations.

The cost per immunised child was only reported by the authors in 5 of 22 evaluations and for the remaining 16 was estimated by the authors to compile Table 15.

The number of vaccine doses received per treated child was reported in only four evaluations; in 15 studies, vaccine doses per child were estimated based on information ‘per protocol’ which does not account for children's partial vaccination status at the point of enrolment in the study.

Fewer than half of the evaluations included the key information needed to adjust for time, currency, inflation or base year differences in the timing of expenditures. This lack of reporting makes it very difficult to have complete confidence in the comparability of the cost estimates we generated.

Only three evaluations reported any kind of sensitivity analysis, an indication of analytical robustness.

OPEN IN VIEWER

Figure 8

Risk of bias assessment of cost and cost-effectiveness estimates used in the analysis. Green cells indicate that clear information was presented in the report to assess the risk of bias; yellow cells indicate incomplete information; and red cell indicate we found very little or no information to assess bias risk.

Quantitative synthesis of results

Below we report the results related to our first review question, which aims to examine the evidence that exists regarding the effectiveness of community engagement interventions in improving routine immunisation coverage of children in LMICs. A total of 61 studies were determined to be includable based on full-text screening, and were assigned to research assistants for double coding and data extraction. Out of the 61 studies assigned for coding, there were five that did not have sufficient quantitative data for us to be able to calculate an effect size. For example, the reports by Tandon and Sahai (1988) and Roy et al. (2008) did not provide the standard deviations necessary to compute a SMD, and Goel et al. (2012) did not report any sample sizes. Memon and colleagues (2015) reported regression coefficients, but did not report the associated standard errors, nor was there a p-value reported from which we could approximate a t-value. This left us with 56 studies includable in the quantitative analysis. Many studies reported on multiple immunisation-related outcomes, thus the 56 studies contributed a total of 738 effect sizes. Throughout, where a study reports outcomes for which a negative movement actually indicates something positive (e.g., a reduction in diarrhea), the sign of the effect has been reversed. Thus, in all cases an increase indicates a positive impact of community engagement interventions.

We began with an overall test examining the impact of any type of community engagement intervention on any immunisation-related outcome, using robust variance estimation to account for the dependency in the data (i.e., a single sample contributing multiple outcomes to the analysis). This omnibus analysis indicated that the estimated average outcome based on the random-effects model was

μ ˆ = 0.08 $\hat{\mu }=0.08$

0.04 $0.04$

12 $12$

p < 0.001 $p\lt 0.001$

μ ˆ = − 0.02 $\hat{\mu }=-0.02$

− 0.03 $-0.03$

− 0.004 ] , p = 0.02 $-0.004],{p}=0.02$

Though the significant and positive movement on immunisation-related outcomes is encouraging, it does not give us sufficient insights into what intervention types might work best for which outcomes. Thus, below we examine more closely the impact of community engagement interventions on specific outcomes, with subgroup analyses for specific intervention types. All moderators (listed in Section 5.9) were tested for each of the analyses below whenever the data allows (e.g., there is sufficient power to test, or enough studies per group to make a valid inference). The subsequent sections explore the following outcomes in turn: full routine immunisation (k = 28), partial immunisation (k = 9), measles vaccination (k = 20), BCG vaccination (k = 12), DPT 1 vaccination (k = 8), DPT 2 vaccination (k = 5), DPT 3 vaccination (k = 22), OPV0 vaccination (k = 5), OPV1 vaccination (k = 5), OPV2 vaccination (k = 5), OPV3 vaccination (k = 9), vaccination timeliness for all vaccinations (k = 11), timeliness of DPT3 (k = 7), timeliness of measles vaccination (k = 2), timeliness of full immunisation (k = 5), dropout rate (k = 5), childhood morbidity (k = 10), childhood mortality (k = 6), knowledge about immunisation (k = 9), attitudes about immunisation (k = 6), and vaccination health card availability/retention (k = 4). Within each outcome category, we examined the specific impacts of four broad categories of community engagement interventions using subgroup analyses: engagement as the intervention, engagement in the design of the intervention, engagement in implementation autonomy, and interventions with multiple engagement types.

There were several outcomes mapped in our initial framework that were either not reported in any of the included studies, or were reported in only one study and thus a synthesis was not possible. There were no included studies that reported on attitudes about health providers, readiness to vaccinate, reasons for not vaccinating, actual cost of vaccination, perceived convenience of vaccination, perception of vaccination side effects, formal health worker supply, availability of health workers at vaccination point of service, administrative staffing, capacity of health administrators responsible for vaccination, quality and completeness of immunisation data collection, defaulter tracing, supply chain management, availability and transparency of immunisation data, quality of cold chain infrastructure, national or subnational vaccine financing and IPV vaccination. There was only one study reporting on community norms around immunisation, household norms and decision making, awareness of place, time and schedule for vaccination, community health worker capacity, supply chain of community health workers and vaccine stockouts.

A summary of the results for all of the main analyses can be found in Table 8. A summary of all moderator analyses can be found in Supporting Information: Appendix I Table 3. Detailed results for the analyses are reported below.

Full routine childhood immunisation

For full routine childhood immunisations, we examined all outcomes where the study reported on the full sample (i.e., we did not include effects for any subgroup analyses). A random-effects model was fitted to the data, using only independent effects chosen as outlined in the methods section. A total of

k = 28 $k=28$

studies examined the relationship between community engagement interventions and full childhood immunisation. The estimated average outcome was

μ ˆ = 0.14 $\hat{\mu }=0.14$

([95% CI:

0.06 $0.06$

to

0.23 ] $0.23]$

,

z = 3.28 $z=3.28$

,

p = 0.001 ) $p=0.001)$

, indicating a small but significant benefit for the treated group of.14 standard deviation units (see Figure 9). The rank correlation test indicated funnel plot asymmetry (

p = 0.03 $p=0.03$

) but not the regression test (

p = 0.57 $p=0.57$

; Figure 10). The true outcomes appear to be heterogeneous (

Q ( 27 ) = 486.98 $Q(27)=486.98$

,

p < 0.0001 $p\lt 0.0001$

,

τ ˆ 2 = 0.06 ${\hat{\tau }}^2=0.06$

,

I 2 = 94.5 $I^2=94.5$

%). Outlier analyses revealed that one study (Banerjee, 2010) may be a potential outlier in the context of this model, and sensitivity analyses leaving each study out indicated that removing Banerjee (2010) would reduce the overall average effect (

μ $\mu$

 = 0.08 [95% CI: 0.04 to 0.12]), but the effect would still be positive and significant (

z $z$

 = 4.12,

p $p$

 < 0.001).

OPEN IN VIEWER

Figure 9

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on full childhood immunisation.

OPEN IN VIEWER

Figure 10

Funnel plot for studies examining the effect of community engagement interventions on full immunisation.

We conducted sensitivity analyses to examine the robustness of the results to the exclusion of low-quality studies. When studies assessed as high risk of bias were removed (leaving four medium/high quality studies), the resulting effect was slightly larger and still statistically significant (

μ ˆ = 0.18 $\hat{\mu }=0.18$

[95% CI:

0.08 $0.08$

to

0.27 ] ) $0.27])$

,

z = 3.67 $z=3.67$

,

p < 0.001 ) $p\lt 0.001)$

. We examined several potential sources of heterogeneity, including exposure to the intervention, evaluation period, study design, year, region, data source, whether the intervention was implemented by a government agency (either alone or in combination with another agency), whether new cadres of health workers were established, presence of vaccine hesitancy, and baseline vaccine coverage rates. There were no significant moderators in the context of this model (see Supporting Information: Appendix I Table 3).

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 53 effects from the same 28 studies (df = 25.5). The overall average effect was slightly smaller but still significant (

μ ˆ = 0.11 $\hat{\mu }=0.11$

[95% CI:

0.04 $0.04$

to

0.18 ] $0.18]$

, p = 0.002). Sensitivity analyses show the effect to be sensitive to all values of ρ. When using robust variance estimation, one can only make valid inferences from the results if there are at least 4 degrees of freedom for the moderator test. For this analysis, all moderators except for evaluation period and publication year met this criteria and thus were tested as potential sources of heterogeneity, but none were significant predictors of the variation among effects (see Supporting Information: Appendix I Table 3).

Next, we examined full immunisation outcomes in the context of the four types of community engagement interventions. All subgroup analyses were conducted using the RE models where all effects are independent. We did not do robustness checks using RVE for subgroup analyses.

Effects of engagement as the intervention on full childhood immunisation

A total of

k = 12 $k=12$

examined the effects of interventions with engagement as the intervention on full childhood immunisation. Studies that used engagement as the intervention had a significant effect on full childhood immunisation (

μ ˆ = 0.08 $\hat{\mu }=0.08$

[95% CI:

0.03 $0.03$

to

0.13 ] $0.13]$

). Again, the average outcome differed significantly from zero (

z = 3.02 $z=3.02$

,

p = 0.003 $p=0.003$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 11.

OPEN IN VIEWER

Figure 11

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement as the intervention on full immunisation.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 11 ) = 36.67 $Q(11)=36.67$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 70.00 $I^2=70.00$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.87 $\pm 2.87$

and hence there was no indication of outliers in the context of this model. Likewise, according to the Cook's distances, none of the studies could be considered to be overly influential.

With only one high or moderate quality study, sensitivity analysis by study quality could not be completed for this body of evidence. A funnel plot of the estimates is shown in Figure 12. Neither the rank correlation nor the regression test indicated any funnel plot asymmetry (

p = 0.20 $p=0.20$

and

p = 0.07 $p=0.07$

, respectively). Exposure to the intervention (in months) was the only significant source of heterogeneity such that for each additional month of intervention exposure, effects decreased by 0.003 standard deviation units (

B ˆ = − 0.003 , p = 0.049 $\hat{B}=-0.003,{p}=0.049$

[95% CI:

− 0.01 $-0.01$

to

− 0.00001 ] $-0.00001]$

).

OPEN IN VIEWER

Figure 12

Funnel plot for interventions with community engagement as the intervention on full immunisation.

Effects of engagement in the intervention design on full childhood immunisation

A total of

k = 5 $k=5$

studies were included in the analysis. The observed outcomes ranged from

− 0.06 $-0.06$

to

0.22 . $0.22.$

The estimated average outcome based on the random-effects model was

μ ˆ = 0.10 $\hat{\mu }=0.10$

[95% CI:

0.02 $0.02$

to

0.19 $0.19$

]. Therefore, the average outcome differed significantly from zero (

z = 2.40 $z=2.40$

,

p = 0.02 $p=0.02$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 13.

OPEN IN VIEWER

Figure 13

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design on full immunisation.

According to the

Q $Q$

-test, there was no significant amount of heterogeneity in the true outcomes (

Q ( 4 ) = 5.25 $Q(4)=5.25$

,

p = 0.26 $p=0.26$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 23.83 $I^2=23.83$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.58 $\pm 2.58$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential.

With no heterogeneity among effects, moderator analyses were not appropriate. When low quality studies were removed, two studies remained, and the summary effect increased (

μ ˆ = 0.14 $\hat{\mu }=0.14$

[95% CI:

− 0.01 $-0.01$

to

0.29 $0.29$

, but the effect was no longer significant

( z = 1.89 $(z=1.89$

,

p = 0.06 $p=0.06$

).

Effects of engagement in implementation autonomy on full childhood immunisation

Only

k $k$

 = 2 studies using engagement in implementation autonomy examined the effect on full childhood immunisation. The estimated average outcome based on the random-effects model was

μ ˆ = 0.23 $\hat{\mu }=0.23$

([95% CI:

− 0.001 $-0.001$

to

0.47 ] , z = 1.95 $0.47],z=1.95$

,

p = 0.051 $p=0.051$

), indicating no effect of these programmes on full immunisation (see Figure 14). According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 3.71 $Q(1)=3.71$

,

p = 0.054 $p=0.054$

,

τ ˆ 2 = 0.02 ${\hat{\tau }}^2=0.02$

,

I 2 = 73.07 $I^2=73.07$

%). There was no indication of outliers in the context of this model, and with only two studies, we could not test for moderation or publication bias.

OPEN IN VIEWER

Figure 14

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in implementation autonomy on full childhood immunisation.

Effects of interventions with multiple engagement types on full childhood immunisation

We included a total of

k = 10 $k=10$

studies in the analysis. The observed outcomes ranged from

− 0.12 $-0.12$

to

1.27 $1.27$

. The estimated average outcome based on the random-effects model was

μ ˆ = 0.22 $\hat{\mu }=0.22\,$

(95% CI:

− 0.12 $-0.12$

to

0.56 $0.56$

, and the average outcome did not differ significantly from zero (

z = 1.27 $z=1.27$

,

p = 0.20 $p=0.20$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 15.

OPEN IN VIEWER

Figure 15

Forest plot showing the observed outcomes and the estimate of the random-effects model for multiple engagement types on full childhood immunisation.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 8 ) = 388.56 $Q(8)=388.56$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.25 ${\hat{\tau }}^2=0.25$

,

I 2 = 97.94 $I^2=97.94$

%). There was no indication of outliers in the context of this model, and no high or moderate quality studies, so we were unable to perform a sensitivity analysis. None of the moderators were significant sources of heterogeneity in the context of this model, including whether the specific combination of engagement types led to different effects (see Supporting Information: Appendix I Table 3), but for full childhood immunisation we found no differences by engagement packages (

β ˆ = 0.36 , p = 0.30 $\hat{\beta }=0.36,p=0.30$

[95% CI:

− 0.31 $-0.31$

to

1.02 ] $1.02]$

).

Full childhood immunisation for girls

Wherever possible, we sought to examine subgroups in cases where results of primary studies were disaggregated by group (e.g., child sex, SES, etc.). For full immunisation, two studies reported disaggregated data on the effect of community engagement interventions on full immunisation for female children, thus we included k = 2 studies in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = − 0.02 $\hat{\mu }=-0.02$

(95% CI:

− 0.10 to 0.07 $-0.10\,{to}\,0.07$

). Therefore, the average outcome did not differ significantly from zero (

z = − 0.36 $z=-0.36$

,

p = 0.72 $p=0.72$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 16. Given the small number of studies, this result should be interpreted with caution. With only two studies, moderator analyses were not possible and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 16

Forest plot showing the observed outcomes and the estimate of the random-effects model for full childhood immunisation for girls.

Partial immunisation

We included a total of

k = 9 $k=9$

studies the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.23 $\hat{\mu }=0.23$

([95% CI:

0.09 $0.09$

to

0.37 ] $0.37]$

,

z = 3.15 $z=3.15$

,

p = 0.002 $p=0.002$

), indicating a benefit for the intervention group compared to the control group (see Figure 17).

OPEN IN VIEWER

Figure 17

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on partial childhood immunisation.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 8 ) = 219.11 $Q(8)=219.11$

,

p < 0.001 $p\lt 0.001$

,

τ ˆ 2 = 0.04 ${\hat{\tau }}^2=0.04$

,

I 2 = 96.351 $I^2=96.351$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.77 $\pm 2.77$

and hence there was no indication of outliers in the context of this model. Likewise, according to the Cook's distances, none of the studies could be considered to be overly influential. With eight out of the nine studies being assessed as high risk of bias, we were unable to conduct sensitivity analysis by study quality.

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 13 effects from the same 9 studies (df = 7.96). The overall average effect was slightly smaller but still significant (

μ ˆ = 0.21 $\hat{\mu }=0.21$

[95% CI:

0.03 $0.03$

to

0.38 ] $0.38]$

, p = 0.03). Sensitivity analyses show the effect to be sensitive to all values of ρ.

Effects of interventions with engagement as the intervention on partial childhood immunisation

Only two studies reporting on partial immunisation used interventions with community engagement as the intervention. The estimated average outcome based on the random-effects model was

μ ˆ = 0.31 $\hat{\mu }=0.31$

([95% CI:

− 0.25 $-0.25$

to

0.87 ] , z = 1.10 $0.87],{z}=1.10$

,

p = 0.27 $p=0.27$

), indicating no significant difference between the intervention group and the control group on partial immunisation (see Figure 18). Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 1 ) = 95.93 $Q(1)=95.93$

,

p < 0.001 $p\lt 0.001$

,

τ ˆ 2 = 0.16 ${\hat{\tau }}^2=0.16$

,

I 2 = 98.96 $I^2=98.96$

%). With only two studies, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 18

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement as the intervention on partial immunisation.

Effects of interventions with engagement in the intervention design on partial childhood immunisation

Only two studies reporting on partial immunisation used interventions with community engagement in the intervention design. The estimated average outcome based on the random-effects model was

μ ˆ = 0.14 $\hat{\mu }=0.14$

(

[ 95 % CI : 0.01 to 0.27 ] , z = 2.05 $[95 \% \text{CI}:0.01\,\text{to}\,0.27],{z}=2.05$

,

p = 0.04 $p=0.04$

), indicating a small but significant benefit to the intervention group compared to the control group (see Figure 19). Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.91 $Q(1)=0.91$

,

p = 0.34 $p=0.34$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies and no heterogeneity, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 19

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design of the intervention on partial childhood immunisation.

Effects of interventions with engagement in implementation autonomy on partial childhood immunisation

Only one study reporting on partial immunisation utilised an intervention which used engagement in the implementation (Webster, 2019), so a quantitative synthesis was not possible. This cluster RCT implemented in Uganda found a null effect of their programme on partial childhood immunisation (g = 0.02 [95% CI: −0.08 to 0.13]), but like most studies, it was assessed as having a high risk of bias.

Effects of interventions with multiple engagement types on partial childhood immunisation

We included a total of

k = 4 $k=4$

studies in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.28 $\hat{\mu }=0.28$

(95% CI:

− 0.01 $-0.01$

to

0.56 $0.56$

). Therefore, the average outcome did not differ significantly from zero (

z = 1.91 $z=1.91$

,

p = 0.06 $p=0.06$

), indicating no difference between the intervention and control groups (see Figure 20). According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 3 ) = 101.75 $Q(3)=101.75$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.08 ${\hat{\tau }}^2=0.08$

,

I 2 = 97.05 $I^2=97.05$

%).

OPEN IN VIEWER

Figure 20

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with multiple engagement types on partial immunisation.

An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.50 $\pm 2.50$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. Of the moderators we were able to test, none were significant sources of heterogeneity (see Supporting Information: Appendix I Table 3). We were unable to test for differences among engagement packages because only one study used a combination of engagement in the design and engagement as the intervention (Gurley et al., 2020) while the remaining studies used a combination of engagement in implementation autonomy and engagement as the intervention.

Measles vaccination

A total of

k = 20 $k=20$

studies examined the relationship between community engagement interventions and measles vaccination. The estimated average outcome was

μ ˆ = 0.07 $\hat{\mu }=0.07$

([95% CI:

0.03 $0.03$

to

0.11 ] $0.11]$

,

z = 3.21 ${z}=3.21$

,

p < 0.01 $p\lt 0.01$

) indicating a very small but significant benefit for the treated group compared to the untreated group (see Figure 21). The true outcomes appear to be heterogeneous (

Q ( 19 ) = 72.07 $Q(19)=72.07$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.01 ${\hat{\tau }}^2=0.01$

,

I 2 = 73.64 $I^2=73.64$

%). An examination of the studentized residuals revealed that one study (Sankar, 2013) had a value larger than

± 3.02 $\pm 3.02$

and may be a potential outlier in the context of this model. Indeed, sensitivity analysis leaving out Sankar (2013) would result in an increase in the average effect (

μ ˆ = 0.08 $\hat{\mu }=0.08$

[95% CI:

0.05 $0.05$

to

0.12 ] $0.12]$

), and it was still statistically significant (

z = 4.44 $z=4.44$

,

p < 0.001 $p\lt 0.001$

). According to the Cook's distances, none of the studies could be considered to be overly influential. A funnel plot of the estimates is shown in Figure 22. Both the rank correlation and the regression test indicated potential funnel plot asymmetry (

p = 0.05 $p=0.05$

and

p = 0.02 $p=0.02$

, respectively).

OPEN IN VIEWER

Figure 21

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on measles vaccination.

OPEN IN VIEWER

Figure 22

Funnel plot for studies examining the effect of community engagement interventions on measles vaccination.

When low quality studies are removed, the average effect increases (

μ ˆ = 0.09 $\hat{\mu }=0.09$

,

k $k$

 = 6, [95% CI:

0.03 $0.03$

to

0.15 ] ) $0.15])$

and is still statistically significant

( z = 2.98 $(z=2.98$

,

p = 0.003 ; $p=0.003{\rm{;}}\unicode{x02007}$

Figure 23).

OPEN IN VIEWER

Figure 23

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on measles vaccination when low quality studies are removed.

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 32 effects from the same 20 studies (df = 15.5). The overall average effect was slightly smaller but still significant (

μ ˆ = 0.06 $\hat{\mu }=0.06$

[95% CI:

0.01 $0.01$

to

0.11 ] $0.11]$

, p = 0.03). Sensitivity analyses show the effect to be sensitive to all values of ρ.

Effects of engagement as the intervention on measles vaccination

We included a total of

k = 10 $k=10$

studies in the analysis. The observed outcomes ranged from

0.03 $0.03$

to

0.30 $0.30$

. The estimated average outcome based on the random-effects model was

μ ˆ = 0.10 $\hat{\mu }=0.10$

(95% CI:

0.05 $0.05$

to

0.15 $0.15$

). Therefore, the average outcome differed significantly from zero (

z = 3.89 $z=3.89$

,

p < 0.001 $p\lt 0.001$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 24.

OPEN IN VIEWER

Figure 24

Forest plot showing the observed outcomes and the estimate of the random-effects model of community engagement as the intervention on measles.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 9 ) = 22.67 $Q(9)=22.67$

,

p = 0.007 $p=0.007$

,

τ ˆ 2 = 0.003 ${\hat{\tau }}^2=0.003$

,

I 2 = 60.29 $I^2=60.29$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.81 $\pm 2.81$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential.

A funnel plot of the estimates is shown in Figure 25. The regression test indicated funnel plot asymmetry (

p < 0.01 $p\lt 0.01$

) but not the rank correlation test (

p = 0.38 $p=0.38$

). None of the moderators we tested were significant sources of heterogeneity (see Table 3, Supporting Information: Appendix I).

OPEN IN VIEWER

Figure 25

Funnel plot showing studies of community engagement as the intervention on measles.

Effects of engagement in the intervention design on measles vaccination

Only two studies on measles vaccination used interventions with engagement in the design. The estimated average outcome based on the random-effects model was

μ ˆ = 0.11 $\hat{\mu }=0.11$

(95% CI:

0.02 $0.02$

to

0.21 $0.21$

), indicating a small but significant benefit to the intervention participants compared to the control group (

z = 2.36 $z=2.36$

,

p = 0.02 $p=0.02$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 26. Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, there was no significant amount of heterogeneity in the true outcomes (

Q ( 1 ) = 0.56 $Q(1)=0.56$

,

p = 0.46 $p=0.46$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies and no heterogeneity among effects, moderator analyses and tests of publication bias were not appropriate.

OPEN IN VIEWER

Figure 26

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design on measles vaccination.

Effects intervention with engagement in implementation autonomy on measles vaccination

Only two studies on measles vaccination used interventions with engagement in implementation autonomy. The estimated average outcome based on the random-effects model was

μ ˆ = 0.03 $\hat{\mu }=0.03$

(95% CI:

− 0.09 $-0.09$

to

0.15 $0.15$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.47 $z=0.47$

,

p = 0.64 $p=0.64$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 27. Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 2.21 $Q(1)=2.21$

,

p = 0.14 $p=0.14$

,

τ ˆ 2 = 0.004 ${\hat{\tau }}^2=0.004$

,

I 2 = 54.84 $I^2=54.84$

%; see Figure 27). With only two studies we were unable to test for publication bias or sources of heterogeneity.

OPEN IN VIEWER

Figure 27

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in implementation autonomy on measles vaccination.

Effects of interventions with multiple engagement types on measles vaccination

A total of

k = 6 $k=6$

studies were included in the analysis. The observed outcomes ranged from

− 0.27 $-0.27$

to

0.58 $0.58$

. The estimated average outcome based on the random-effects model was

μ ˆ = 0.03 $\hat{\mu }=0.03$

(95% CI:

− 0.10 $-0.10$

to

0.16 $0.16$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.44 $z=0.44$

,

p = 0.20 $p=0.20$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 28.

OPEN IN VIEWER

Figure 28

Forest plot showing the observed outcomes and the estimate of the random-effects model.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 5 ) = 37.76 $Q(5)=37.76$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.02 ${\hat{\tau }}^2=0.02$

,

I 2 = 86.76 $I^2=86.76$

%; see Figure 28). An examination of the studentized residuals revealed that one study (Sankar, 2013) had a value larger than

± 2.64 $\pm 2.64$

and may be a potential outlier in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. Indeed, sensitivity analyses leaving each study out indicated that removing Sankar (2013) would increase the overall average effect (

μ ˆ = 0.09 $\hat{\mu }=0.09$

[95% CI: −0.02 to 0.19]), but the effect would still be nonsignificant (

z = 1.58 $z=1.58$

,

p = 0.11 $p=0.11$

). With only one high or medium quality study, we were unable to conduct sensitivity analysis by study quality. None of the moderators that could be tested were significant in the context of this model (see Supporting Information: Appendix I Table 3). For measles vaccinations, we were unable to test whether different combinations of community engagement produced different results because all but one study used a combination of engagement in the implementation and engagement as the intervention.

BCG vaccination

We included a total of

k = 12 $k=12$

studies in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.06 $\hat{\mu }=0.06$

([95% CI:

0.01 $0.01$

to

0.11 $0.11$

],

z = 2.28 ${z}=2.28$

,

p = 0.02 $p=0.02$

), indicating a very small but significant benefit to the intervention participants compared to the control participants (see Figure 29). According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 11 ) = 84.22 $Q(11)=84.22$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 86.94 $I^2=86.94$

%). An examination of the studentized residuals revealed that one study (Banerjee, 2010) had a value larger than

± 2.87 $\pm 2.87$

and may be a potential outlier in the context of this model. According to the Cook's distances, Banerjee (2010) could also be considered to be overly influential. Sensitivity analyses leaving each study out indicated that removing Banerjee (2010) would reduce the overall average effect (

μ ˆ = 0.02 $\hat{\mu }=0.02$

(95% CI:

− 0.005 $-0.005$

to

0.04 $0.04$

), and the resulting effect would be nonsignificant (

z = 1.51 $z=1.51$

,

p = 0.13 $p=0.13$

). Moderator analysis revealed that publication year was a significant predictor such that more recent studies find smaller effects that older studies, with each additional year reducing the size of the effect by 0.03 standard deviation units (

β = − 0.03 $\beta =-0.03$

[95% CI:

− 0.05 $-0.05$

to

− 0.001 $-0.001$

],

p $p$

 = 0.04). No other moderators were significant (see Supporting Information: Appendix I Table 3). Only three studies were high or moderate quality. When those three studies were synthesised, the resulting effect was reduced

( μ ˆ = 0.01 $(\hat{\mu }=0.01$

[95% CI:

− 0.08 $-0.08$

to

0.10 ] $0.10]$

) and no longer significant (

z = 0.22 $z=0.22$

,

p = 0.82 $p=0.82$

).

OPEN IN VIEWER

Figure 29

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on BCG vaccination.

A funnel plot of the estimates is shown in Figure 30. The regression test indicated funnel plot asymmetry (

p = 0.04 $p=0.04$

) but not the rank correlation test (

p = 0.20 $p=0.20$

).

OPEN IN VIEWER

Figure 30

Funnel plot for the impact of community engagement interventions on BCG vaccination.

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 16 effects from the same 12 studies (df = 7.4). The overall average effect was slightly smaller and nonsignificant (

μ ˆ = 0.04 $\hat{\mu }=0.04$

[95% CI:

− 0.02 $-0.02$

to

0.10 ] $0.10]$

, p = .20). Sensitivity analyses show the effect to be sensitive to all values of ρ.

Effects of engagement as the intervention on BCG vaccination

We included a total of

k = 4 $k=4$

studies the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.02 $\hat{\mu }=0.02$

(95% CI:

0.01 $0.01$

to

0.03 $0.03$

). Therefore, the average outcome differed significantly from zero (

z = 3.32 $z=3.32$

,

p < 0.01 $p\lt 0.01$

), indicating a very small but significant benefit to the treated group compared to the control group (see Figure 31). According to the

Q $Q$

-test, there was no significant amount of heterogeneity in the true outcomes (

Q ( 3 ) = 2.98 $Q(3)=2.98$

,

p = 0.39 $p=0.39$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%), thus we did not examine potential sources of heterogeneity for this model. With three of the four studies assessed as high risk of bias, we were unable to conduct a sensitivity analysis by study quality. An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.50 $\pm 2.50$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential.

OPEN IN VIEWER

Figure 31

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement as the intervention on BCG vaccination.

Effects of engagement in the intervention design on BCG vaccination

Only two studies reporting on BCG vaccination used interventions with community engagement in the intervention design. The estimated average outcome based on the random-effects model was

μ ˆ = 0.02 $\hat{\mu }=0.02$

([95% CI:

− 0.09 $-0.09$

to

0.13 ] , z = 0.32 $0.13],{z}=0.32$

,

p = 0.75 $p=0.75$

), indicating no difference between the intervention group and the control group (see Figure 32). Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.11 $Q(1)=0.11$

,

p = 0.75 $p=0.75$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies and no heterogeneity, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 32

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design of the intervention on BCG vaccination

Effects of intervention with engagement in implementation autonomy on BCG vaccination

Only two studies reporting on BCG vaccination used interventions with community engagement in implementation autonomy. The estimated average outcome was

μ ˆ = 0.03 $\hat{\mu }=0.03$

([95% CI:

− 0.05 $-0.05$

to

0.11 ] $0.11]$

,

z = 0.75 $z=0.75$

,

p = 0.46 $p=0.46$

) indicating no difference between the treated group and the untreated group (Figure 33). According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.37 $Q(1)=0.37$

,

p = 0.54 $p=0.54$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 00.00 $I^2=00.00$

%). With only two studies and no heterogeneity, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 33

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in implementation autonomy on BCG vaccination. BCG, Bacillus Calmette-Guerin.

Effects of interventions with multiple engagement types on BCG vaccination

A total of

k = 4 $k=4$

studies were included in the analysis. The observed outcomes ranged from

− 0.08 $-0.08$

to

0.56 . $0.56.$

The estimated average outcome based on the random-effects model was

μ ˆ = 0.22 $\hat{\mu }=0.22$

(95% CI:

− 0.07 $-0.07$

to

0.52 $0.52$

). Therefore, the average outcome did not differ significantly from zero (

z = 1.48 $z=1.48$

,

p = 0.14 $p=0.14$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 34.

OPEN IN VIEWER

Figure 34

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with multiple engagement types on BCG vaccination. BCG, Bacillus Calmette-Guerin.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 3 ) = 75.36 $Q(3)=75.36$

,

p < 0.001 $p\lt 0.001$

,

τ ˆ 2 = 0.08 ${\hat{\tau }}^2=0.08$

,

I 2 = 96.01 $I^2=96.01$

%). An examination of the studentized residuals revealed that one study (Banerjee, 2010) had a value larger than

± 2.50 $\pm 2.50$

and may be a potential outlier in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. Indeed, sensitivity analyses leaving each study out indicated that removing Banerjee (2010) would reduce the overall average effect (

μ ˆ = 0.05 $\hat{\mu }=0.05$

(95% CI:

− 0.13 $-0.13$

to

0.23 $0.23$

), but the effect is still positive and nonsignificant (

z = 0.56 $z=0.56$

,

p = 0.58 $p=0.58$

).

We tested all moderators and only post-intervention versus change from baseline was a significant predictor of BCG vaccination, such that studies examining change from baseline had higher effects than studies examining post-intervention changes by 0.55 standard deviation units (

μ ˆ = − 0.55 $\hat{\mu }=-0.55$

(95% CI:

− 0.69 $-0.69$

to

− 0.40 , p < 0.001 $-0.40,{p}\lt 0.001$

).

DPT 1 vaccination

We included a total of

k = 8 $k=8$

studies in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.04 $\hat{\mu }=0.04$

([95% CI:

− 0.04 $-0.04$

to

0.11 ] , z = 0.99 $0.11],{z}=0.99$

,

p = 0.32 $p=0.32$

), indicating no difference between the intervention group and the control group (see Figure 35). According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 7 ) = 30.18 $Q(7)=30.18$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.01 ${\hat{\tau }}^2=0.01$

,

I 2 = 76.81 $I^2=76.81$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.73 $\pm 2.73$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. When we removed studies assessed as high risk of bias, two studies remained, and the average effect increased slightly (

μ ˆ = 0.04 $\hat{\mu }=0.04$

([95% CI:

− 0.04 $-0.04$

to

0.11 ] ) , $0.11]),$

but was still nonsignificant (

z = 0.99 $z=0.99$

,

p = 0.32 $p=0.32$

). Again, with only two studies contributing effects, this must be interpreted with caution.

OPEN IN VIEWER

Figure 35

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on DPT1 vaccination. DPT, diphtheria, pertussis tetanus.

We tested for potential sources of heterogeneity and found that study design was a significant predictor of the effect such that studies using quasi-experimental designs had smaller effects than RCTs by 0.15 standard deviation units (

β = − 0.15 $\beta =-0.15$

[95% CI:

− 0.29 $-0.29$

to

− 0.02 $-0.02$

],

p = 0 $p=0$

 0.03). We also found that there was a significant difference in the size of effects between programmes implemented by government agencies and those that were not, such that programmes implemented by government agencies (either alone or in tandem with another agency) had larger effects than programmes not implemented by a government agency by 0.17 standard deviation units (

β = 0.17 $\beta =0.17$

[95% CI:

0.01 $0.01$

to

0.34 $0.34$

],

p $p$

 = 0.04). No other moderators were significant (see Supporting Information: Appendix I Table 3).

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 21 effects from the same 8 studies (df = 5.24). The overall average effect was smaller and still nonsignificant (

μ ˆ = 0.01 $\hat{\mu }=0.01$

[95% CI:

− 0.06 $-0.06$

to

0.09 ] $0.09]$

, p = 0.66). Sensitivity analyses show the effect to be sensitive to all values of ρ.

Effects of engagement as the intervention on DPT 1 vaccination

Only

k $k$

 = 3 studies on DPT1 vaccination used interventions with engagement as the intervention. The estimated average outcome based on the random-effects model was

μ ˆ = 0.10 $\hat{\mu }=0.10$

([95% CI:

− 0.03 $-0.03$

to

0.22 ] , z = 1.53 $0.22],{z}=1.53$

,

p = 0.13 $p=0.13$

), indicating no difference between the treatment and control groups (see Figure 36). According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 2 ) = 20.42 $Q(2)=20.42$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.01 ${\hat{\tau }}^2=0.01$

,

I 2 = 90.21 $I^2=90.21$

%). An examination of the studentized residuals revealed that one study (Costa-Font & Parmar, 2017) had a value larger than

± 2.39 $\pm 2.39$

and may be a potential outlier in the context of this model. Indeed, sensitivity analyses leaving each study out indicated that removing Costa-Font and Parmar (2017) would increase the overall average effect (

μ ˆ $\hat{\mu }$

= 0.16 (95% CI: 0.09 to 0.22), and the resulting effect would be positive and significant (

z $z$

= 4.72,

p $p$

< 0.001). According to the Cook's distances, none of the studies could be considered to be overly influential. With only three studies, we were unable to conduct moderator analyses or test for publication bias. Only one study (Banerjee et al., 2020) was not assessed as high risk of bias.

OPEN IN VIEWER

Figure 36

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement as the intervention on DPT1 vaccination. DPT, diphtheria, pertussis tetanus.

Effects of engagement in the intervention design on DPT 1 vaccination

Only two studies on DPT1 vaccination used interventions with engagement in the design. The estimated average outcome based on the random-effects model was

μ ˆ = 0.03 $\hat{\mu }=0.03$

(95% CI:

− 0.08 $-0.08$

to

0.14 $0.14$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.60 $z=0.60$

,

p = 0.55 $p=0.55$

, see Figure 37), indicating no difference between the treatment group and the control group on DPT1 vaccination. Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.48 $Q(1)=0.48$

,

p = 0.49 $p=0.49$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies, moderator analyses were not possible and test of publication bias are not valid.

OPEN IN VIEWER

Figure 37

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design on the intervention on DPT1 vaccination. DPT, diphtheria, pertussis tetanus.

Effects of intervention with engagement in implementation autonomy on DPT 1 vaccination

Only one study reporting on DPT1 vaccination used interventions with engagement in the implementation (Webster, 2019), thus we were unable to perform a statistical synthesis. This cluster RCT from Uganda found a null effect of their programme on DPT1 vaccination (g = 0.04 [95% CI: −0.07 to 0.14]), but like most studies, it was assessed as having a high risk of bias.

Effects of interventions with multiple engagement types on DPT 1 vaccination

Only k = 2 studies reporting on DPT1 vaccination used interventions with multiple engagement types. The estimated average outcome based on the random-effects model was

μ ˆ = − 0.17 $\hat{\mu }=-0.17$

([95% CI:

− 0.29 $-0.29$

to

− 0.05 ] , z = − 2.81 $-0.05],{z}=-2.81$

,

p = 0.005 $p=0.005$

), thus there was a significant negative impact of the programmes on DPT1 vaccination (see Figure 38). Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.03 $Q(1)=0.03$

,

p = 0.87 ${p}=0.87$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 38

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with multiple engagement types on DPT1 vaccination. DPT, diphtheria, pertussis tetanus.

DPT 2 vaccination

We included a total of

k = 5 $k=5$

studies in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.07 $\hat{\mu }=0.07$

(95% CI:

0.01 $0.01$

to

0.12 $0.12$

). Therefore, the average outcome differed significantly from zero (

z = 2.23 $z=2.23$

,

p = 0.03 $p=0.03$

), indicating very small but significant benefit to the treated group compared to the control group (see Figure 39). According to the

Q $Q$

-test, there was no significant amount of heterogeneity in the true outcomes (

Q ( 4 ) = 3.34 $Q(4)=3.34$

,

p = 0.50 $p=0.50$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.58 $\pm 2.58$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. With no heterogeneity present, we did not examine potential sources of variation. When high risk of bias studies were removed, the resulting average effect increased slightly (

μ ˆ = 0.11 $\hat{\mu }=0.11$

[95% CI:

0.01 $0.01$

to

0.20 $0.20$

]) and was still significantly different from zero (

z = 2.12 $z=2.12$

,

p = 0.03 $p=0.03$

).

OPEN IN VIEWER

Figure 39

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on DPT2 vaccination. DPT, diphtheria, pertussis tetanus. This group of studies was not sufficiently powered for a robustness check using robust variance analysis (df = 3.51).

Effects of engagement as the intervention on DPT 2 vaccination

Only one study (Banerjee et al., 2020) fell into this intervention/outcome category, thus we were unable to perform a statistical synthesis. This cluster RCT from India found a small but significant positive effect of their programme on DPT2 vaccination (g = 0.15 [95% CI: 0.01 to 0.29]), but like most studies, it was assessed as having a high risk of bias.

Effects of engagement in the intervention design on DPT 2 vaccination

Only two studies on DPT2 vaccination used interventions with engagement in the design. The estimated average outcome based on the random-effects model was

μ ˆ = 0.05 $\hat{\mu }=0.05$

(95% CI:

− 0.06 $-0.06$

to

0.16 $0.16$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.83 $z=0.83$

,

p = 0.41 $p=0.41$

, see Figure 40), indicating no difference between the treatment group and the control group on DPT2 vaccination. Given the small number of studies, this result should be interpreted with caution. According to the

Q $Q$

-test, the true outcomes appear to be homogeneous (

Q ( 1 ) = 0.07 $Q(1)=0.07$

,

p = 0.41 $p=0.41$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). With only two studies and no heterogeneity, moderator analyses were not appropriate and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 40

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design on the intervention on DPT2 vaccination. DPT, diphtheria, pertussis tetanus.

Effects of intervention with engagement in implementation autonomy on DPT 2 vaccination

Only one study (Webster et al., 2019) fell into this intervention/outcome category, thus we were unable to perform a statistical synthesis. This cluster RCT from Uganda found a null effect of their programme on DPT1 vaccination (g = 0.01 [95% CI: −0.10 to 0.11]), but like most studies, it was assessed as having a high risk of bias.

Effects of interventions with multiple engagement types on DPT 2 vaccination

Only one study (Sankar, 2013) fell into this intervention/outcome category, thus we were unable to perform a statistical synthesis. This quasi-experimental study from India found a null effect of their programme on OPV0 vaccination (g = 0.10 [95% CI: −0.02 to 0.22]), but like most studies, it was assessed as having a high risk of bias. Their programme included a combination of engagement in the implementation and engagement as the intervention.

DPT 3 vaccination

A total of

k = 22 $k=22$

studies examined the relationship between community engagement interventions and DPT3 vaccinations. The estimated average outcome was

μ ˆ = 0.10 $\hat{\mu }=0.10$

([95% CI:

0.06 $0.06$

to

0.14 ] $0.14]$

,

z = 4.75 $z=4.75$

,

p < 0.001 $p\lt 0.001$

) indicating a small but significant benefit to the treated group compared to the untreated group (Figure 41). The rank correlation test indicated funnel plot asymmetry (

p = 0.04 $p=0.04$

) but not the regression test (

p = 0.06 $p=0.06$

), but trim and fill analyses indicated an identical effect size (see Figures 42 and 43). When low quality studies were removed, the average effect increased slightly (

μ ˆ = 0.11 $\hat{\mu }=0.11$

[95% CI:

0.05 $0.05$

to

0.17 ] ) $0.17])$

, and was still statistically significant (

z = 3.70 $z=3.70$

,

p < 0.001 ; $p\lt 0.001;$

see Figure 44). Only four studies were of high or moderate quality, so this analysis is based on a small sample of studies (Figure 44).

OPEN IN VIEWER

Figure 41

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

OPEN IN VIEWER

Figure 42

Funnel plot for studies examining the studies effect of community engagement interventions on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

OPEN IN VIEWER

Figure 43

Trim and fill plot for examining the effect of community engagement interventions on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

OPEN IN VIEWER

Figure 44

Forest plot showing the observed outcomes and the estimate of the random-effects model for the impact of community engagement interventions on DPT3 vaccination when low quality studies are removed. DPT, diphtheria, pertussis tetanus.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 21 ) = 90.43 $Q(21)=90.43$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.01 ${\hat{\tau }}^2=0.01$

,

I 2 = 76.78 $I^2=76.78$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 3.05 $\pm 3.05$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential.

Publication year was a significant moderator such that each additional year reduced the size of the effect by 0.014 standard deviation units (

B ˆ = − 0.014 , p = 0.019 $\hat{B}=-0.014,{p}=0.019$

[95% CI:

− 0.03 $-0.03$

to

− 0.002 ] $-0.002]$

). In other words, new studies have found smaller effects. There were no other significant moderators in the context of this model (see Supporting Information: Appendix I Table 3).

As a robustness check, we used robust variance analysis and included all dependent effects in the analysis, totalling 36 effects from the same 22 studies (df = 18.2). The overall average effect was slightly smaller but still significant (

μ ˆ = 0.10 $\hat{\mu }=0.10$

[95% CI:

0.05 $0.05$

to

0.15 ] $0.15]$

, p < .001). Sensitivity analyses show the effect to be sensitive to all values of ρ.

Effects of engagement as the intervention on DPT 3 vaccination

We included a total of

k = 6 $k=6$

studies in the analysis. The observed outcomes ranged from

0.00 $0.00$

to

0.23 $0.23$

. The estimated average outcome based on the random-effects model was

μ ˆ = 0.09 $\hat{\mu }=0.09$

(95% CI:

0.03 $0.03$

to

0.15 $0.15$

). Therefore, the average outcome differed significantly from zero (

z = 3.01 $z=3.01$

,

p < 0.01 $p\lt 0.01$

), indicating a benefit to the intervention participants compared to the control group. A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 45.

OPEN IN VIEWER

Figure 45

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement as the intervention on DPT3. DPT, diphtheria, pertussis tetanus.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 5 ) = 18.64 $Q(5)=18.64$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 73.17 $I^2=73.17$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.64 $\pm 2.64$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. None of the moderators we tested were significant sources of heterogeneity (see Supporting Information: Appendix I Table 3).

Effects of engagement in the intervention design on DPT 3 vaccination

A total of

k = 6 $k=6$

studies were included in the analysis. The observed outcomes ranged from

− 0.14 $-0.14$

to

0.10 $0.10$

. The estimated average outcome based on the random-effects model was

0.04 $0.04$

[95% CI:

− 0.01 $-0.01$

to

0.08 ] $0.08]$

. Therefore, the average outcome did not differ significantly from zero (

z = 1.69 , p = 0.09 $z=1.69,p=0.09$

), indicating no benefit to intervention participants compared to control participants. A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 46.

OPEN IN VIEWER

Figure 46

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in the design on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

According to the

Q $Q$

-test, there was no significant amount of heterogeneity in the true outcomes (

Q ( 5 ) = 4.36 $Q(5)=4.36$

,

p = 0.50 $p=0.50$

,

τ ˆ 2 = 0.00 ${\hat{\tau }}^2=0.00$

,

I 2 = 0.00 $I^2=0.00$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.64 $\pm 2.64$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. With no heterogeneity, we did not perform moderator analyses. With only one study of low or medium quality, we were also unable to complete sensitivity analyses for this body of evidence.

Effects of engagement in implementation autonomy on DPT 3 vaccination

Only three studies on DPT3 vaccination used interventions with engagement in implementation. The outcomes ranged from −0.03 to 0.20. The estimated average outcome based on the random-effects model was

μ ˆ = 0.11 $\hat{\mu }=0.11$

(95% CI:

− 005 $-005$

to

0.28 $0.28$

). Therefore, the average outcome did not differ significantly from zero (

z = 1.38 $z=1.38$

,

p = 0.17 $p=0.17$

) indicating no benefit to the intervention participants compared to the control participants. A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 47. Given the small number of studies, this result should be interpreted with caution.

OPEN IN VIEWER

Figure 47

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with community engagement in implementation autonomy on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 2 ) = 10.06 $Q(2)=10.06$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.02 ${\hat{\tau }}^2=0.02$

,

I 2 = 80.12 $I^2=80.12$

%). An examination of the studentized residuals revealed that one study (Webster, 2019) had a value larger than

± 2.39 $\pm 2.39$

and may be a potential outlier in the context of this model. Indeed, sensitivity analyses leaving each study out indicated that removing Webster (2019) would increase the overall average effect (

μ ˆ $\hat{\mu }$

= 0.20 [95% CI: 0.10 to 0.29]), with the effect still positive and significant (z = 3.99, p < .001). According to the Cook's distances, none of the studies could be considered to be overly influential.

With only three studies, moderator analyses and tests of publication bias were not appropriate. With two of the three studies assessed as high risk of bias, we were also unable to conduct a sensitivity analysis by study quality.

Effects of interventions with multiple engagement types on DPT 3 vaccination

A total of

k = 8 $k=8$

studies were included in the analysis. The observed outcomes ranged from

− 0.01 $-0.01$

to

0.62 $0.62$

. The estimated average outcome based on the random-effects model was

μ ˆ = 0.20 $\hat{\mu }=0.20$

[95% CI:

0.06 $0.06$

to

0.34 ] $0.34]$

). Therefore, the average outcome differed significantly from zero (

z = 2.85 $z=2.85$

,

p < 0.01 $p\lt 0.01$

), indicating a benefit to the intervention participants compared to the control group. A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 48.

OPEN IN VIEWER

Figure 48

Forest plot showing the observed outcomes and the estimate of the random-effects model for interventions with multiple engagement types on DPT3 vaccination. DPT, diphtheria, pertussis tetanus.

According to the

Q $Q$

-test, the true outcomes appear to be heterogeneous (

Q ( 6 ) = 49.71 $Q(6)=49.71$

,

p < 0.01 $p\lt 0.01$

,

τ ˆ 2 = 0.03 ${\hat{\tau }}^2=0.03$

,

I 2 = 87.93 $I^2=87.93$

%). An examination of the studentized residuals revealed that none of the studies had a value larger than

± 2.69 $\pm 2.69$

and hence there was no indication of outliers in the context of this model. According to the Cook's distances, none of the studies could be considered to be overly influential. With no high or moderate quality studies, we were unable to perform a sensitivity analysis. There were several moderators that were significant sources of heterogeneity. Evaluation period was significant such that each additional month between the end of the intervention and the collection of outcome data reduced the size of the effect by 0.01 standard deviation units (

β ˆ = − 0.01 , p = 0.003 $\hat{\beta }=-0.01,{p}=0.003$

[95% CI:

− 0.02 $-0.02$

to

− 0.004 ] $-0.004]$

), suggesting smaller long term effect of the interventions. Publication year was also significant, such that each additional year reduced the size of the effect by 0.03 standard deviation units (

β ˆ = − 0.03 , p = 0.03 $\hat{\beta }=-0.03,{p}=0.03$

[95% CI:

− 0.05 $-0.05$

to

− 0.003 ] $-0.003]$

). In other words, more recent studies have found smaller effects. Finally, baseline DPT3 coverage rates were significant such that a one unit increase in baseline DPT3 coverage was associated with a decrease of.56 in the effect of the programme (

β ˆ = − 0.56 , p = 0.002 $\hat{\beta }=-0.56,{p}=0.002$

[95% CI:

− 0.90 $-0.90$

to

− 0.21 ] $-0.21]$

). In other words, the programmes were significantly more effective in areas with lower baseline coverage rates. We also tested whether the specific combination of engagement types led to different effects, but for DPT3 vaccination we found no differences by engagement packages (

β ˆ = − 0.07 , p = 0.66 $\hat{\beta }=-0.07,{p}=0.66$

[95% CI:

− 0.40 $-0.40$

to

− 0.25 ] $-0.25]$

).

DPT3 vaccinations for girls

Wherever possible, we sought to examine subgroups in cases where results of primary studies were disaggregated by group (e.g., child sex, SES, etc.). For DPT3 vaccinations, two studies reported disaggregated data for full immunisation for female children, thus k = 2 studies were included in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.08 $\hat{\mu }=0.08$

(95% CI:

− 0.09 to 0.25 $-0.09\,\mathrm{to}\,0.25$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.96 $z=0.96$

,

p = 0.33 $p=0.33$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 49. Given the small number of studies, this result should be interpreted with caution. With only two studies, moderator analyses were not possible and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 49

Forest plot showing the observed outcomes and the estimate of the random-effects model for effects of community engagement on DPT3 immunisation for girls. DPT, diphtheria, pertussis tetanus.

DPT3 vaccinations for boys

Wherever possible, we sought to examine subgroups in cases where results of primary studies were disaggregated by group (e.g., child sex, SES, etc.). For full immunisation, two studies reported disaggregated data for full immunisation for female children, thus k = 2 studies were included in the analysis. The estimated average outcome based on the random-effects model was

μ ˆ = 0.08 $\hat{\mu }=0.08$

(95% CI:

− 0.09 to 0.25 $-0.09\,\mathrm{to}\,0.25$

). Therefore, the average outcome did not differ significantly from zero (

z = 0.97 $z=0.97$

,

p = 0.33 $p=0.33$

). A forest plot showing the observed outcomes and the estimate based on the random-effects model is shown in Figure 50. Given the small number of studies, this result should be interpreted with caution. With only two studies, moderator analyses were not possible and tests of publication bias are not valid.

OPEN IN VIEWER

Figure 50

Forest plot showing the observed outcomes and the estimate of the random-effects model for effects of community engagement on DPT3 immunisation for boys. DPT, diphtheria, pertussis tetanus.