PROTOCOL: Effectiveness of social accountability interventions in low- and middle-income countries: An evidence and gap map

The problem, condition or issue

A study conducted by the Institute of Development Studies (IDS) in 2010 to research the “Impact of Social Accountability” found that almost no “meta literature” on accountability impact and effectiveness was present, and the existing literature was variable and scattered (McGee & Gaventa, 2011). Moreover, the existing literature was largely theoretical with under-specified assumptions with very few studies of the impact and effectiveness in real terms (Boydell & Keesbury, 2014). Social accountability (SA) is an advancing umbrella category that includes citizen monitoring and oversight of public and/or private sector performance, user-centered public information access/dissemination systems, public complaint and grievance redress mechanisms, as well as citizen participation in actual resource allocation decision-making (Fox, 2015).

Joshi (2017) explained SA “as a subset of accountability more broadly comprises … citizens' efforts at ongoing meaningful collective engagement with public institutions for accountability in the provision of public goods.” Lodenstein et al. (2013) mentioned that “social accountability (also called citizen-driven accountability or bottom-up accountability) refers to the strategies, processes or interventions whereby citizens voice their views on the quality of services or the performance of service providers or policymakers who, in turn, are asked to respond to citizens and account for their actions and decisions.” The paper emphasized that the relevance of SA can be described from two perspectives. According to “Institutional economic perspective,” SA works as “complementary” to the administration that emphasizes on the “regulatory mechanism of monitoring.” However, besides this, SA can be described from broader trends in development, such as “participatory governance.” Based on these perspectives, the systematic review listed the expected outcomes of the SA initiatives to be “reduction in corruption; better governance and policy design; enhanced voice, empowerment and citizenship of marginalized groups; responsiveness of service providers and policymakers to citizens' demands and, ultimately, the achievement of rights, health and developmental outcomes” (Lodenstein et al., 2013). Grandvoinnet et al. (2015) presented a chart to show the “expanded impact” of SA (Chart 1). The author argued that it is difficult to conclude whether SA works or not. One of the main reasons is complexity of potential outcomes from SA process. The paper includes a wide range of important outcomes, such as “benefits in the state (better governance), in state-society relationships (increased legitimacy), and in society (improved provision of public goods) as well as instrumental benefits (improved provision of public goods) and institutional benefits (state building)” (Grandvoinnet et al., 2015). Marston et al. (2020) in their systematic review mentioned that SA interventions may ensure better service delivery by providing for example “better delivery of services (e.g., via citizen report cards, community monitoring of services, social audits, public expenditure tracking surveys, and community-based freedom of information strategies); better budget utilization (e.g., via public expenditure tracking surveys, complaint mechanisms, participatory budgeting, budget monitoring, budget advocacy, and aid transparency initiatives); improved governance outcomes (e.g., via community scorecards, freedom of information, World Bank Inspection Panels, and Extractives Industries Transparency Initiatives); and more effective community involvement and empowerment (e.g., via right to information campaigns/initiatives, and aid accountability mechanisms that emphasize accountability to beneficiaries).” With the development of the concept of SA over the decades, new outcomes such as community empowerment, providers' responsiveness are at the heart of discussions too (Hamal et al., 2018).

OPEN IN VIEWER

Figure Chart 1

Adopted from Grandvoinnet et al. (2015).

We can see that SA measures can go a long way in helping the poor, vulnerable and marginalized groups of society. In Asian countries, the marginalized represents a majority of the society and research suggests that compared to other regions, SA initiatives across Asia, particularly the south and south-east, have a much greater element of community participation and involvement (World Bank, 2007). Likewise, in many low- and middle-income countries (LMICs), the engagement between civil society groups and governments in many of these initiatives is very prominent in comparison to that in other developed regions. Also, focusing on LMICs would help to maintain consistency because SA mechanisms tend to be contextual specific. There is a growing recognition that context is critical in shaping, making, and breaking SA interventions (O'Meally, 2013).

Recent years have witnessed growing concerns about issues of governance and accountability in developing countries. There is an emergent need for identifying and promoting approaches towards building accountability that rely on civic engagement. SA includes an ever-widening range of concepts and distinct practices that needs to be identified, analyzed and researched upon in depth.

SA is about including citizens and communities within the forms of administration so that choices and activities of the individuals and associations with control are made open and can be addressed. To respond to the developing requirement for accountability as well as transparency, numerous multi-stakeholder activities have been made to reinforce public information revelation. These activities create a platform point to empower the society organizations to utilize the data and engage with the government for better development outcomes. These practices are initiated by a wide variety of factors including government agencies, different community platforms, political leaders, and aid and donor agencies. They use different forms of informal and formal strategies via support for formal and informal groups or organizations.

The intervention

The main objective is to map the size of the evidence of the SA interventions in LMICs. There can be multiple interventions through which SA outcomes can be achieved. The dimensions of this evidence and gap map (EGM) will place interventions in rows and outcomes in columns. The interventions related to access to information, citizens' voice and representation, governance, mobilization actions, resource monitoring actions will be listed. The column will be arranged by outcomes related to performance improvement of public and private institutions, service quality, allocative efficiency, citizen engagement, inclusion, responsiveness, quality of life, knowledge and learning outcomes from this project. This map outline is elaborately discussed in the EGM framework section later on.

The map will summarize the SA interventions in LMICs. We will map the studies published after the Year 2000. The map will accommodate systematic reviews, randomized control trials, non-experimental design and before-after design.

Why it is important to develop the EGM

An evidence gap map on the effectiveness of SA interventions in LMICs will help users to identify the size and quality of the evidence base of SA interventions in these countries; guide users to available high quality evidence to inform strategy and program development; show users where there is no high quality evidence; identify gaps to be filled by evidence synthesis and new studies for researchers and research commissioners – thus, providing a more strategic, policy-oriented approach to research agenda. This approach will assist in effective result based decision making and action oriented planning.

There is one related map on state society relations which consolidates evidence on interventions to improve state-society relations in LMICs (Phillips et al., 2017), however the focus is different from ours. There are other EGMs on SA interventions specific to different sectors (Rathinam et al., 2019; Munar et al., 2018). As far as we know, there are no EGMs that address the objectives that we have.

EGM: Definition and purpose

Mapping the evidence in an existing area has been used since the early 2000s (Saran & White, 2018). An EGM, being a systematic evidence synthesis by nature requires some systematic steps to follow. EGM primarily identifies the evidence gap in a selected field that can be filled by new evidence; thus, making it easier for experts to introduce new evidence in their given field by looking at the map. It also allows existing evidence to be easily discovered by researchers, policy makers and “research commissioners” (White et al., 2020). To map high quality evidence and measure the size and quality of the evidence base of SA interventions in LMICs, we aim to develop this map following stages like Welch et al. (2021):

Define a framework with stakeholder engagement

Framework development and scope

Our EGM will focus on the wide range of interventions on SA. We will use a well-known conceptual framework to define our intervention and outcome categories. To do this, we will use existing literature and consult with subject experts to develop a concrete framework. The field of SA is growing rapidly, hence we will ensure our framework captures new interventions and outcomes along with traditional ones. To generate the framework, we will follow the following steps:

Brainstorming workshop: In this workshop, the research team has identified possible intervention and outcome categories.

Stakeholder engagement

We have developed a team consisting of experts within BIGD, along with the researchers. The researchers developed a draft framework in consultation with internal expert. The draft was then presented to a team at Global Partnership for Social Accountability (GPSA). After multiple consultations with GPSA and Campbell South Asia, the framework has been finalized. The framework was tested through pilot coding of 20 studies. Although the core research team does not include an information specialist, the search strategy has been reviewed by the information specialist from Campbell South Asia. The team also presented the EGM framework at the Global evidence summit, 2020 and gathered feedback. The stakeholders will be engaged at the time of report finalization and dissemination.

Conceptual framework

The concept of SA is continuously evolving. The concept involved a wide range of interventions that builds citizens' as well as the state's power. It includes different categories such as access to information, citizen's voice and representation, governance, mobilization actions, and resource monitoring actions (Table 1). The interventions are intended to bring about different outcomes (Table 2) such as performance improvement of public and private actors/institutions, service quality, allocative efficiency, citizen engagement, inclusion, responsiveness, quality of life, knowledge and learning outcomes from this project. Thus, this map will be built on a combination of frameworks of different interventions. Figure 1 shows a causal line from interventions to impact for SA interventions. This has been developed based on the causal map and literature by Itad (2016), Franco & Shankland (2018), Joshi (2014) and Fox (2015).

OPEN IN VIEWER

Figure 1

Theory of change for social accountability interventions. Source: Itad (2016), Franco & Shankland (2018), Joshi (2014).

Types of study design

This EGM will focus on the effectiveness of SA intervention. This will include all effectiveness studies and systematic reviews. Studies that follow the below designs will be included:

Systematic review: Review of primary studies adopting systematic approach, which includes: (1) clear inclusion and exclusion criterion, (2) an explicit search strategy, (3) a systematic coding and analysis of included studies, (4) meta-analysis (where possible)

Types of intervention/problem

The EGM will focus on five broad categories of intervention. Within each of these categories, there will be different sub-categories (Table 1). The definitions can be found in Supporting Information: Additional Table A1 at the end.

Access to information

Types of population (as applicable)

This EGM will focus on the intervention of SA in both rural and urban areas of LMICs. The following population will be included:

Women and girls,

Types of outcome measures (as applicable)

This EGM will focus on 8 categories of outcome (Table 2): (The definitions can be found at Supporting Information: Additional Table A2).

Performance improvement of public and private actors/institutions

Types of location/situation (as applicable)

Our EGM location is LMICs.

Types of settings (as applicable)

Studies in the following settings will be included and excluded:

Report structure

Our EGM report will include the standard sections: executive summary, background, methods, results, and a conclusion. We will indicate any changes we made between the protocol and the final report. The executive summary will contain the summary and findings of the report. The background will describe the SA literatures and provide a brief description of how SA interventions have been evolving. We will also present a theory of change, followed by the scope of this particular EGM. This sub-section will have description on our framework, interventions categories and outcome categories.

The report will also present a detailed search strategy, methods of including or excluding studies and how exactly the data is extracted from the included studies. A comprehensive quality appraisal plan will also be included. This section will contain a PRISMA flow chart, followed by a full search strategy for some databases in the appendix.

The results section will present the number of studies retrieved from our database search and provide an overview of the types of studies by intervention, outcomes, auspices and other filters used. The conclusion will provide a detailed account of policy implications for researchers and decision-makers and highlight key areas for research commissioning.

We will have some main figures and tables such as:

Conceptual framework/theory of change

Filters for presentation

The unit of analyses will be number studies that are included under each sub-category combination of intervention and outcome. We will present the EGM in a standard format: total number of studies, broken down into sub-categories of intervention and outcome. We will also present a narrative report using figures, tables to narrate the whole process, summaries of number of included studies and coding process. The report will also discuss the limitations of the map while discussing the use of the EGM. The key dimensions will be interventions (presented in rows) and outcomes (presented in columns). We will use bubbles of varying sizes to present included studies. Different colors will be used for different types of studies. Final decisions about the filters will be made based on the number of included studies and coded information. The online interactive map will be hosted by Campbell.

Dependency

We will treat multiple reports (e.g., secondary analyses or protocols) of a single study as one. Systematic reviews will likely include primary studies that are included in the map, and there may be more than one systematic review which may include the same primary study. Primary studies which meet our eligibility criteria will be included in the map regardless of whether they are included in a systematic review.

Screening and study selection

A team of two independent researchers will be responsible for searching and screening the studies initially. Any disagreement on particular studies will be resolved by the third researchers.

Data extraction and management

All the authors will be involved in data extraction, coding and management. Three co-authors will independently code all eligible studies and reviews. Any disagreement will be resolved by the lead authors. Besides the “interventions” and “outcomes” mentioned in previous section, we will code several other factors to filter the studies in EGM (Table 3).

Tools for assessing risk of bias/study quality of included reviews

We will appraise the methodological quality of systematic reviews with (1) clear inclusion and exclusion criterion, (2) an explicit search strategy, (3) a systematic coding and analysis of included studies, (4) meta-analysis (where possible) in duplicate for 20% of eligible studies. If the agreement is over 80%, we will proceed with single data extraction with verification by a second reviewer for the remainder of studies. The methodological quality of primary studies is not usually assessed in EGMs.

Methods for mapping

EPPI reviewer 4 is used to perform screening and coding and we will be using the EPPI mapper utility to develop the online map.

Plans for updating the EGM

Ashrita Saran and Iffat Zahan will be responsible to update the EGM provided availability of funds.

Internal sources

Research, Policy, and Governance (RPG) program, BIGD, BRAC University, Bangladesh

RPG is an internal program of BIGD. This evidence and gap map is funded by this program.

External sources

Campbell Collaboration, India

Campbell collaboration is supporting BIGD intellectually to develop this EGM.