Interventions promoting uptake of water,sanitation and hygiene (WASH) technologies in low- and middle-income countries: An evidence and gap map of effectiveness studies

“How”: Intervention mechanisms

Intervention mechanisms were defined so that personal and household WASH promotional approaches would be comprehensively included, and the categories defined mutually exclusive. Table 2 presents the main categories and subdivisions. Mechanisms for providing WASH technologies can be categorised into demand and supply side intervention mechanisms.

On the supply side, intervention mechanisms include: direct provision of technology by an external body (e.g., government, NGO); improving operator performance (e.g., institutional reform, capacity building, operator financing, legal regulation of providers, and accountability); privatisation and nationalisation of service delivery; and promoting small-scale independent provider (SSIP) involvement (e.g., sanitation marketing through microloans and capacity building for providers). Direct provision of hardware covers all intervention mechanisms where WASH technology is provided at zero capital cost to users (e.g., Feachem et al., 1978). Hardware may be for use in private (household and yard) or public spaces (shared facilities, WASH in health facilities and schools, places of work, commerce, recreation, streets, fields and transit hubs). Measures to improve service provider performance, such as enaction and implementation of water quality standards (Cairncross et al., 1996), government regulation of private utility providers (e.g., Ministry of Foreign Affairs, 2011), and reforms to operator financing (e.g., output-based aid or payment-by-results) (Trémolet & Evans, 2010). Stimulating involvement of the private sector, including privatisation (e.g., Galiani et al., 2005) (or renationalisation), contracting out to encourage involvement of the private sector and SSIPs, including nonprofits, in WASH services provision (Sansom et al., 1999), and capacity building of independent providers. As an example of the latter, sanitation marketing aims to increase availability of sanitation technology and maintenance services (such as pit emptying), by training local artisans to produce sanitation products that are suitable for the varying needs of consumers (e.g., Cameron et al., 2013).

A final group of interventions intervening on demand and supply sides includes decentralisation (Poulos et al., 2006). Decentralised delivery places community representatives at the core of the planning, design, implementation, and operation of the WASH service provider. For example, community-driven development (CDD) uses a participatory approach, block grants with cost sharing, and often a component of local institutional strengthening (White et al., 2018). Another approach is Water User Associations, where management is devolved to the community group while government retains some powers (e.g., Barde, 2017; Waddington et al., 2019). Demand and supply intervention mechanisms may also be combined, on the demand side (e.g., where financing is combined with health messaging or BCC) and in combinations of demand- and supply-side (e.g., direct provision with BCC).

“What” and “where”: WASH technologies and places of use

The quality of water supply, sanitation and hygiene facilities—that is, the extent to which they are likely to provide drinking water of sufficient quantities for basic needs, enable hygienic hand-washing and food preparation, and safe removal of excrement from the human environment—is dependent on the type of facility as shown in the WASH ladders (Table 3).

An important dimension of the technology is the social and physical environment where the participants interact with it. Categorising WASH by place of use emphasises the differential effect, and potentially different causal pathways, of providing the same technology in different locations (Table 4). Place of use affects acceptability and convenience to users, and therefore adoption rates, as well as how the intervention disrupts the causal chain of disease transmission.

For example, hygiene behaviour change is more likely to break disease transmission in households, whereas investments in infrastructure such as drains and excreta disposal systems are more likely to affect disease transmission in public spaces (Cairncross et al., 1996). Factors of control are likely to be weaker in community settings than institutional settings, such as schools and day care centres, where simple hygiene messaging and “behavioural nudges” are more likely to be effective (e.g., Ryan et al. 2001). The four main spaces in which WASH technologies are provided are the home and yard (for use by an individual household), the community (spaces shared by two or more households, including fields, streets and places of work, commerce and recreation), at school, and at a health facility. The evidence map therefore includes filters for the WASH technology provided and the place of use, as well as location (rural, urban, informal (peri-) urban settlement and refugee camp).

“For whom”: WASH technology users

The final relevant dimension for the intervention is the suitability of the WASH technology to different users. For example, women's needs change over their life-cycle, hence WASH service provision needs to be suitable for different points in the reproductive life-cycle, including menarche (e.g., separate toilets for girls at school, promotion of menstrual hygiene management approaches) and maternity (e.g., WASH in health facilities, promotion of hygienic weaning practices) (Figure 5). Caruso et al. (2017) define sanitation insecurity as “[i]nsufficient and uncertain access to socio-cultural and social environments that respect and respond to the sanitation needs of individuals, and to adequate physical spaces and resources for independently, comfortably, safely, hygienically, and privately urinating, defecating, and managing menses with dignity at any time of day or year as needs arise” (p. 9). Other group who are disadvantaged or vulnerable may also have particular needs, such as water and sanitation facilities for the elderly and infirm, or drinking water treatment for immunocompromised people (e.g., those living with HIV). For example, walkways may need to be constructed to prevent falling and elevated seats or rails installed to help elderly people, those with disabilities, and pregnant women (Caruso et al., 2017).

OPEN IN VIEWER

Figure 5

Female reproductive health over the life course. Source: Water Supply and Sanitation Collaborative Council

Type of populations

Studies were included on any population in LMICs, as defined by the World Bank at the time the research was carried out. Populations of any age, sex, gender, disability or socioeconomic status were included, provided the study was conducted in endemic conditions found regularly in LMICs. Hence, studies that were conducted under outbreak conditions, such as cholera epidemics, were excluded (e.g., Daniels et al., 1999; Snow, 1855).

Types of interventions

Studies were included that measured receipt of a clearly defined WASH intervention.

Types of outcome measures

Studies measuring outcomes among human populations were included. Studies that collected outcomes on environmental or animal subjects were excluded, for example, those measuring the efficacy of water treatment technology in the environment (e.g., Crump et al., 2004). Table 5 lists eligible outcomes, which included behaviours (e.g., time savings from improved water availability), water-related ill-health (e.g., diarrhoea and ARIs), other ill-health (e.g., musculoskeletal injury, stress), nutrition (e.g., height-for-age), mortality, and socioeconomic outcomes (e.g., income, education, employment). To be included, the outcome needed to clearly relate to a WASH intervention mechanism or exposure. For example, where some programme evaluations of CDD—an approach that is used to provide projects in multiple sectors such as infrastructure, education and health—did not give estimates of outcomes separately for WASH projects, these outcomes were excluded from the map (e.g., diarrhoeal infection in Beath & Enikolopov, 2013).

All measures of intervention outputs, such as participant knowledge, awareness and attitudes were excluded from the map, which is concerned with programme effectiveness, rather than implementation. Facility access was only included when an intervention mechanism promoted construction of latrine, water supply or hand-washing facilities. This means, for example, that studies of direct hardware provision reporting only latrine ownership, or knowledge resulting from hygiene education, or awareness and attitudes (e.g., Stockman et al., 2007), were omitted. In contrast, studies (and outcomes in such studies) were included that measured latrine ownership (e.g., Guiteras, Levinsohn, et al., 2015) in the context of latrine promotional intervention mechanisms (CLTS, subsidies).

Willingness-to-pay was included where the method used real purchase decisions and was excluded where based on hypothetical scenarios which are unreliable, partly because survey respondents may strategically overstate it (Null et al., 2012; Whittington, 2002). Where studies collected data on pre-existing health status, which was subsequently used in stratified analysis, such as reporting differential effects on stunted versus nonstunted children (Luby et al., 2005), pre-existing health was not included as an outcome or population filter.

Adverse outcomes were eligible but no studies reported any; for example, the effect of hygiene on reducing microbial exposure, potentially leading to rise in allergy and auto-immune diseases, also known as the “hygiene hypothesis” (Bloomfield et al., 2016; Strachan, 2000).

Types of evidence

The evidence map includes impact evaluations and systematic reviews of the effectiveness of WASH intervention mechanisms and technologies. Impact evaluations were defined as programme evaluations or field experiments that used quantitative approaches applied to experimental or observational data to measure the average effect of participating in a programme relative to a control or comparison group (counterfactual) representing what would have happened to the same group in the absence of the programme. Eligible impact evaluations also tested different intervention mechanisms or technologies (i.e., active controls). Both completed and on-going impact evaluations and systematic reviews were searched for and included.

The following study designs were included: (1)

Studies explicitly described as systematic reviews or meta-analyses, which synthesised evidence on effectiveness of WASH intervention mechanisms or exposures, and described methods used for searching, data collection. For completeness, two early literature reviews on which much of the subsequent systematic review literature is based, were also included (Esrey et al., 1985, 1991). Systematic reviews of effectiveness were eligible, not those addressing other aspects such as prevalence of child faeces disposal in LMICs (Gil et al., 2004).

In addition, the following study designs were included for specific outcomes:

Study designs that were not related to a clearly defined intervention were excluded (e.g., Feachem et al., 1978; Jalan & Ravallion, 2003; Root, 2001; Wagner & Lanoix, 1959), or those without a comparator receiving a different intervention or service (e.g., Israel, 2007). This excluded some natural experiments where no intervention could be identified (Geruso & Spears, 2018).

Studies, or components of studies, that collected and analysed purely qualitative evidence were excluded. For example, in a controlled study of slum upgrading by Parikh and McRobie (2009) in Gujarat, India, women reported saving time and labour, and having fewer back problems, as a result of no longer having to carry buckets of water. This information was collected using qualitative interviews and presented in quotation rather than quantitatively; therefore, it was excluded.

Search

Systematic searches were done for both published and “grey” (i.e., nonpeer reviewed) literature. The existing electronic database searches for the 2014 EGM and a 2017 systematic review, on hygiene and sanitation behaviour change intervention mechanisms (De Buck et al., 2017), were updated to March 2018. Searches were also run to cover the rest of the extended scope, particularly water behaviour change and WASH in health care facilities. All search word lists were developed by an information retrieval expert and, in February 2018, eleven academic databases and four trial registry databases were searched (Supporting Information Appendix A). To capture grey literature, hand searches were conducted of key organisation websites. These included the Impact Evaluation Repository of the International Initiative for Impact Evaluation (3ie), the Asian Development Bank, African Development Bank, Inter-American Development Bank, Department for International Development, IMPROVE International, IRC (WASH),

Excluded studies

At the title and abstract stage, most studies were excluded because they were clearly not related to WASH, they used reflexive control (uncontrolled before-after study) design without collecting data on time-savings, or they were an efficacy or laboratory-based study. One ITS study of hygiene promotion in Burkina Faso with three preintervention measurement rounds but only one post intervention round measuring of children and mother's latrine and hygiene behaviour was therefore excluded (Curtis et al., 2001). Nearly half the papers were excluded based on study design at full text stage; this was often because they were studies of WASH exposures not intervention mechanisms. While exposure studies are useful in establishing the relationship between WASH infrastructure and practices and health outcomes (e.g., Geruso & Spears, 2018; Jalan & Ravallion, 2003), they do not provide evidence of the intervention mechanisms used to improve WASH access and use. The second biggest reason for exclusion was on intervention, where the WASH component could not be isolated as it was provided with other non-WASH cointerventions. For example, the Sanitation, Hygiene, Infant Nutrition Efficacy (SHINE) trial (Humphrey et al., 2019) combined WASH with nutrition. The WASH Benefits trials (Luby et al., 2018; Null et al., 2018) also intervened in nutrition and WASH, but had trial arms with only WASH interventions, which were eligible for inclusion in the map. Beyond combined WASH and nutrition interventions, the other common combination was WASH with deworming medication, a famous example being Miguel and Kremer (2004) which incorporated a hygiene intervention. Most of the studies excluded for outcome measured only knowledge or attitudes. To avoid duplication, 44 papers were excluded because they were a different version of a study already included. Finally, 41 studies were excluded because they could not be located (e.g., the search had found only a conference abstract with insufficient detail to code the study in full) or accessed in full text. A complete list of the studies excluded at full text is included in the references section.

Visualisation and analysis

The online, and interactive, EGM provides a matrix of intervention mechanisms against outcomes as described in Section 2. In brief, the matrix displays intervention mechanisms (e.g., direct hardware provision) against outcomes along the causal pathway (from behaviour change to socioeconomic impacts). There are then several filters to further breakdown the evidence. The most important of these is the WASH technology filter (e.g., latrines for household use), but other filters include region, country, study design (RCT, nonrandomised study, or natural experiment), and population (e.g., people living with HIV or disability).

This report presents the major trends in the interventions that were researched, outcomes reported, participants being targeted, and the findings from the systematic reviews. It was decided to use 2008 as a cut-off to present the overview of research in the WASH field, demarcating “first period” and “second period” rigorous evaluation studies. The International Year of Sanitation in 2008 brought attention to the importance of sanitation technologies to the policy and research communities, and catalysed funding for WASH evaluations from traditional and nontraditional organisations (e.g., Gates Foundation) in an area that was previously considered too costly for impact evaluations to be applied (Cairncross et al., 2014).

Search results

The number of rigorous impact evaluations and systematic reviews of water, sanitation and hygiene interventions has grown exponentially over both the last decade and even in the years since the initial WASH EGM was produced (Waddington et al., 2014). In this section, we use the International Year of Sanitation, 2008, as the cut-off marking a “behavioural revolution” in impact studies and systematic reviews in the WASH sector. At the time the searches were completed, there were 337 completed and 46 on-going impact evaluations using quantitative counterfactual methods in LMICs, nearly three-quarters of which had been conducted after the end of the International Year of Sanitation, from 2009 onwards. There were also 42 completed systematic reviews and three protocols, of which all but four had been published since 2008. Since the initial WASH evidence map was published, at least 250 additinoal impact evaluations have been found, and 20 systematic reviews have been completed. One RCT of hand sanitiser produced by Proctor and Gamble, which had been identified in a previous systematic review by the author, remained unpublished (Odio et al., 2004). Sufficient information was available in that study's abstract for inclusion in the map. In total, therefore, there are at least 359 completed and 22 on-going impact evaluations of WASH interventions in LMICs, nearly three-quarters of which have been completed since 2008. There are also at least 43 systematic reviews and 2 protocols, of which all but four were completed after 2008.

Searches were also done in 2020 to locate impact evaluations and systematic reviews that had been found in trial and protocol registries in 2018, which had since been completed. These searches found 20 impact evaluations had been published by May 2020 (Acey & Norman, 2017; Armand et al., 2020; Augsburg & Oteiza, 2014; Batmunkh et al., 2019; Chauhan et al. (for Curtis et al., n.d.); Cocciolo et al., 2017; Viswanathan et al., 2020; Delea et al., 2020 (for Freeman et al., 2017); Trent et al., 2018; Dreibelbis et al., 2019; Dupas et al., 2016; Friedrich et al., 2018; Gray, 2019 (for Stewart, 2013); McGuinness et al., 2017; Nagel et al., 2016; Oviedo & Rounseville, 2017; Peletz et al., 2019; Rabbani, 2015; Reese et al., 2017; Vijayaraghavan & Kilroy, 2017), and one systematic review (Majorin et al., 2019).

Figure 8 presents the preferred reporting items for systematic reviews and meta-analyses (PRISMA) study search flow diagram. A complete characterisation of each included study, including its intervention mechanism, technology, region, country, target population, and study design or quality, is provided in the supplementary online material.

OPEN IN VIEWER

Figure 8

PRISMA study search flow diagram

Interventions

Impact evaluations of WASH interventions have been conducted in 83 LMICs (Figure 9). There is a high concentration of studies in Bangladesh, Kenya and India, each having over 50 WASH intervention studies. In addition, Bolivia, Cambodia, Ethiopia, Ghana, Pakistan, Rwanda and Uganda each have 10 or more. Figure 10 also shows the global distribution of impact evaluations with hygiene studies (either solely or combined with other WASH interventions). The most hygiene studies have been done in Bangladesh, with 32 studies, while Ethiopia, India and Kenya all have 10 or more.

OPEN IN VIEWER

Figure 9

Map of WASH impact evaluation interventions in LMICs. Source: created using http://chartsbin.com/

OPEN IN VIEWER

Figure 10

Map of hygiene impact evaluation interventions in LMICs. Source: created using http://chartsbin.com/. LMIC, low- and middle-income country

Over the past 15 years the focus of impact evaluation research has shifted from analysis of direct WASH technology provision by external authorities (e.g., governments and NGOs) to promotional interventions. There has been a particular increase in studies of BCC using psychosocial “triggering”, particularly of latrines and hand-hygiene. In sanitation, this is most commonly CLTS. CLTS aims to ensure open defecation free (ODF) environments and increase the use of latrines by leveraging social cohesion to make collective behavioural changes, but can also include information campaigns focused on disgust or being a good parent, as well as incentives for community leadership to achieve ODF (Spears, 2013). Hygiene promotion includes approaches like “super-Amma” (super-Mum), which used disgust and pride to incentivise improved hand-washing practices (Biran et al., 2014). Having said this, the traditional approaches of direct hardware provision (e.g., dispensing water filters) or health education (i.e., providing information about the consequences for health of not washing one's hands) remain common intervention mechanisms, even among new studies (Figure 11).

OPEN IN VIEWER

Figure 11

WASH intervention mechanisms by publication date

Up until the end of the International Year of Sanitation, 2008, interventions to provide clean drinking water and hand hygiene had been the priority for intervention research. Almost 50% of studies had looked at how to improve water supply and quality. Household water treatment is still the most studied technology (around 30%). However, more studies (e.g., Brown et al., 2012; Klasen et al., 2012) including two randomised encouragement trials (Ben Yishay et al., 2017; Devoto et al., 2010), have broadened the evidence base on health impacts of water supply provision. The number studies of sanitation technology has also increased from 8 to 62, and there has been a similar increase in the numbers of studies examining hygiene (from 23 to 97). Figure 12 shows the change in the studies examining each category before and after 2008.

OPEN IN VIEWER

Figure 12

WASH technologies by publication date

The place of use further distinguishes technologies that are physically tied to a geographic location (Figure 13). Traditionally hardware has either been provided directly to a household or for communal use by a village, informal urban settlement (slum), or neighbourhood. There has, however, been increasing interest in the effect of providing WASH facilities at schools over the last 20 years. SDG target 4.A highlights the need for adequate WASH in schools. Correspondingly, most impact evaluations provide hardware directly to households. However, there are 39 studies on the communal provision of water supply (e.g., shared boreholes) and water quality technologies. Furthermore, although there were only two studies specifically examining WASH infrastructure in schools before 2008, there are now 39. The majority of these are combined water, sanitation, and hygiene technologies that act as a comprehensive overhaul for the school. Only one controlled impact evaluation was found of provision of handwashing supplies in healthcare facilities, and none were found in healthcare facilities for any other kind of WASH technology (Figure 15). It is worth noting the growing interest in WASH in healthcare facilities (WHO/UNICEF, 2019b) and the World Health Assembly, WASH in healthcare facilities is likely to be very high in people's priorities.

OPEN IN VIEWER

Figure 13

WASH technologies by place of use

Outcomes

The community of research producing WASH health impact evaluation studies is highly active. The total population included in WASH impact evaluations in LMICs included in this map is at least 5 million participants. Diarrhoeal disease, particularly carer-reported diarrhoeal morbidity among children, remains the standard health impact measure used, and is by far the most commonly reported outcome (Figure 15). A systematic review from 2009 estimated 71 completed studies of WASH projects had been conducted measuring diarrhoeal morbidity (Waddington et al., 2009). The most recent systematic review of WASH and diarrhoea morbidity (Wolf et al., 2018) included 135 studies, and the evidence map presented here includes 186 studies measuring diarrhoea morbidity, 119 of which were in studies published after 2008. This map also indicates more than a million people have taken part in trials measuring, or had data collected on, diarrhoea morbidity.

Recognising the importance of WASH for controlling ARIs, the coverage of studies examining impacts on ARIs of hygiene promotion has also increased, with 35 studies measuring ARIs including 31 in studies published post-2008, including large-scale studies in in Vietnam (Chase & Do, 2012), Colombia (Correa et al., 2012), Bangladesh (Huda et al., 2012), Guatemala (Arnold et al., 2009) and Egypt (Talaat et al., 2011). Studies of the transmission of causative agents in unhygienic environments in LMICs have also measured the presence of respiratory infections like coronaviruses in stool samples (e.g., Esrey et al., 1987). However, given the importance of ARIs in the global burden of disease, due to pneumonia and influenza—as well as currently enhanced importance during the COVID-19 pandemic—the total of number of participants in WASH studies of ARIs, at only only 125,000 in LMICs, remains very limited (Howard et al., 2020).

In line with the other changes, there has been a shift in the commonly reported outcomes, including an increase in studies reporting behavioural outcomes since the early 2000s (Figure 14). This is an important shift as the principal argument used by proponents of alternative delivery mechanisms is that they are more effective at changing these behaviours and therefore improving lives (e.g., Kar & Chambers, 2008).

OPEN IN VIEWER

Figure 14

Number of impact evaluations by outcomes

However, interventions fostering marginal improvements in personal WASH behaviour may not cause sufficient changes at community level to improve quality of life outcomes like child nutrition or diarrhoea mortality (Geruso & Spears, 2018). Hence, some behavioural studies also attempt to measure these outcomes too, since changing individual behaviours might not be enough to affect these “endpoint” outcomes. In other cases, repeated exposure to enteric infection may cause chronic malabsorption and food wastage (enteropathy), which one included study attempted to measure using a reference population of healthy individuals (Shiffman et al., 1978).

Although there has been an increase in studies in schools, it is still relatively few studies compared to other settings, given the importance of schools for transmission of infection in the public domain. Only three studies measured WASH in refugee camps or humanitarian emergency. Two of the studies were of water treatment technologies in refugee camps in Africa (Doocy & Burnham, 2006; Roberts et al., 2001). It is likely that water supplies are insufficient in refugee camp settings for studies of hygiene and insufficient space for latrines to be provided in sufficient density to impact on health. Very few studies have been done measuring time use and labour market outcomes, willingness-to-pay based on real purchase decisions, or studies measuring psychosocial health or injury (e.g., relating to attack or pedestrian road traffic accidents). Additionally, and critically, evidence of longer-term behaviours, including slippage back to bad practices, is very limited.

The most commonly reported behaviours are handwashing, water treatment and handling, and latrine use. Many of the studies reporting hygiene behaviour include measures of personal food hygiene; nearly 50 studies specifically collected data on handwashing before food preparation, five reported on the microbial contamination of food or eating utensils, and 17 reported on other food hygiene outcomes, such as whether food was stored properly and dishes washed appropriately. It is important that hygiene studies examine food hygiene outcomes, given the importance of food in faecal-oral disease transmission (Wagner & Lanoix, 1957). Studies collecting water supply behaviour outcomes included 40 interventions to reduce faecal contamination and six of chemical contamination due to arsenic in Bangladesh.

No studies estimating impacts on macro-level outcomes such as private sector investment, employment generation or environmental impacts such as dumping or river water quality were found.

Gender and vulnerable populations

Existing impact evaluations cover rural, urban, and slum populations with the vast majority either targeting or including rural communities (295 studies), which are usually the most disadvantaged areas in terms of WASH service provision.

Research in WASH has long paid attention to vulnerable groups, such as poor women and girls, and their specific needs (Cairncross & Cliff, 1987; White et al., 1972). However, there is a long way to go before impact evaluation research and systematic reviews of these studies achieve the full integration of gender, and other equity concerns and become a consistently transformative sector (Interagency Gender Working Group [IGWG], 2018).

As discussed in the introduction, globally women and girls carry most of the burden of water collection, including time, calories spent, musculoskeletal injuries, having to use unsafe places to defecate, where water and sanitation services are not accessible, risking assault by people or attack by wild animals—so-called “pests and perverts” (Campbell et al., 2015). Impacts on time use were not frequently studied in the context of intervention research in WASH. For example, a previous review of diarrhoea morbidity cited three studies which examined time savings from improved water supplies in India (Pattanayak et al., 2007), Nigeria (Blum et al., 1990) and China (Wang et al., 1989).

In the last 10 years, new studies have been conducted that evaluated interventions and outcomes that disproportionately affect women and girls. This includes the 23 studies measuring various aspects of time savings and alternative uses of time due to water supply improvements, including in Argentina (Galiani et al., 2009), Bangladesh (Hasan & Gerber, 2016; Madajewicz et al., 2007), Benin (Ruben & Kirk, 2011), Burkina Faso (Briand & Lare-Dondarini, 2017), Ghana (Arku, 2010; Okyere et al., 2017), Guinea (Ziegelhoefer, 2012), Honduras (Instituto Apoyo, 2000), India (Jeuland et al., 2015; Pattanayak et al., 2010; World Bank, 1998a), Kenya (Jack et al., 2015), Lesotho (Feachem et al., 1978), Morocco (Devoto et al., 2012), Mozambique (Cairncross & Cliff, 1987; UNICEF, 2011), Nigeria (Toonen et al., 2014), Pakistan (Rauniyar et al., 2009), Philippines (Aiga, 2002), St. Lucia (David et al., 2004), Yemen (Dahl-Østergaard et al., 2010; Klasen et al., 2011). In addition, one study measured time use due to improved latrine coverage (Dickinson et al., 2015).

There are four completed impact evaluations measuring psychosocial health outcomes in India (Khush et al., 2009), Morocco (Devoto et al., 2012), Uganda (Montgomery et al., 2016), and safety and vulnerability in Brazil (Bobonis et al., 2017). Five more were identified as ongoing in 2018, two of which are now completed (Dreibelbis et al., 2019; Delea et al., 2020).

Women and adolescent girls also experience hardships where inadequate services constrain menstrual hygiene management. There are now also five studies of menstrual hygiene management in Ghana (Montgomery et al., 2012), Iran (Fakhri et al., 2012), Kenya (Phillips-Howard et al., 2016), Nepal (Oster & Thornton, 2011) and Uganda (Montgomery et al., 2016) and two systematic reviews (Hennegan & Montgomery, 2016; Sumpter & Torondel, 2013). However, only a single study in Pakistan (Rauniyar et al., 2009) reported impacts from improved water provision on self-reported drudgery, measured as incidence of muscle strain, blisters or backache. No study has measured musculoskeletal disorders arising from long-term water carrying.

Gender analysis is rarely used as part of the framework for understanding programme effects in impact evaluations and systematic reviews. In fact, only a minority of studies included in the map (19% of impact evaluations and 20% of systematic reviews) reported any sex disaggregated outcomes. In impact evaluations, psychosocial health (43%), education and cognitive development (40%), open defecation (33%), and time use (26%) were some of the most commonly sex disaggregated outcomes. It is also notable that the map was not able to identify any controlled impact evaluations reporting sanitation use by nonbinary or transgender individuals.

There were no controlled studies either examining WASH interventions and technologies that targeted people living with a disability or, most strikingly, reporting on the success of standard WASH approaches in improving outcomes for these disadvantaged groups. This may be in part due to large sample size requirements necessary to measure outcomes among these populations, although there are presumably places where studies could be done relatively easily (e.g., special schools). For instance, an uncontrolled cohort study was conducted of hygiene in child day care facilities in Brazil (Barros et al., 1999), and intervention studies have been conducted of day care in high income countries (e.g., Black et al., 1981; see also Wolf et al., 2018).

The findings are also sparse when looking at other vulnerable populations. There is one impact evaluation (Abebe et al., 2014) and two systematic reviews (Peletz et al., 2013; Yates et al., 2015) that examined people living with HIV, who often have different social constraints and medical needs. There are two impact evaluations (Doocy & Burnham, 2006; Roberts et al., 2001) that specifically look at the needs of those living in refugee camps, and one impact evaluation (Chase et al., 2002) and two systematic reviews (Brown et al., 2012; Ramesh et al., 2015) looking at those living through, or in the aftermath of, a humanitarian crises.

Study design

Well over half of impact evaluations (comprising over 250 trial arms) used randomised assignment (RCTs) (Table 7), indicating the extent of support in academic and research funding communities for this study method. Some RCTs have taken full advantage of the power of the methodology by conducting comparative designs with prospective randomised assignment to alternate intervention mechanisms. Guiteras, Levinsohn, et al. (2015) provided an example in Bangladesh comparing different groups that were provided community sanitation promotion (CLTS) or latrine subsidies, on rates of open defaecation. Other recent examples of comparative designs include Luby et al. (2018) in Bangladesh and Null et al. (2018) in Kenya which compared WASH interventions individually and in combination.

Typical nonrandomised study designs include cross-section studies with statistical matching (e.g., Abou-Ali et al., 2009), group level panel data studies analysed at aggregated administrative levels (e.g., Galiani et al., 2005), individual-level panel data studies (e.g., Galiani et al., 2009), pseudo-panels with repeated cross-section from the same clusters (Galdo & Briceno, 2005), case-control studies (e.g., Meddings et al., 2004), and nonrandomised pipeline studies (e.g., Cairncross & Cliff, 1987). In NRS using matching, the matching was usually done using statistical methods, although a few used “naïve” matching (e.g., World Bank, 1998).

A few NRS (11) have taken advantage of existing data to conduct rigorous, and potentially highly cost-effective, evaluations with selection on unobservables, here called natural experiments (Ao, 2016; Calzada & Iranzo, 2013; Galiani et al., 2005; Galiani et al., 2009; Granados & Sanchez, 2014; Kosec, 2013; Spears, 2013; Tiwari et al., 2017; Ziegelhoefer, 2012). Methods used to analyse data in natural experimental frameworks include RDDs (Ziegelhoefer, 2012), ITS (e.g., Duflo et al., 2015) and panel data regression (e.g., Galiani et al., 2005). Figure 15 presents the evolution of study designs over time, indicating the marked increase in studies produced after the end of the International Year of Sanitation, from 2009 onwards, of completed and registered studies.

OPEN IN VIEWER

Figure 15

WASH impact evaluations by study design. Dotted line shows the end of the International Year of Sanitation (2008). The apparent decline in production of studies post-2018 reflects the limited searches done in this map after 2018. NRS, nonrandomised studies; RCT, randomised controlled trial

Natural experiments that applied statistical methods of correction for unobservable confounding to existing surveys, therefore, remain an under-utilised methodological approach in the WASH sector, given the large numbers of existing household survey datasets available that contain questions on access to WASH services which, for example, could be analysed as pseudo-panels using double-difference methods. It is also important to note that there continues to be a great number of uncontrolled studies that simply measure outcomes before and after the intervention. Most of these studies were excluded from the map as they are not usually able to attribute changes to the intervention, the exception being for the immediate outcomes of time savings due to provision of a new water supply or sanitation source (e.g., Cairncross & Cliff, 1987).

Impact evaluations

Blum and Feachem (1983) conducted a review of six areas where WASH impact evaluation designs were suboptimal: use of a control group, adjustment for confounding, definition of the outcome, length of recall, analysis of use, and group sample size, which they referred to as “one-to-one” comparison.

Studies addressed confounding either through random assignment, group or individual level matching on observables before analysis, or directly in adjusted analysis. For example, nearly all RCTs used centrally administered randomisation, although there was the occasional exception where a study used quasi-randomisation through alternation (Montgomery et al., 2016). Some studies used randomisation over very small samples, such as Stone and Ndagijimana (2018) who randomised across two districts in Rwanda.

Outcomes were nearly always clearly defined for diarrhoea (95% of cases) usually being the WHO definition of “three or more loose stools in a 24 h period”, and where diarrhoea incidence was recorded “three intervening diarrhoea-free days” were required to define a new episode (Bacqui et al., 1991). Outcomes data collection, whether on behaviour or quality of life, remains heavily reliant on self-reporting, which is thought to be highly susceptible to biases, including social desirability, also called “courtesy bias”. A different form of courtesy bias may also manifest in reported data due to participant fatigue when repeated follow-up data are collected, also called the “Bugger-Off Effect” (Clasen, 2013; see also Schmidt & Cairncross, 2009). “Survey effects”, or measurement as treatment, have also been empirically observed in repeated follow up studies (Schmidt et al., 2011; Zwane et al., 2011). Indeed, following the publication of papers on bias arising due to repeated measurement (Schmidt et al., 2011), the mean number of survey rounds for health impact studies measuring self-reported diarrhoea fell from 23 to 7, and the median fell from 12 to 3.

Other factors that may cause bias in reported data have also improved. For example, for self-reported diarrhoeal disease, only a minority of studies used recall periods longer than two weeks (Elbers et al., 2012; Galiani et al., 2009; Iijima et al., 2001; Pradhan & Rawlings, 2002; Walker, 1999). Studies measuring respiratory infection by self-report used recall periods of at most seven days. Some studies used observation, hospital records, and medical samples to improve the objectivity of the measures (e.g., Khan, 1987), although this remains rare.

Study sample sizes have also increased alongside greater availability of research resources. The median number of clusters is 21 (and the mean 79), whether cluster is defined as community, village, informal settlement, neighbourhood, municipality, school or health care facility. For example, until 2008 the median number of clusters was only 10 (the mean was 49), whereas post-2008 it was 31 (mean of 92); less than a quarter of studies published from 2009 onwards have cluster sample sizes of less than ten.

It is necessary to go beyond “bare bones” in impact evaluations (Mark & Lenz-Watson, 2011) by collecting data to answer relevant questions about implementation and causal mechanisms, not just on effects. As noted above, the collection of data on access to and use of facilities (behaviour change) is well-established (WHO, 1983). Analysis of this causal pathway was done from the earliest WASH health impact evaluations, including trials of hygiene (e.g., Torun, 1983) and water treatment technology (e.g., Kirchhoff et al., 1985). Over half of studies collected data on behavioural outcomes, a share that increased over time particularly for evaluations of sanitation and hygiene technologies (Figure 17).

OPEN IN VIEWER

Figure 17

Measurement of behaviour change

Follow-up length varied by outcome. Studies of direct provision and health messaging, those measuring diarrhoea and ARIs or water treatment and hygiene behaviours, were conducted over relatively shorter periods, with a median number of 12 months each. In contrast, studies of intervention mechanisms such as decentralisation (e.g., CDD, median 24 months) or those measuring socioeconomic outcomes, which may take longer to materialise as they are further down the causal pathway than behaviours and health, tended to be conducted of longer follow-ups (median of 19 months for education outcomes, 30 months for income, and 48 months for labour market outcomes).

Researchers and funders appear to have been sensitive to calls for greater examination of sustainability of interventions and outcomes (e.g., Waddington et al., 2009). For example, evaluations of CLTS, all of which were published since 2012, include studies measuring open defaecation several years after implementation—4 years in the case of Adank et al., 2016, and 10 years for Orgill (2017), which also measured education outcomes. The increased value in longer follow-up periods is well-recognised as a necessary check on slippage (Adank et al., 2016).

However, current standards for reporting of prospective studies leave much room for improvement. The Consolidated Standards of Reporting Trials (CONSORT) standards (Moher et al., 1998) provides basic standards for reporting participant flow in trials, from recruitment to allocation and follow up (Figure 18). Information should be provided for each study group at each stage in the trial, in order for analysis of bias, particularly risk of selection bias.

OPEN IN VIEWER

Figure 18

The CONSORT flow diagram. Source: http://www.consort-statement.org/consort-statement/flow-diagram

However, the basic requirements of reporting participant flow adherence in field trials according to CONSORT standards are frequently unmet. If the reporting in environmental health is substandard, with less than 50% of studies presenting participant flows, the reporting in social sciences may go as far as being deliberately misleading (Figure 19). Only two out of 54 prospective studies in social science presented a participant flow diagram or the data from which it could be fully reconstructed (Beath & Enikolopov, 2013; Guiteras, Jannat, et al., 2015). Some newer studies by social scientists are starting to exhibit flow diagrams for the full trial period, at the cluster level (e.g., Armand et al., 2020).

OPEN IN VIEWER

Figure 19

Participant flow diagrams by academic discipline

Data were also collected on ethical review reported in WASH impact evaluations (Table 7). Again, while standards in environmental health, where over half of studies that would need ethical review, could be improved, the standards in social science leave much to be desired. Only 22% transparently indicated an IRB had approved the evaluation, and even fewer (16%) had done IRB in-country; nothing was indicated about ethical review in 67% of cases. In over 10%, no ethical review procedures were reported. Thus, no study published by a United Nations (UN) body, whether the World Bank, a regional development bank or other UN organisation indicated that an institutional review process was undertaken before study implementation.

It is possible that programme evaluations, which are the studies conducted by UN organisations, are thought not to require ethical approval, as they are being rolled out anyway. For example, Semenza et al. (1998) indicated that “IRB review was not required because the study did not fall under the human subjects regulations” (p. 941) as it was a programme evaluation. That evaluation included a component where participants were randomised to receive chlorine and a safe storage device. In the case of prospective evaluations done by UN bodies including the development banks, there may be ethical issues relating to withholding treatment from control communities, or the ethical standards around, for example, compensating participants for their time (White, 2013), and possibly by offering health treatment to the severely ill, such as oral rehydration salts and health check-ups for severe diarrhoea.

Systematic reviews

The included systematic reviews were critically appraised using the tool developed by Jimenez et al. (2018). Systematic reviews were critically appraised according to the methods of search, analysis, and synthesis, as well as the types of evaluation questions answered and the incorporation of mixed-methods evidence. Table 8 presents a summary of the quality assessments for each included review.

The results showed that only five reviews were assessed as having “minor limitations”, thus meeting the criteria of “high confidence in the review findings” (Dangour et al., 2013; Ejemot-Nwadiaro et al., 2015; Ejere et al., 2012; Rabiu et al., 2012). These reviews were restricted to RCTs and typically included few studies, hence were unable to support strong policy implications. Several reviews of sanitation omit a quasi-experiment of sewage connections in Salvador da Bahia, Brazil (Barreto et al., 2007).

Overall, 13 reviews were assessed as being of “medium confidence”—but having limitations—and the remaining 22 reviews were of “low confidence”—that is, having major limitations. Reviews attaining “medium confidence” generally conducted systematic searches of published and unpublished literature but either used critical appraisal methods which were insufficiently transparent or did not account for some aspects of study design such as unit of analysis errors in cluster designs or lack of adjustment for study dependency (i.e., where evidence is double-counted inappropriately). Studies appraised as being of “low confidence” were largely restricted to published literature, so they are liable to publication bias, inappropriate critical appraisal methods, risking confounding bias, and inappropriate methods of synthesis based on “vote-counting” direction of effects (Hedges & Olkin, 1980).

Unfortunately, it is still fairly common for WASH systematic reviews to be restricted to studies published in peer review journals. These reviews therefore have inbuilt publication bias, defined as the greater likelihood for studies demonstrating positive and/or statistically significant findings to appear in peer review journals. A comprehensive search should cover both published and unpublished papers, and so avoid publication bias by which null, and possibly negative, findings are less likely to be published (e.g., Waddington et al., 2012).

It is also possible to test for, and correct, publication bias statistically, using clever methods that are based on hypothetical relationships between study size and probability of publication, called “small-study effects” (Egger, 1997; Peters et al., 2008). Some meta-analyses have done this (e.g., Clasen et al., 2007b, 2015; Curtis & Cairncross, 2003; Fewtrell & Colford, 2004; Waddington et al., 2009), although most reviews do not. Most recently, Wolf et al. (2018) stated “[t]here was no evidence of funnel-plot asymmetry and small study effects in any of the WaSH meta-analyses” included in that review (p. 519). This is a surprising finding, as publication bias is usually found in all disciplines (e.g., Rothstein et al., 2005). However, examination of their funnel graphs indicates they did not use methods of small-study analysis which take into account other sources of funnel graph asymmetry, such as risk of bias in effect estimation (Peters et al., 2008).

As noted above, the quality of self-reported diarrhoea morbidity has been questioned, particularly when based on repeated measurement (Peterson Zwane et al., 2011; Schmidt & Cairncross, 2009). It appears to be increasingly common for systematic reviews to adjust for lack of blinding using Bayesian methods. Hunter (2009) was the first to propose a bias correction procedure to water treatment studies using bias coefficients from meta-epidemiology findings, presented in Wood et al. (2008). In the updated Cochrane water treatment review by Clasen et al. (2015), similar bias correction factors were also applied, although the authors note that “we urge caution in relying on these adjusted estimates since the basis for the adjustment is from clinical (mainly drug) studies that may not be transferable to field studies of environmental interventions” (p. 9). Freeman et al. (2014) and Wolf et al. (2018) also adjusted for bias due to lack of blinding, including hand hygiene interventions, but not water supply and sanitation, arguing that water supply and sanitation have recognised benefits over and above health impacts, whereas water treatment and hygiene “usually aim exclusively to improve health which is apparent to the recipient” (p. 512). However, the hygiene adjustments lead to the estimated ineffectiveness of hygiene in combating diarrhoea illness, which is not supported by high confidence systematic evidence (Ejemot-Nwadiaro et al., 2015).

There were no overall trends in confidence in the findings over time. As would be expected of standards setting agencies, reviews conducted under the auspices of systematic review standards-setting organisations such as Cochrane and the Campbell Collaboration were nearly always done to higher standards of confidence. These include, for example, avoiding language bias by searching for and incorporating studies in languages other than English, of which relevant studies of WASH interventions are known to exist in Chinese, French, Portuguese and Spanish. However, although stating they are open to incorporating studies in other languages, most systematic reviews, including this evidence map, do not actually search non-English language databases such as Latin American and Caribbean Health Sciences (LILACS) or the China National Knowledge Infrastructure (Fung, 2008; cited in Clasen et al., 2010), or search general resources using non-English language keywords. Clasen et al. (2010) is therefore particularly noteworthy as they searched these databases, identifying seven additional studies not included in any other systematic reviews of sanitation, more than doubling the number included in the review to 12 studies.

Finally, a large proportion of reviews have been conducted without protocols, risking publication bias in the findings of the review or analyses conducted, although there is a trend recently for some new reviews to indicate “protocols are available on request”. Protocols are important, particularly where they undergo peer review, to provide the scope of research, including to avoid duplication of effort, and prevent results-based decision making in analysis. Deviations from protocol are possible, and should be indicated clearly the published review, as required of all studies for Cochrane and the Campbell Collaboration.

On quantitative evidence appraisal and synthesis, reviews tended to score highest on effect size calculation and reporting of heterogeneity—that is, they calculated effect sizes using comparable metrics and statistically analysed between-study differences. Reviews did worse on critical appraisal (using appropriate risk of bias assessment), synthesis methods (including reporting findings by bias categories), and worst on reporting characteristics of included studies (usually due to lack of independent coding by two reviewers).

On the use of mixed methods approaches, including formative evidence from qualitative studies, there seems to have been an evolution in approaches in WASH sector systematic reviewing. Some authors incorporated qualitative evidence contained in the studies eligible for the quantitative review of effects (e.g., Waddington et al., 2009). This approach could be a model for evidence synthesis if the impact evaluations on which reviews of effects are based typically use theory-based approaches, where they report comprehensive information on the intervention (what was provided, to whom, by whom, at what time), and also present outcomes along the causal pathway, including intermediate and endpoint outcomes. However, as this is often not the case (e.g., White, 2009) mixed-methods systematic reviews need to be increasingly inclusive, usually by undertaking additional searches for qualitative studies linked to the included quantitative studies or by conducting full searches for qualitative studies to answer specific review questions, as done by De Buck et al. (2017) and Venkataramanan et al. (2018).

It is at the initial stages of the review process that formal guidance is most lacking on effective mixed methods approaches, especially convening the study team and constructing the conceptual framework to support the integration of qualitative and quantitative evidence. Establishing teams with appropriate qualitative and quantitative skills, preferably drawing on broad academic disciplines, is usually needed for high quality mixed methods reviews to be done efficiently (White, 2018).

Water improvements

The first systematic review of water improvements concluded that “safe excreta disposal and proper use of water for personal and domestic hygiene appear to be more important than drinking water quality in achieving broad health impacts” (Esrey et al., 1991, p. 31). A subsequent meta-analysis by Clasen et al. (2007b, updated in 2015) found water treatment at point-of-use (POU), particularly filtration, was more effective in reducing diarrhoea risk than other types of water improvements, on average by about 40% in LMICs. The findings suggested that other methods of household water disinfection (e.g., chlorination) only reduced diarrhoea infections by one-fifth, if at all, due to biases. Water treatment was found to be more effective when a safe water storage container was also provided (Clasen et al., 2015), as they are for water filters from which drinking water is accessed from a tap. However, reviews have also raised concerns about bias in self-reporting diarrhoea outcomes (Clasen et al., 2015; Waddington et al., 2009), adherence to water treatment technology (Arnold & Colford, 2007; Clasen et al., 2015; Waddington et al., 2009), and consequently lack of sustained health impacts (Waddington et al., 2009). Higher quality studies also found piped water to households significantly reduced diarrhoea (Waddington et al., 2009).

Sanitation improvements

Until the last decade there were few impact evaluations of improving access to sanitation facilities, such as a latrine or sewer connenction, covering a small number of clusters. Replacing on-site sanitation with water-based sewerage was estimated to reduce the incidence of diarrhoea by 30% (Waddington et al., 2009), though it may not always be a suitable solution given the maintenance costs. However, early reviews had not taken clustering into account. The Cochrane review by Clasen et al. (2010) did not conduct meta-analysis because none of the studies at that point had taken clustering of observations into account in calculating standard errors.

The evidence suggests that community-based approaches to participatory psychosocial triggering, and social marketing are more effective than other approaches in promoting sanitation behaviour change (De Buck et al., 2017). CLTS may be effective at reducing open defecation in trials, at least in the short term, but this evidence does not corroborate the wide-spread claims of ending open defecation found in village case studies using qualitative evaluation designs without control groups (Venkataramanan et al., 2018). De Buck et al. (2017) also synthesised evidence from qualitative studies on the implementation of hygiene and sanitation promotion. They concluded that factors such as community involvement in design and implementation, as well as interpersonal communication in communication strategies, were particularly effective when incorporated into programmes.

Hygiene improvements

A review of the effects of handwashing on respiratory illness did not find any studies in LMICs (Rabie & Curtis, 2006). Meta-analyses suggested hand-hygiene interventions reduce reported gastrointestinal illness by between 30% and 50% (Cairncross et al., 2010; Curtis & Cairncross, 2003; Ejemot-Nwadiaro et al., 2015; Waddington et al., 2009). Hygiene education with soap provision appears to be the most effective technology, as soap is more effective in the mechanical removal of pathogens than water alone. Issues affecting the quality of self-reported diarrhoea morbidity also affect hygiene interventions (Ejemot-Nwadiaro et al., 2015). Provision of soap and other water quality interventions may also lead to a small increase in height for children under the age of five (Dangour et al., 2013). There is also evidence that handwashing interventions are protective against diarrhoea for people living with HIV. However, there are no studies of the effects of water supply or sanitation interventions for immunocompromised populations (Peletz et al., 2013).

The evidence suggests that community-based approaches to participatory psychosocial triggering, where a two-way dialogue is established, are particularly effective in promoting handwashing (De Buck et al., 2017).

Characteristics and trends of the evidence base from impact evaluations

WASH impact evaluations have been conducted in 83 LMICs. The distribution of studies is uneven with high concentrations of studies in Bangladesh, India, and Kenya, which have 50 study arms each.

Over the last decade the focus of practitioners, policymakers, and researchers alike has shifted from “what” to provide to “how” to do so, and “where”. This change is reflected in the research covering a broader set of intervention mechanisms. In particular, there has been an increased focus on BCC that uses psychosocial “triggering” and a shift away from direct provision, particularly of latrines. In sanitation, this is most commonly CLTS, which aims to reduce open defecation and increase the use of latrines by leveraging social cohesion to make collective behavioural changes, but can also include information campaigns focused on disgust or being a good parent. Having said this, the traditional approaches of directly providing hardware (e.g., handing out water filters) or health messages (factual information about the consequences for health of not washing ones hands) remain the most common interventions mechanisms even among new studies.

The International Year of Sanitation, in 2008, brought attention to the importance of sanitation technologies, and major donors like Gates Foundation brought resources for doing studies, especially RCTs, in LMICs. Before this, interventions to provide clean drinking water and hand hygiene had been the priority for intervention research and many studies had looked at how to improve water quality. Water treatment is still the most studied WASH technology (over one-quarter of the total), but the percentage of studies looking at sanitation has nearly doubled and there has also been an increase in studies examining hygiene promotion. Up until 2008, only six studies had been conducted on promoting, or providing, latrines; there are now over 50.

“Where”—the place of use dimension—further distinguishes technologies that are physically tied to a geographic location. Traditionally hardware was either provided directly to a household or for communal use by a village, informal community, or neighbourhood. There has, however, been increasing interest in the effect of providing WASH facilities at schools over the last 20 years and the SDGs have highlighted the need to incorporate both schools and health facilities. The vast majority of impact evaluations provide hardware directly to households. Research has also moved with shifting interests and while there were only two rigorous studies specifically examining WASH infrastructure in schools before 2008, there are now 39. The majority of these combine water, sanitation, and hygiene technologies as a comprehensive overhaul for the school. Only one controlled study was found on handwashing supplies in healthcare facilities, and none were found in healthcare facilities for any other kind of technology.

In line with the other changes, there has been a shift in the commonly reported outcomes. While diarrhoeal disease, particularly carer-reported diarrhoeal morbidity among children, remains by far the most commonly reported outcome, there has been a big increase in the number of studies reporting on behavioural outcomes (see Figure 9). This is an important change as the principal argument used by proponents of alternative delivery mechanisms is that they are more effective at changing these behaviours and therefore improving lives. However, interventions which foster marginal improvements in personal WASH behaviour may not cause sufficient changes at community levels to improve quality of life outcomes like child nutrition or diarrhoea mortality.

The most commonly reported on behaviours are handwashing, water treatment and handling, and latrine use. Many of the studies reporting on hygiene behaviour, include measures of personal food hygiene; nearly 50 studies specifically look at handwashing before food preparation, five report on the microbial contamination of food or eating utensils, and 17 report on other food hygiene outcomes, such as whether food is stored properly and dishes washed appropriately. It is important that hygiene studies examine food hygiene outcomes, given the importance of food in faecal-oral disease transmission (Wagner & Lanoix, 1957). There has also been an increase in the reporting of social and economic impacts. This is principally driven by a large increase in the number of studies reporting measures of education and cognitive development, and mainly reflects the increase of studies being conducted in schools.

However, outcomes data collection, whether on behaviour or quality of life, remains heavily reliant on self-reporting, which is highly susceptible to biases, including social desirability (courtesy) bias.

Despite the importance of sustaining the use of hardware and preventing slippage back to open defecation, and other poor practices, only 18 studies measure sustainability of behaviours, defined here as being measured 12 or more months after implementation. Even this is most commonly related to sustained handwashing practices rather than, for example, sustainability of latrine use or community ODF status, which is only measured in three completed latrines studies (Alzua et al., 2015; Crocker et al., 2017 in Ethiopia and Ghana; Orgill, 2017 in India) and in one study of CLTSH in Ethiopia (Delea et al., 2020).

Existing impact evaluations cover rural, urban, and slum populations with the vast majority either targeting or including rural communities (295 studies), which are usually the most disadvantaged areas in terms of WASH service provision. As noted below, however, more studies are needed to address the “for whom” technology aspects among vulnerable groups.

Finally, over half of controlled study designs now use randomised assignment, indicating the extent of support in academic and research funding communities for the approach. Some RCTs are taking full advantage of the power of the methodology by conducting comparative designs with prospective randomised assignment at baseline to alternate intervention delivery mechanisms. Guiteras, Levinsohn, et al. (2015) provide an example in Bangladesh comparing community promotion (CLTS) with subsidies. Other recent examples of comparative designs include Luby et al. (2018) in Bangladesh and Null et al. (2018) in Kenya which compare water, sanitation, hygiene and nutrition support interventions individually and in combination.

A small number of studies are taking advantage of natural experiments using existing data to conduct rigorous, and potentially highly cost-effective, evaluations, such as RDDs which estimate the effects of interventions which are assigned to individuals or communities based on a threshold on a continuous variable, for example a poverty index or an administrative boundary (Spears, 2013; Ziegelhofer, 2012). Methods used to analyse data in natural experimental frameworks include IV estimation (Ziegelhofer, 2012) and difference in differences (Galiani et al., 2005). However, natural experiments applying rigorous methods to existing surveys remain an under-utilised methodological approach in the WASH sector, given the large numbers of existing household survey datasets available that contain questions on access to WASH services which, for example, could be analysed as pseudo-panels using double-difference methods. It is also important to note that there continues to be a great number of uncontrolled studies that simply measure outcomes before and after the intervention. Most of these studies have been excluded from the map as they are not usually able to attribute changes to the intervention. The exception is for some immediate outcomes such as time spent collecting water immediately before and after a new water supply is provided, as noted above.

Characteristics and trends of the evidence base from systematic reviews

A systematic review will be most relevant when the methodology is applied to a clearly defined research question, and preferably where eligible evidence is known about a priori. A common approach used in WASH systematic review and meta-analysis is to ask a question answerable using health impact evaluations; for example, “interventions to improve water quality for preventing diarrhoea” (Clasen et al., 2015). In recent years, there has also been a movement towards reviews covering multiple research questions answerable using different types of evidence, such as “effectiveness and factors influencing implementation of handwashing and sanitation promotion” (De Buck et al., 2017). Broader reviews enable greater statistical precision and systematic analysis of bias, as noted by Gøtzsche (2000): “[a] broad meta-analysis increases power, reduces the risk of erroneous conclusions, and facilitates exploratory analyses which can generate hypotheses for future research” (p. 586).

A related issue is whether to set the question around an outcome—for example, “water, sanitation and hygiene to tackle childhood diarrhoea morbidity in low- and middle-income countries” (Cairncross et al., 2010; Fewtrell & Colford, 2004; Waddington et al., 2009; Wolf et al., 2014, 2018)—or an intervention—“effect of hand-washing on infectious diseases” (Aiello et al., 2008). Some would further delimit by combining the two; for example, “effect of hand-washing on diarrhoea” (Ejemot-Nwadiaro et al., 2015), or perhaps “the effect of improved water supply on women's time use” (a review which remains to be undertaken). But others might argue that hygiene can have a broader range of benefits in fighting respiratory infections (Rabie & Curtis, 2006), and so should not be assessed on its impact on diarrhoea alone.

Another example is the effectiveness of WASH improvements on diarrhoea, which are expected to be bigger when access to water and sanitation facilities in the comparison condition is worse (Fewtrell & Colford, 2004). One area where there does appear agreement is on the splitting of evidence collected under endemic versus epidemic conditions, since the effects of WASH in outbreaks are known to be much larger (e.g., Curtis & Cairncross, 2003; Gundry et al., 2004). This also includes emergency situations, where separate reviews of WASH have been completed (Brown et al., 2012; Yates et al., 2017).

Early reviews of diarrhoea did examine behaviour change—household water treatment in the case of Gundry et al. (2004) and Arnold and Colford (2007), and water treatment, hand hygiene and sanitation use in the case of Waddington et al. (2009). But it is also becoming more common for reviews to focus solely on behaviour change (e.g., Garn et al., 2017; Lucas et al., 2011). What is innovative is, firstly, that reviews are collecting and meta-analysing effect sizes relating to behavioural outcomes along the causal pathway (e.g., Ejemot-Nwadiaro et al., 2015). It seems to be increasingly recognised that causal pathway analysis is highly policy relevant as it enables an understanding of heterogeneneity and therefore the circumstances in which review findings are applicable (e.g., Waddington et al., 2012; White, 2018). Secondly, some reviews are starting to change the focus from WASH technologies to specific intervention mechanisms like social marketing (Evans et al., 2014), sanitation and hygiene behaviour change promotion (De Buck et al., 2017), and CLTS (Venkataramanan et al., 2018). These two latter reviews also used mixed methods to synthesise impact evaluations and qualitative studies.

It is debatable whether analysis of behaviour requires incorporation of qualitative evidence systematically. Impact evaluations and systematic reviews drawing solely on quantitative evidence from impact evaluations are commonly thought to be unable to answer questions about why interventions are successful or not. However, theory-based systematic reviews, which draw on an explicit theory of change (or logic model) and collect evidence on outcomes along the causal pathway, can explain heterogeneity in findings. For example, studies that have explained variation in quality of life outcomes have done so with reference to correlations (or lack therefor) with programme adherence (Waddington et al., 2009; Welch et al., 2017). However, analysis of lower reaches of the causal chain, in particular the “intervention black box” usually requires programme literature and qualitative evidence (White, 2018). Mixed-methods reviews can answer the most pressing questions for policy and practice—design, implementation, participation, targeting, adverse or unintended effects, cost-effectiveness (Waddington et al., 2018). For example, the systematic review by De Buck et al. (2017) covered the full range of relevant behavioural and health outcomes associated with hygiene and sanitation provision, incorporating systematic qualitative evidence on “barriers and enablers” of implementation and adherence.

Equity and vulnerable populations

Impact evaluation research in WASH does pay attention to the specific needs of vulnerable groups, but there is still some way to go before most studies achieve the full integration of gender and other equity concerns, to become a consistently transformative sector (IGWG, 2018).

Globally women and girls carry most of the burden of water collection (including time, calories spent, musculoskeletal injuries, and risk of assault by people or attack by wild animals) and having to use unsafe places to defecate, where water and sanitation services are not accessible. They also experience particular hardships where inadequate services constrain menstrual hygiene management. In the last ten years, new studies have been conducted that evaluate interventions and outcomes that disproportionately affect women and girls; this includes measuring time use (22 studies), psychosocial health outcomes (7 studies), and safety and vulnerability (4 studies), as well as MHM. However, gender analysis is rarely used as part of the framework for understanding programme effects in impact evaluations and systematic reviews. In fact, only a minority of studies included in the map (19% of impact evaluations and 20% of systematic reviews) reported sex disaggregated outcomes. In impact evaluations, psychosocial health (43%), education and cognitive development (40%), open defecation (33%), and time use (26%) were some of the most commonly sex disaggregated outcomes. It is also worth noting that there were no rigorous impact evaluations of sanitation for nonbinary or transgender individuals.

The findings are even more sparse when looking at other categories of vulnerability. One impact evaluation (Abebe et al., 2014) and two systematic reviews (Peletz et al., 2013; Yates et al., 2015) looked at people living with HIV, who often have different social constraints and medical needs. Two impact evaluations (Doocy & Burnham, 2006; Roberts et al., 2001) specifically looked at the needs of those in refugee camps. One impact evaluation (Chase et al., 2002) and two systematic reviews (Brown et al., 2012; Ramesh et al., 2015) examined those living through, or in the aftermath of, a humanitarian crisis. Finally, and most strikingly, no controlled evaluations examined WASH interventions that either targeted people living with a disability or the success of standard WASH interventions in improving outcomes for them.