Improving Energy Access,Climate and Socio-Economic Outcomes Through Off-Grid Electrification Technologies: A Systematic Review

Included Studies

We included 47 rigorous impact evaluations, primarily focusing on Sub-Saharan Africa (SSA). We included experimental and quasi-experimental designs (QEDs), though the majority were assessed as having a high risk of bias. The extent of evidence available across outcomes varied widely, hence for some of them, we were unable to draw conclusive findings.

Effects of Off-Grid Technologies on Energy Access Outcomes

We found that off-grid technology interventions had a significant effect on reducing kerosene consumption (the only traditional fuel for which there was comparable data available) as well as on increasing the use of off-grid technologies, though these findings are based on six and five studies, respectively. We found no statistically significant effect on other energy access outcomes.

Effects of Off-Grid Technologies on Climate Outcomes

We identified five studies, predominantly with high risk of bias, reporting the effect of off-grid technologies on air quality up to 24 months post-intervention, but did not find a statistically significant effect. Only one study analysed the effect of off-grid technologies on CO₂ emissions, so no conclusions could be drawn.

Effects of Off-Grid Technologies on Socio-Economic Outcomes

Off-grid technologies had a small positive effect on income, a finding derived from 13 studies across nine countries. We also found an increase in time spent studying, based on 13 studies, though this result may be overstated due to a risk of publication bias. There was no significant effect on school attendance or test scores. Based on three studies, we found a significant increase in women's decision-making. No significant effects were identified for other socio-economic outcomes.

Main Barriers and Facilitators to Off-Grid Technology Effectiveness

Drawing from qualitative evidence, we identified three key aspects to increasing engagement and effectiveness of programmes: information on how and why technologies should be used, allowing avenues for local involvement in the implementation and design of programmes, and setting appropriate financial models.

Direct Provision

Direct provision encompasses interventions where a technology is provided to an individual, household or community free of charge. There is no limitation on the scale of technologies that may be provided. For instance, this mechanism may be utilised as part of smaller-scale programmes where individuals or households receive individual technologies, such as solar lamps/lanterns (e.g., Kelly 2018; Nyakato et al. 2018) or may be used as part of regional and national programmes where technologies are provided to serve a community as a whole, such as solar mini-grids (e.g., Bensch et al. 2012; Koima 2024). This category covers the free provision of the technology, but not necessarily the financial provision for running said technology.

Opportunity to Access/Market Expansion

As a variation of directly providing technologies to populations, this mechanism encompasses interventions where technologies are brought to new areas/markets. These programmes offer the chance to access new technologies which would not be available in the absence of the intervention. This includes offering populations the opportunity to access new technologies conditional on payment (e.g., Aklin et al. 2017), as well as programmes which introduce technologies to new markets with no further uptake mechanisms (e.g., IDInsight 2018).

Credit and Subsidies

This mechanism encompasses interventions which aim to promote the use and uptake of off-grid technologies by reducing the financial hurdle populations may face in accessing said technologies. Credit includes interventions where participants are provided with a loan or other repayable amount to access a technology (e.g., Wagner et al. 2021), or when participants rent a technology (e.g., Bensch et al. 2015). Subsidies cover interventions where participants are offered the opportunity to purchase technologies at prices lower than market value (e.g., Aevarsdottir et al. 2017).

Information

The last mechanism involves promoting the use and uptake of off-grid technologies by providing information to either off-grid technology suppliers or consumers. Suppliers may be provided with information on how to market products and increase business sales, often through training programmes (e.g., Bensch et al. 2021). On the consumer side, marketing programmes and information awareness campaigns can highlight the benefits of off-grid technologies and encourage their uptake (e.g., Urpelainen and Yoon 2017).

Overcoming Barriers

The direct provision of technologies aims to overcome two of our identified barriers, physical access and financial accessibility. Given the high costs of expanding centralised grid networks to rural areas, off-grid technologies have the benefit of being standalone systems that can produce electricity. Specifically, the establishment of mini-grids can help serve communities, including households and businesses (Kishore et al. 2012). Alternatively, other technologies, such as SHS, are available in multiple sizes, all of which aim to serve a household's energy needs (Bond et al. 2012). Given that these technologies are directly provided to households through this mechanism, they completely bypass the financial barriers to ownership. In the same way that the direct provision of technologies overcomes physical access issues, opportunity to access/market expansion interventions also overcome physical access issues by setting up off-grid technologies in new markets.

Credit and subsidies aim to overcome financial hurdles that may prevent either households or communities from investing in off-grid technologies, or businesses from expanding their off-grid capacity. Subsidies are a common mechanism to help promote off-grid technology ownership at the household level (Kizilcec and Parikh 2020), while community technologies, such as mini-grids, face high startup costs, and so financial support can be required to install these technologies (Shahzad et al. 2023; Peters et al. 2019).

Once off-grid technologies are available and financially accessible to communities, there may still be social barriers that prevent households from accessing said technologies. Social acceptance of new technologies may be limited when there is no clear communication and information on the benefits as well as the limitations of new technologies. If communities gain a negative perception or false expectation of new technologies, they may be less willing to engage with programmes (Feron 2016; Urmee and Md 2016; Ahlborg and Hammar 2014). Information provision seeks to overcome this barrier by ensuring that the benefits from off-grid technologies are made available to intervention communities to encourage households to move away from traditional energy sources.

Pathways to Final Outcomes

Once barriers to the uptake and use of off-grid technologies are overcome, we defined two pathways to improving socio-economic outcomes. First, with an increase in ownership of off-grid technologies comes an increase in energy access. There are different ways in which an increase in energy access can lead to improved socio-economic outcomes. For instance, greater access to light means that students can study for longer, which can lead to improved education outcomes (Banerjee et al. 2021).

The second pathway involves improved climate outcomes through the reduction of traditional energy sources. Once households utilise off-grid technologies, they may move away from traditional energy sources. Traditional energy sources such as biomass and kerosene can increase the level of household pollutants, including greenhouse gases (GHGs; Lam et al. 2012), leading to negative health outcomes for households (Epstein et al. 2013; Kim et al. 2011).

Types of Studies

To address research questions 1 and 2, we included impact evaluations that use an experimental or QED to robustly measure a change in outcomes that is attributed to an intervention as compared to an appropriate counterfactual. We included the following study designs, which are widely used to evaluate intervention effectiveness (Aloe et al. 2017; Reeves et al. 2017; further details of the included study designs are available in Appendix A.1): 1.

Randomised controlled trials (RCTs)

To address research question 3, we searched for additional documentation on the programmes identified for RQs 1–2. We included a range of evidence, such as primary qualitative studies, mixed-methods studies and process evaluations.

Types of Participants

We included studies on the effects of interventions on participants residing in LMICs as defined by the World Bank's country and lending groups (a full breakdown of country classifications is available in Appendix A.2). Studies were classified according to their income status during the first year that an intervention began. Studies that included evidence from multiple countries were included so long as results were provided for at least one LMIC.

Aside from exclusion based on the income status of a country, no further restrictions were applied based on the type of participants.

Types of Interventions

We included interventions that aimed to promote the uptake and use of off-grid technologies. Appendix B provides details on how these interventions were selected from our previous EGM. In brief, after consultations with the review co-funder, Sustainable Energy for All (SEforALL), we agreed to focus on off-grid technologies. As our EGM included interventions on off-grid technologies across different intervention categories, so long as the intervention aimed to increase the uptake and use of one of our included off-grid technologies, it was eligible for inclusion within our review (Table 1).

These technologies, which vary in their use, may be provided as part of programmes for differing purposes. For example, a solar oven may aim to increase health outcomes through a reduction in the use of traditional cookstoves, while solar lighting might hope to increase educational outcomes through an increase in time studying. Despite these varying aims, so long as the technology provided constituted an off-grid technology and the study evaluated one of our eligible outcomes (alongside our other eligibility criteria), it was included in the review.

As the focus of the review is on the promotion of uptake and use of off-grid technologies, some studies from our EGM that focused on renewable energy were not eligible. For instance, we did not include studies that aimed to increase the performance and efficiency of the renewable energy sector regarding the generation of energy (including solar) and the creation of renewable energy technologies (Wu et al. 2023; Luan and Lin 2022). While the development of renewable energy technologies is a prerequisite for the promotion of said technologies, development subsidies are not focused on increasing the use of technologies.

For the purpose of our review, an ‘intervention’ is defined as a decision or set of activities deliberately undertaken by a specific entity—such as a government body, NGO, private firm or consortium—intended to influence events or outcomes, which another such entity could emulate. Studies where the independent variable is not an intervention but a state of affairs were excluded. For instance, a household's access to off-grid electricity may be influenced by an intervention, but when the variation in the independent variable has arisen as a result of uncoordinated activity (i.e., some people purchased off-grid technologies and some people did not), we did not classify this as the result of an implementable intervention. This is true even when a counterfactual study design is used to identify the causal impact of the independent variable, rather than mere associations.

Types of Outcome Measures

We included studies that evaluated energy access, climate and socio-economic development outcomes. The exact outcomes for our review were decided upon after the agreement that our review would focus on off-grid technologies. A full breakdown of this process is outlined in Appendix B. After identifying eligible papers from our EGM based on their intervention, we then mapped the outcomes reported in these studies. From here, and in consultation with SEforALL, we agreed to prioritise 18 includable outcomes in our review (Table 2, see Appendix B for further details on this process).

All included outcomes were deemed as primary outcomes. Energy access outcomes cover measures of a household's access and utilisation of energy. This includes whether there is a secure energy service, how much energy is used and measures of whether there is access to energy through the use and uptake of off-grid technologies.

From our EGM, we knew that few studies focusing on off-grid technologies evaluated climate outcomes. Only air quality and GHG emissions were included under this category.

The EGM covered a broad range of socio-economic development outcomes. For this review, we focused on 10 outcomes. We included one economic measure, income; five measures of time allocation: household/domestic activities, leisure activities, active hours, employment and study hours; two education measures: school attendance and test scores; and one measure of health and women's empowerment.

Duration of Follow-Up

We placed no restrictions on the duration of follow-up within a study.

Types of Settings

Aside from the restrictions on intervention participants having to reside in LMICs, no other eligibility requirement was placed on settings.

Additional Criteria

We included studies in any language, although search terms were developed in English. Both academic and grey literature studies were included. Only studies published since 2000 were included in our review. From 3ie's experience developing other EGMs, less than one per cent of impact evaluations implemented in LMICs predate the Year 2000, hence the likelihood of missing eligible studies is small.

Electronic Searches

We adopted a systematic search strategy following guidelines for systematic literature searching (Kugley et al. 2017). The search strategy was developed in collaboration with the information specialist Alison Bethel. The search was designed to address potential publication bias issues by systematically searching academic bibliographic databases and implementing additional searches for grey literature in specialist organisational websites, websites of bilateral and multilateral agencies and repositories of research in international development. The full list of literature sources searched is available in Table 3, and an example of the search strings used for one database is presented in Appendix C. The precise strings and logic (e.g., index terms and truncation operators) were adapted for each database and platform.

In creating our search terms, we utilised pearl-harvesting techniques as suggested by Sandieson (2006). This technique involved identifying a small sample of known studies relevant to the review. These studies were identified from previous 3ie work in the energy sector (Berretta et al. 2021; Moore et al. 2020) and from 3ie's Development Evidence Portal (DEP). Once these initial studies were identified, as search terms were developed and tested, we ensured that our sample of studies was captured in relevant databases.

Searching Other Resources

To minimise the likelihood of missing studies, we conducted forward and backward citation tracking of studies included in the academic and grey literature search. Backwards citation tracking included checking the reference lists of all included studies for other potentially relevant papers. Forwards citation tracking, conducted through Google Scholar, included identifying studies which had cited our included papers and checking these for relevance. We also published a blog ¹ and contacted key researchers and organisations for additional studies.

To address research question 3, we conducted an additional search for supporting documentation on the programmes identified for research questions 1 and 2. This search was conducted in two steps. Firstly, in the same manner as outlined above, we conducted both backward and forward citation tracking of our included studies. We supplemented this by searching Google using relevant terms related to our included programmes, such as the programme name or specific technologies that may have been deployed.

Description of Methods Used in Primary Research

Based on the eligibility requirements set out above, we anticipated including studies with an appropriate counterfactual, based on an experimental or QED.

Selection of Studies

Our EGM report (Gonzalez Parrao et al. 2024) outlines the full methods used to screen studies. In brief, we began by importing the search results into EPPI-Reviewer (Thomas et al. 2022), the software we used to manage our records. We then removed duplicate records. Once we had completed backwards and forward citation tracking, these records were also imported and de-duplicated.

After de-duplication, we screened records based on their title and abstract. We trained a set of reviewers on batches of records until we reached an 80% inter-coder reliability rating. Once we had reached this mark, reviewers independently screened records based on their title and abstract. At all times, reviewers were able to select an ‘unsure’ code, whereby a member of the core research team would then make an eligibility decision. At this stage, we also utilised EPPI's classifier function, which estimates the probability that a given record will be of relevance based on its title and abstract. We utilised 3ie's DEP classifier model, which has been trained with screening decisions on over 100,000 records. We ultimately excluded any record that scored less than 15% likelihood of inclusion. All other records were screened.

After this stage, we attempted to retrieve the full text of all included records to begin screening based on their full text. Any record for which we could not retrieve the full text was excluded. Two reviewers then independently screened each full text based on our eligibility requirements. In case of disagreements, a third core research team member reviewed the full text and resolved the disagreement.

The above steps outline the process used to select studies for our EGM. For this review, we took further steps to ensure the relevance of the studies. After discussions with SEforALL to define the scope of this review, one core team member listed the relevant studies from the EGM. All studies that did not focus on an off-grid technology and report an outcome of interest were excluded. This stage required no further screening and utilised the data that had been extracted during the EGM.

Data Extraction and Management

The majority of the study characteristics data had already been extracted as part of our EGM. For the data extracted as part of the EGM, trained reviewers extracted data with a second core research team member checking a random subset of this data. Appendix D provides the full data extraction forms. For the extraction of quantitative data to calculate effect sizes, two reviewers extracted all data, and any disagreements were reconciled with a core research team member.

Once all effect sizes were calculated and converted to a standardised mean difference (SMD) (as described in detail below), we examined the data for outliers. We defined outliers as any effect sizes ± 3.29 standard deviations from the mean, following the guidance of Tabachnick and Fidell (2001). Sensitivity to outliers was examined as discussed in the sensitivity analysis section below.

Assessment of Risk of Bias in Included Studies

We assessed the risk of bias of included studies using 3ie's risk of bias tool (see Appendix E). This tool covers both internal validity and statistical conclusion validity of experimental and quasi-experimental impact evaluations (Hombrados and Waddington 2012). Two reviewers undertook the risk of bias assessment, and any disagreements were reconciled through discussion with a core research team member. We conducted the risk of bias assessment for each estimate we extracted, reflecting that estimates of different outcomes within the same study may score differently in the assessment. After assessing each bias dimension, a study was coded as ‘low risk of bias’ if all dimensions were assessed as low; as ‘high risk of bias’ if any of the dimensions were scored as high; and all other estimates were categorised as ‘some concerns’.

Measures of Treatment Effect

An effect size expresses the magnitude (or strength) and direction of the relationship of interest (Valentine et al. 2015; Borenstein et al. 2009). We extracted data from each study to calculate standardised effect sizes for cross-study comparison wherever possible. For continuous outcomes comparing group means in a treatment and control group, we calculated the SMDs, or Cohen's d, its variance and standard error using formulae provided in Borenstein et al. (2009). An SMD is a difference in means between the treatment and control groups divided by the pooled standard deviation of the outcome measure. Cohen's d can be biased in cases where sample sizes are small. Therefore, in all cases, we simply adjusted Cohen's d using Hedges' method.

As primary studies have become increasingly complex, it has become commonplace for authors to extract partial effect sizes (e.g., a regression coefficient adjusted for covariates) in the context of meta-analysis. For studies reporting regression results, we followed the approach suggested by Keef and Roberts (2004) using the regression coefficient and the pooled standard deviation of the outcome. When the pooled standard deviation of the outcome was unavailable, we used regression coefficients and standard errors or t-statistics.

Members of the core research team reviewed all estimates and sample sizes extracted to ensure that the correct formulae were employed in effect size calculations. After synthesis, we converted pooled effect sizes to commonly used metrics such as percentage changes and mean differences in outcome metrics typically used (e.g., weight in kg), whenever feasible.

To our knowledge, effect benchmarks have not yet been developed for off-grid energy programmes. Following the Grading of Recommendations Assessment, Development, and Evaluation guidelines (GRADE; Guyatt et al. 2008), we interpreted the magnitude of effect sizes against the benchmarks listed in Table 4.

Unit of Analysis Issue

Unit of analysis errors can arise when the unit of allocation of treatment is different to the unit of analysis of the effect size estimate, and this is not accounted for in the analysis (e.g., by clustering standard errors at the level of allocation). We assessed studies for unit of analysis errors (The Campbell Collaboration 2019), and when they existed, we corrected them by adjusting the standard errors according to the following formula (Higgins et al. 2020; Waddington et al. 2012; Hedges 2009):

SE ( d ) ′ = SE ( d ) × 1 + ( m – 1 ) c , $\mathrm{SE}(d)^{\prime} =\mathrm{SE}(d)\times \sqrt{1+(m\mbox{--}1)\,c},$

Criteria for Determination of Independent Findings

Complex data structures are a common occurrence in meta-analyses of impact evaluations. There are several scenarios through which these complex structures with dependent effect sizes might occur. For instance, there could be several publications that stem from one study, or several studies based on the same data set. Some studies might have multiple treatment arms that are all compared to a single control group. Other studies may report outcome measurements from several time points or use multiple outcome measures to assess related outcome constructs. All such cases yield a set of statistically dependent effect size estimates (Borenstein et al. 2009).

The research team assessed the extent to which relationships exist across the studies included in the review. When we had several publications reporting on the exact same effect, we used effect sizes from the most recent publication. We utilised information provided in studies to support these assessments, such as sample sizes, programme characteristics and key implementing and/or funding partners.

We extracted effects reported across different outcomes or subgroups within a study, and where information is collected on the same programmes for different outcomes at the same or different periods of time, we extracted information on the full range of outcomes over time. Where studies reported effects from multiple model specifications, we used the author's preferred model specification. If this was not stated or was unclear, we used the effect with the greatest number of controls. Where studies reported multiple outcome sub-groups for the same outcome construct, we calculated a ‘synthetic effect size’ (Borenstein et al. 2009). Where studies reported multiple outcomes or evidence according to sub-groups of participants, we recorded and reported data on relevant sub-groups separately.

Dealing With Missing Data

In cases of relevant missing or incomplete data in studies identified for inclusion, we made every effort to contact study authors to obtain the required information. When we were unable to obtain the necessary data, we reported the characteristics of the study stating that it could not be included in the meta-analysis or reporting of effect sizes due to missing data.

Assessment of Heterogeneity

We assessed heterogeneity by calculating the Q-statistic, I ² and Tau² to provide an estimate of the amount of variability in the distribution of the true effect sizes (Borenstein et al. 2009). We complement this with an assessment of the heterogeneity of effect sizes graphically using forest plots.

Assessment of Reporting Biases

To reduce the possibility of publication bias, we searched for and included unpublished studies in the review. We also tested for the presence of publication bias through the use of contour-enhanced funnel graphs (Peters et al. 2008) and statistical tests (Egger et al. 1997) for outcomes for which we identified at least 10 studies.

Data Synthesis

We conducted meta-analyses of studies that we assessed to be sufficiently similar. We combined studies using meta-analysis when we identified two or more effect sizes using a similar outcome construct and where the comparison group state was judged to be similar across the two, in line with the approach taken by Wilson et al. (2011). We combined studies in the same analysis when they evaluated the same outcome type. Moderator analyses took into account multiple interventions as moderator variables, allowing us to also examine the impact of different intervention types by outcome. Where there were too few studies, or included studies were considered too heterogeneous in terms of interventions or outcomes, we present a discussion of individual effect sizes along the causal chain. As heterogeneity exists in theory due to the variety of interventions and contexts included, we used inverse-variance weighted, random effects meta-analytic models (Higgins et al. 2020). We used the metafor package (Viechtbauer 2010) in R software to conduct the meta-analyses (R Core Team 2020). To facilitate the interpretation of effect sizes in the discussion section, we converted statistically significant findings from the meta-analysis (Cohen's d) into probabilities of superiority. This indicates the probability that a participant picked at random from the treatment group will score higher in the outcome measure compared to a random participant from the control group (Magnusson 2023).

To enrich our findings and to be able to draw out policy recommendations, we integrated our quantitative and qualitative findings. This process involved identifying where our findings from each synthesis overlapped and where they contrasted. This was completed in a qualitative manner with no additional quantitative analysis being conducted based on these overlaps.

Subgroup Analysis and Investigation of Heterogeneity

When feasible, we conducted moderator analyses to investigate sources of heterogeneity. Following the PROGRESS-PLUS approach (Oliver et al. 2017), we assessed moderators falling into three broad categories of extrinsic, methodological and substantive characteristics to address inequity aspects within the off-grid technology context. The final list of moderators was decided in consultation with SEforALL and includes:

Extrinsic characteristics: publication date.

We used random effects meta-regression to investigate the association between moderator variables and heterogeneity of treatment effects (Borenstein et al. 2009) and sub-group analyses to investigate heterogeneity by treatment sub-groups (e.g., men and women, poor and non-poor, and so on). If the latter strategies were not possible (i.e., if we did not have a sufficient number of studies or data), we discussed and explored the factors which may be driving heterogeneity of results narratively by conducting cross-case comparisons (Miles and Huberman 1994).

Sensitivity Analysis

We conducted a sensitivity analysis to assess whether the results of the meta-analysis are sensitive to the removal of any single study. We did this by removing studies from the meta-analysis one by one and assessing changes in results. We also assessed the sensitivity of results to the inclusion of high-risk of bias studies by removing these studies from the meta-analysis and comparing results to the main meta-analysis results. Finally, we assessed sensitivity to outliers by comparing results with and without outliers included.

Treatment of Qualitative Research

We used qualitative research to supplement the findings of the interventions covered by the included studies. We did not seek out all qualitative studies relating to off-grid technologies in LMICs, but under the ‘effectiveness +’ framework (Snilstveit 2012), we searched for qualitative studies to provide additional information about the context and implementation of relevant interventions. This included primary qualitative studies, mixed-methods studies and process evaluations.

One coder independently appraised these studies and documents based on an adapted version of the Critical Appraisal Skills Programmes checklist utilised in previous 3ie work (Snilstveit 2012). We assessed the quality of qualitative and descriptive quantitative studies by appraising the adequacy of reporting, data collection, presentation, analysis and conclusions drawn. Finally, project documents provide information about the design or resources available for a project. As these documents present factual information about interventions, we did not formally appraise the quality of such documents but instead assessed the relevance of the documents against the interventions included in the review.

Summary of Findings and Assessment of the Certainty of the Evidence

We did not plan to include a summary of findings and an assessment of the certainty of the evidence.

Results of the Search

The PRISMA diagram (Figure 2) shows the results of our search to identify studies for research questions 1 and 2. From our EGM's initial search ran between June and September 2023, we identified 144,393 records. After removing duplicates, we screened 94,443 records based on their title and abstract. Ultimately, we included 848 records in our EGM (our EGM report provides greater details about why studies were excluded).

OPEN IN VIEWER

Figure 2

PRISMA flowchart.

From our EGM, we identified 97 records related to the promotion and uptake of off-grid technologies. From these, six records were excluded as they evaluated a technology outside the scope of our review (our excluded studies section below provides examples of these studies). A further 16 records were excluded as they evaluated an outcome outside of our framework, while 2 records were excluded as they were ongoing evaluations. In both cases, we sought to identify a completed evaluation of these studies, but this was not available in either case. Finally, due to misalignment between our EGM and review, we excluded a further two studies as they did not utilise a method from our inclusion criteria. A systematic review allows for a more thorough exploration of studies, and hence, we identified the methodological issues with these studies at this stage.

Ultimately, we included 71 records. Of these, 47 records were categorised as the ‘main’ study, while 24 were deemed ‘linked’ studies. ² From our 47 included records, two programmes were evaluated in multiple papers, this is the PVPA programme in China (Li et al. 2023; Xiao et al. 2023) and the Cooking and Pneumonia Study (CAPS) in Malawi (Nightingale et al. 2019; Kelly 2018). Throughout our review, we present results at the study level and refer to the 47 included studies throughout.

Included Studies

Geography

The map in Figure 3 displays the geographical distribution of the 47 studies included in our analysis. Most of these studies evaluated programmes within SSA (n = 30) and South Asia (n = 13), while only two studies evaluated programmes within Latin America and the Caribbean (LAC) and two in East Asia and the Pacific. Within regions, we also find a concentration of evidence. Within SSA, 23 studies (75%) took place within countries in eastern Africa, while in South Asia, 6 studies (46%) took place in India.

OPEN IN VIEWER

Figure 3

Geographical distribution of studies.

The country with the highest number of studies in our sample is Kenya (n = 8), closely followed by India (n = 6). We also identified multiple studies in Rwanda and Uganda (n = 4), Bangladesh and Nepal (n = 3) and Malawi, Senegal and Zambia (n = 2). In contrast to our EGM, we only identified two studies from China. Despite identifying 249 impact evaluations from China within our EGM, many of these studies focused on the business performance and generation of renewable energies and hence were excluded.

Excluded Studies

Given that our EGM report provides a breakdown of excluded studies based on full text screening (including examples of specific cases), we instead focus on studies that were ‘near misses’ for our review. These studies have been screened out from our review based on the additional criteria we have added since our EGM (further examples are available in Appendix G).

Based on their outcomes, 16 papers were excluded from our review. These studies evaluated outcomes that were relevant to the EGM but excluded from our review's energy access, climate and socio-economic groups. For instance, only two papers evaluated an energy innovations outcome (a category from our EGM), which was not included in the review's scope. These papers focused on the evaluation of government subsidies in China, specifically provided to businesses to increase research and development output (Wu et al. 2023; Jiang et al. 2021). Other papers were ultimately not included due to the outcome prioritisation process outlined in Appendix B. For instance, studies that measured health outcomes but through indicators that were not common among other papers reporting health outcomes, such as nutrition outcomes (e.g., Alaofè et al. 2019) and low birth weight (e.g., Best et al. 2022).

Aside from the outcomes measured, other papers were excluded from our review due to the off-grid technology that they evaluated. While many technologies were common across papers, six studies evaluated niche technologies. Studies on solar dryers (Matavel et al. 2022; Nagwekar et al. 2020), solar suitcases (Rokicki et al. 2019), solar cold storage systems (Takeshima et al. 2021) and solar mosquito systems (Briët et al. 2017; Homan et al. 2016) were excluded.

Quantitative Synthesis

We present the results of the quantitative synthesis organised according to the three main outcome groups of the review: energy access, climate and socio-economic development outcomes. If meta-analysis was not possible due to insufficient data, we present a narrative description of the outcome. Where meta-analysis was feasible, the results for each outcome include a discussion of moderator, sensitivity and publication bias analyses, if applicable.

We assessed 13 moderators to help explain differences in results among studies, including contextual factors (such as the region and country of intervention) and key features of the intervention (such as technology type, level of implementation, intervention mechanism and scale, target actor and population, supply/demand focus and implementing agency), as well as study characteristics (overall risk of bias, study design and year of publication). Complementary information, including funnel plots and tables of these analyses, is provided in Appendix I.

As a summary of all analyses conducted, Table 8 displays the meta-analysis and publication bias results across outcome categories covered in subsequent sections. Due to a limited number of studies, we were only able to assess publication bias for nine of the outcomes included in this review. Among these, evidence of publication bias was found in just one outcome—time spent studying, applicable to both children and adults—which reduces the confidence in the reliability and generalisability of the pooled effect for this outcome.

Energy Access Group

The energy access outcome group covers measures of a household's access and utilisation of energy. This includes measures of use and uptake of off-grid technologies and energy in general, energy security and expenditure on different energy sources.

Energy Consumption

To assess the effects of off-grid energy interventions on energy consumption, we considered the estimates reported for lighting hours and kerosene consumption. We found a significant reduction in kerosene consumption but no effect on lighting hours, as detailed below.

Hours of Lighting Use

We found 12 studies evaluating the effects of off-grid interventions on the number of hours per day a household or individual uses lighting. Off-grid electricity technologies did not change the hours of lighting use. The overall average weighted effect was small and positive but not statistically significant (

SMD ^ $\widehat{\mathrm{SMD}}$

 = 0.16; 95% CI: −0.08 to 0.40; p = 0.187). The heterogeneity among the studies was substantial (I ² = 95.54%). We rated all but two of the included effect sizes as having a high risk of bias (Figure 5).

OPEN IN VIEWER

Figure 5

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on hours of lighting use.

While one study reported a higher estimate than that of the other studies (Ballón et al. 2019), removing this study would reduce the overall average effect but remain within the confidence interval of the main effect, suggesting that this study is not overly influential (

SMD ^ = 0.08 , p = 0.34 $\widehat{\mathrm{SMD}}=0.08,\,p=0.34$

).

None of the moderator variables significantly contributed to the variation in estimates. The results from all but two of the included studies for this analysis were rated as having a high risk of bias. The average effect between the two studies with some concerns or low risk of bias was much larger, but, because of the substantial level of heterogeneity between both studies, the effect on the use of hours of light was still not statistically different from zero (

SMD ^ Low or med ROB = 0.60 ; 95 % CI: − 0.15 to 1.35 ; p = 0.12 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{ROB}}=0.60;\,95 \% \,\mathrm{CI:}-0.15\,\mathrm{to}\,1.35;\,p=0.12$

).

Kerosene Consumption

We found six studies measuring the effects of off-grid interventions on litres of kerosene consumed or purchased. Removing barriers to off-grid electricity technologies through direct provision, market expansion, financing opportunities or information campaigns reduces kerosene consumption. The overall weighted effect was large, negative and statistically significant (

SMD ^ = − 0.57 ; 95 % CI: − 0.92 to − 0.23 ; p = 0.001 $\widehat{\mathrm{SMD}}=-0.57{\rm{;}}\,95 \% \mathrm{CI:}\,-0.92\,\mathrm{to}\,-0.23;\,p=0.001$

). All the effect sizes in this analysis were negative and varied between −1.29 and −0.14 standard deviations. However, the heterogeneity among the studies was substantial (

I 2 = 98.91 $I^2=98.91$

%). Further, the effects from all included studies but one were rated as having a high risk of bias, and the effect size from the only study rated as having some concerns was not statistically different from zero (Figure 6).

OPEN IN VIEWER

Figure 6

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on kerosene consumption.

None of the studies were overly influential or identified as an outlier, so results did not change when removing one study at a time. There was no difference in the effect sizes based on the type of technology, the scale of the intervention or its supply/demand focus. However, moderator analysis suggested that the effect was on average smaller in SSA interventions compared to South Asia (

p < 0.01 $p\lt 0.01$

), but greater for interventions implemented by a government agency with or without for-profit firms compared to other implementation agencies (

p < 0.01 $p\lt 0.01$

). Also, adding an informational component to financial opportunities for off-grid technologies further reduced kerosene consumption when compared to interventions providing only financial support (

p < 0.01 $p\lt 0.01$

). Other factors could not be tested as there was not enough variation.

Energy Expenditure

A total of 18 studies reported 36 estimates of the effect on aggregated energy expenditure. For six studies reporting multiple estimates for different populations, treatment arms, follow-up surveys or energy sources, we computed a composite measure to be used as an independent effect in the analysis. The random effects model suggested a reduction in energy expenditure. The overall average effect was small but not statistically significant (

SMD ^ = 0.15 ; 95 % CI: − 0.03 to 0.32 ; p = 0.10 $\widehat{\mathrm{SMD}}=0.15{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0.03\,\mathrm{to}\,0.32;\,p=0.10$

) and there was a large and significant level of heterogeneity among studies (

I 2 = 96.73 $I^2=96.73$

%). We rated all but four of the included effect sizes as having a high risk of bias (Figure 7).

OPEN IN VIEWER

Figure 7

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on energy expenditure. We reversed the signs of effects on energy expenditure as we consider a reduction in this outcome to be positive.

We identified two potential outliers in this analysis (Chen et al. 2017; Mahajan et al. 2020). Removing the study by Chen et al. (2017) would increase the overall average effect and make it statistically significant (

SMD ^ = 0 . 20 ; p = 0.01 $\widehat{\mathrm{SMD}}=0.20;\,p=0.01$

). On the other hand, removing Mahajan et al. (2020) would reduce the overall average effect, but it would still not be significant (

SMD ^ = 0 . 08 ; p = 0.04 $\widehat{\mathrm{SMD}}=0.08;\,p=0.04$

). However, no study could be considered overly influential, as the average effect from leaving each of these out one at a time still falls under the confidence interval of the main effect. Therefore, for moderator analysis, we included all studies.

Most of the studies in this analysis came from SSA (

k = 12 $k=12$

). Compared to these, the effects from LAC (

k = 2 $k=2$

) were not statistically different, but the studies from South Asia (

k = 4 $k=4$

) reported larger reductions in expenditure on average (p = 0.02). By country, the studies from Uganda reported a smaller reduction in expenditure on average (p < 0.01). Studies using a QED report larger increases in energy expenditure compared to experimental studies (p = 0.03). Other significant moderators were the type of technology and the intervention mechanism. The effects from studies evaluating SHS were significantly different compared to the effects for all other technologies (p < 0.01), but the average effect was not statistically significant (

SMD ^ SHS = − 0.24 ; 95 % CI: − 0.57 to 0.09 ; p = 0.15 ; k = 6 ${\widehat{\mathrm{SMD}}}_{\mathrm{SHS}}=-0.24{\rm{;}}\,95 \% \,\mathrm{CI:}-0.57\,\mathrm{to}\,0.09;\,p=0.15;\,k=6$

). On the other hand, solar lamp/lanterns with mobile phone chargers had a significantly larger effect compared to other technologies (p = 0.02), and the average effect was large and statistically significant (

SMD ^ lamp + charger = 0.54 ; 95 % CI: 0.21 to 0.87 ; p < 0.01 ; k = 5 ${\widehat{\mathrm{SMD}}}_{\mathrm{lamp}+\mathrm{charger}}=0.54{\rm{;}}\,95 \% \,\mathrm{CI:}\,0.21\,\mathrm{to}\,0.87;\,p\lt 0.01;\,k=5$

). The studies based on financial mechanisms showed a significantly different effect on energy expenditure compared to interventions that directly provided an energy technology (p = 0.04), but again, the average effect of providing credit or subsidies was not statistically significant (

SMD ^ financial = − 0.09 ; 95 % CI: − 0.35 to 0.17 ; p = 0.51 ; k = 7 ${\widehat{\mathrm{SMD}}}_{\mathrm{financial}}=-0.09{\rm{;}}\,95 \% \,\mathrm{CI:}-0.35\,\mathrm{to}\,0.17;\,p=0.51;\,k=7$

). Further, the effects of interventions implemented by for-profit firms were significantly different than those of all other types of implementation agencies (p = 0.02), but the average effect was also not statistically significant (

SMD ^ For − profit = − 0.14 ; 95 % CI: − 0.44 to 0.16 ; p = 0.37 ; k = 7 ${\widehat{\mathrm{SMD}}}_{\mathrm{For}-\mathrm{profit}}=-0.14{\rm{;}}\,95 \% \mathrm{CI:}-0.44\,\mathrm{to}\,0.16;\,p=0.37;\,k=7$

). The technology level, scale, target actor, target population, year of publications and risk of bias rating were not significant moderators of the effect on energy expenditure.

Only four studies were not rated as having a high risk of bias, pooling the effect sizes of these effects resulted in a larger but much less precise and, therefore, still not statistically significant average effect on energy expenditure (

SMD ^ Low or med ROB = 0 . 33 ; 95 % CI: − 0 . 11 to 0 . 77 ; p = 0 . 10 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{ROB}}=0\text{.}33{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}11\,\mathrm{to}\,0\text{.}77;\,p=0\text{.}10$

).

Technology Uptake or Upgrade

We synthesise the effects of off-grid interventions, including measures of the uptake of a new technology or the replacement/upgrade of an old technology, using two different comparison groups. We explored measures related to technology use in the next section.

Technology Uptake or Upgrade Compared to No Intervention

We included two studies evaluating the effects on measures of purchase or adoption of an off-grid technology compared to no intervention (i.e., business as usual). The overall average effect from these two studies was small, positive and statistically significant, suggesting that interventions increased the technology uptake or upgrade (

SMD ^ = 0.18 ; 95 % CI: 0.11 to 0.24 ; p < 0.01 $\widehat{\mathrm{SMD}}=0.18{\rm{;}}\,95 \% \,\mathrm{CI:}\,0.11\,\mathrm{to}\,0.24;\,p\lt 0.01$

). There was no level of heterogeneity in this analysis (

I 2 = 0.0 $I^2=0.0$

%). The effect size from one study was rated as having a high risk of bias, while the other effect size was rated as having a low risk of bias (Figure 8).

OPEN IN VIEWER

Figure 8

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on technology uptake or upgrade, compared to no intervention.

Both interventions included in this analysis were implemented in South Asia, looked at the effects of SHS from the demand-side, targeting households. Other intervention characteristics varied between both studies but the small sample did not allow for sensitivity or moderator analysis.

Technology Uptake or Upgrade Compared to Lower Price Paid/Higher Subsidy

We identified three studies evaluating the effects on purchase or adoption of an off-grid technology comparing different prices paid or subsidy levels. The overall average effect was large, negative and statistically significant (

SMD ^ = − 0 . 52 ; 95 % CI: − 0 . 76 to − 0 . 29 ; p < 0 . 01 $\widehat{\mathrm{SMD}}=-0\text{.}52{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}76\,\mathrm{to}\,-0\text{.}29;p\lt 0\text{.}01$

) and there was substantial heterogeneity in the effects used for this analysis (

I 2 = 92.70 % $I^2=92.70 \%$

). We found a significantly lower adoption of technologies when compared to technologies offered with higher subsidies or lower prices paid (Figure 9). These results should be taken with caution given the small sample size used in the analysis and the high risk of bias identified in two of the three included effect sizes.

OPEN IN VIEWER

Figure 9

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on technology uptake or upgrade, compared to lower price paid/higher subsidy.

All the studies included in this analysis were implemented in SSA, looked at the effects of solar lamps/lanterns from the demand-side, households or individuals. Other intervention characteristics varied between studies but the small sample did not allow for moderator analysis. One study (Mekonnen et al. 2021) may be a potential outlier in the context of this model. However, none of the studies could be considered overly influential and removing Mekonnen et al. did not change the overall results (

SMD ^ = − 0.40 , p < 0.01 $\widehat{\mathrm{SMD}}=-0.40,\,p\lt 0.01$

).

Technology Usage

We explored the effects of off-grid interventions on the usage of a technology after it has been received or bought. This includes usage measures of alternative technologies, for example, those that the intervention aimed to replace. We found a small but positive effect for usage of any technology, but a non-statistically significant result for usage of lights.

Technology Use

We included five independent effect sizes from four studies in the analysis of the effects on technology usage, including electricity, LED light, solar light and solar stove. Off-grid interventions increased the household use of these technologies. The overall average effect based on the random-effects model was large, positive and statistically significant (

SMD ^ = 0.61 ; 95 % CI: 0 . 13 to 1.09 ; p = 0.01 $\widehat{\mathrm{SMD}}=0.61{\rm{;}}\,95 \% \,\mathrm{CI:}0.13\,\mathrm{to}\,1.09;\,p=0.01$

). There was substantial heterogeneity in the effects used for this analysis (

I 2 = 98.28 % $I^2=98.28 \%$

) and all were rated as having a high risk of bias (Figure 10).

OPEN IN VIEWER

Figure 10

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on the usage of technologies. Two independent effect sizes from the same study were included in the analysis (Clarke et al. 2020). These came from different field experiments: one randomly varied the price of a solar light through subsidies and the other one provided the light for free and randomly assigned recharge coupons with reduced fees.

Our main result was driven by one potential outlier in the context of this model (Michler et al. 2021), which measured the effect on the use of solar stoves. According to the Cook's distances, this study could be considered overly influential; removing it would reduce the magnitude of the average effect and it would no longer be statistically significant (

SMD ^ = 0.20 , p = 0.25 $\widehat{\mathrm{SMD}}=0.20,\,p=0.25$

). Further, all the effects from studies in this analysis were considered of high risk of bias.

There was insufficient variation in these studies to test most of the factors identified as potential moderators to explain differences in results. All interventions had a demand focus targeting households, and most were implemented in SSA at a regional level. The implementation agency was not a significant predictor of the effect, nor was the study's year of publication. When looking at the intervention mechanisms, however, direct provision was associated with a larger effect compared to other mechanisms (

p = 0.02 $p=0.02$

).

Technology Use (Light Only)

We excluded the study looking at electricity usage to focus on the effects of the use of light, LED or solar alone. The three effects for this analysis were all positive and ranged from 0.02 to 0.67 standard deviations. However, since two of the effects were not statistically significant, nor was the average effect size from the random-effects model. The pooled effect on light usage was moderate and positive but not statistically different from zero (

SMD ^ = 0.24 ; 95 % CI: − 0.13 to 0.61 ; p = 0.20 $\widehat{\mathrm{SMD}}=0.24;\,95 \% \,\mathrm{CI:}\,-0.13\,\mathrm{to}\,0.61;\,p=0.20$

). Heterogeneity for this analysis was still substantial (

I 2 = 97.59 % $I^2=97.59 \%$

) and we rated all the included effect sizes as having a high risk of bias (Figure 11).

OPEN IN VIEWER

Figure 11

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on the usage of lighting technologies. Two independent effect sizes from the same study were included in the analysis (Clarke et al. 2020). These came from different field experiments: one randomly varied the price of a solar light through subsidies and the other one provided the light for free and randomly assigned recharge coupons with reduced fees.

When we focused on the use of light only, the study by Rom et al. (2023) may be considered a potential outlier. Removing this study reduced the magnitude of the overall average effect (

SMD ^ = 0.03 , p = 0.26 $\widehat{\mathrm{SMD}}=0.03,\,p=0.26$

), which was still statistically insignificant and came from a single high risk of bias study.

Energy Security

We defined energy security as the uninterrupted availability of energy sources at an affordable price. Includable measures of energy security were individual indicators of energy access, quality and reliability. We found mixed results, with a small but positive and significant effect for energy access.

Access

Measures of energy access reported by the three studies in this analysis include household access to electricity, household access to energy for lighting and household connection to the grid. The observed effect sizes were positive and ranged from 0.063 to 0.387. The estimated average effect based on the random-effects model was small, positive and statistically significant (

SMD ^ = 0 . 20 ; 95 % CI: 0 . 01 to 0 . 38 ; p = 0 . 04 $\widehat{\mathrm{SMD}}=0\text{.}20;\,95 \% \,\mathrm{CI:}\,0\text{.}01\,\mathrm{to}0\text{.}38;\,p=0\text{.}04$

). The true effect appears to be heterogeneous (

I 2 = 85.06 % $I^2=85.06 \%$

). Hence, although the average effect is estimated to be positive, in some studies the true effect may be negative (Figure 12). Further, we rated all the effect sizes in this analysis as having a high risk of bias.

OPEN IN VIEWER

Figure 12

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on energy access.

The positive result from this analysis should also be taken with caution due to the small sample size used in the model, and all effects being assessed as having a high risk of bias. There was not enough variation in these studies to allow for moderator analyses. However, removing the study by Bahaj et al. (2019), which may be a potential outlier, would reduce the overall average effect, but the effect would still be positive and statistically significant (

SMD ^ = 0.10 , p = 0.01 $\widehat{\mathrm{SMD}}=0.10,\,p=0.01$

).

Reliability

Two studies reported outcomes on energy reliability using as measures the number of hours a day that electricity is usually available for the household and the probability of lighting interruption. The pooled effect was moderate and positive (improved reliability) but not statistically significant (

SMD ^ = 0.29 ; 95 % CI: − 0.16 to 0 . 73 ; p = 0.21 $\widehat{\mathrm{SMD}}=0.29;\,95 \% \,\mathrm{CI:}\,-0.16\,\mathrm{to}\,0.73;\,p=0.21$

). Even with only two studies, the level of heterogeneity in this analysis was substantial (

I 2 = 96.97 % $I^2=96.97 \%$

). One of the estimates in this analysis was rated as having a low risk of bias and the other as having a high risk of bias (Figure 13). The small sample size used in the model did not allow for sensitivity or moderator analysis.

OPEN IN VIEWER

Figure 13

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on energy reliability.

Affordability

Two studies reported insignificant effects on energy affordability using measures of the average successive days with zero balance before top-up and an indicator for borrowing for solar energy. The pooled effect was very small and not statistically significant (

SMD ^ = − 0 . 01 ; 95 % CI: − 0 . 10 to 0 . 08 ; p = 0 . 83 $\widehat{\mathrm{SMD}}=-0\text{.}01;\,95 \% \,\mathrm{CI:}-0\text{.}10\,\mathrm{to}\,0\text{.}08;\,p=0\text{.}83$

). There was no heterogeneity in this analysis (

I 2 = 0.00 % $I^2=0.00 \%$

). The small sample size used in the model did not allow for sensitivity or moderator analysis, and we rated the effect sizes from both studies as having a high risk of bias (Figure 14).

OPEN IN VIEWER

Figure 14

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on energy affordability.

Climate Group

Among climate outcomes relevant to this review, we included measures of air quality and GHG emissions. Few studies evaluated a climate outcome, and we did not find statistically significant results.

Air Quality

We found five studies exploring the effects of off-grid technologies on indoor air pollution using particulate matter of size 2.5 micrometres in diameter or smaller in the air (PM2.5). There is no consistent evidence of an increase or reduction in air quality associated with off-grid technologies. The average effect size was not statistically different from zero (

SMD ^ = 0 . 16 ; 95 % CI: − 0 . 01 to 0 . 32 ; p = 0 . 07 $\widehat{\mathrm{SMD}}=0\text{.}16{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}01\,\mathrm{to}\,0\text{.}32;\,p=0\text{.}07$

) and the heterogeneity level was large and significant (

I 2 = 80.78 % $I^2=80.78 \%$

). This result should be taken with caution as all effects used in the model were rated as having a high risk of bias (Figure 15).

OPEN IN VIEWER

Figure 15

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on air quality. We reversed the signs of effects on air pollution measures as we consider a reduction in this outcome to be positive and a reflection of improved air quality.

The effects from the two studies evaluating solar improved cookstoves were not statistically different from the effects found for solar lamps/lanterns with or without phone chargers in three studies (

p = 0.16 $p=0.16$

). The intervention mechanism did not help explain differences in the results either, as effects from direct provision in three studies did not differ from other mechanisms in the remaining studies (

p = 0.66 $p=0.66$

). There was not enough variation in other factors to allow for moderator analysis. All studies came from SSA, were implemented at the household level with a demand focus and were rated as having a high risk of bias. Most studies did not target a specific population and did not specify the implementation agency. Finally, none of the studies were considered potential outliers, and leaving out one study at a time did not change the results.

GHG Emissions

We identified one study in the outcome category GHG emissions through measures of carbon related emissions (CO₂) and non-carbon related emissions, for example, methane (CH₄), nitrous-oxide (N₂O) and fluorinated gases (Rom et al. 2023). The study was implemented in SSA and evaluated the effect of owning a solar lamp on CO₂ emissions. We rated this study as having a high risk of bias.

Socio-Economic Development Group

The group of socio-economic development outcomes includes measures of income, time allocation (for remunerated and domestic work, leisure, resting or sleeping, and studying), education (school attendance and test scores), health (respiratory illness) and women's empowerment.

Income

We found 13 studies reporting estimates of the effect of off-grid technologies on measures of a household or individual's income. Off-grid technologies have the potential to increase a household's income. The estimated average effect based on the random-effects model was positive and statistically significant but very small (

SMD ^ = 0.06 ; 95 % CI: 0.02 to 0.10 ; p = 0 . 01 $\widehat{\mathrm{SMD}}=0.06{\rm{;}}95 \% \,\mathrm{CI:}\,0.02\,\mathrm{to}\,0.10;\,p=0\text{.}01$

). The level of heterogeneity in this analysis was moderate (

I 2 = 55.65 % $I^2=55.65 \%$

), and all but a couple of the estimates were assessed as having a high risk of bias (Figure 16).

OPEN IN VIEWER

Figure 16

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on income.

There was no indication of outliers in the context of this model, and none of the studies were considered to be overly influential. Results did not vary by geographic region or country, nor did they differ based on the studies' year of publication. The intervention design features, such as technology type, mechanism, target population and scale, among others, were also not useful in explaining the heterogeneity level in this analysis. If we focus our analysis exclusively on estimates without a high risk of bias, two studies remain (Arraiz and Calero 2015 and Xiao et al. 2023). The overall weighted average effect decreases and is no longer statistically significant (

SMD ^ Low or med ROB = 0 . 05 , 95 % CI: − 0 . 07 to 0 . 18 ; p = 0 . 42 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{ROB}}=0\text{.}05,\,95 \% \,\mathrm{CI:}\,-0\text{.}07\,\mathrm{to}\,0\text{.}18;{\ }p=0\text{.}42$

), but this result should be interpreted with caution given the small number of estimates contributing to this analysis.

Allocation of Time

In this section, we synthesise quantitative effects for all outcomes measured in time units. Other than an increase in the time spent studying, there was no evidence of an effect of off-grid technologies on the time spent in other activities, including remunerated or domestic work, leisure, or resting.

Time Spent Working (Remunerated)

We included a total of 14 studies reporting the effects of off-grid technologies on the time spent by individuals or households on income-generating activities. ⁵ We found no effect on time spent on remunerated work. The overall average effect based on the random-effects model was very small and not statistically different from zero (

SMD ^ = 0.01 ; 95 % CI: − 0.03 to 0.06 ; p = 0 . 58 $\widehat{\mathrm{SMD}}=0.01{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0.03\,\mathrm{to}\,0.06;\,p=0\text{.}58$

). Estimates used in this analysis were moderately heterogeneous (

I 2 = 60.59 % $I^2=60.59 \%$

), and again, most estimates were rated as having a high risk of bias (Figure 17).

OPEN IN VIEWER

Figure 17

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent working.

The type of off-grid technology was the only moderator that helped explain differences in the results among studies. Effects from solar lamps with mobile phone chargers were significantly larger than the effects from other technologies (

p < 0 . 01 $p\lt 0.01$

). Solar lamps/lanterns with mobile charges increased the time spent working (

SMD ^ lamp + charger = 0.20 ; 95 % CI: 0.03 to 0.38 ; p = 0 . 02 ; k = 3 ${\widehat{\mathrm{SMD}}}_{\mathrm{lamp}+\mathrm{charger}}=0.20{\rm{;}}\,95 \% \,\mathrm{CI:}\,0.03\,\mathrm{to}\,0.38;p=0\text{.}02;\,k=3$

). There were no significant differences in the results based on the studies' risk of bias rating. Removing the eight studies rated as having a high risk of bias, the effect of off-grid technologies on working hours was still not statistically different from zero (

SMD ^ Low or med BOB = 0.02 ; 95 % CI: − 0.05 to 0.09 ; p = 0.62 ; k = 6 ${\widehat{\mathrm{SMD}}}_{{}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}}=0.02;\,95 \% \,\mathrm{CI:}\,-0.05\,\mathrm{to}0.09;\,p=0.62;\,k=6$

).

None of the studies could be considered an outlier, nor did results change when we left one study at a time.

Time Spent on Domestic Work

Our quantitative synthesis is based on the effects of the overall time spent on domestic work reported in 14 studies, but we also identified studies reporting the effects on specific activities such as cooking (

k = 4 $k=4$

), caretaking (

k = 2 $k=2$

), and collecting fuel (

k = 3 $k=3$

). The overall pooled effect on domestic work based on the random-effects model was very small and not statistically significant (

SMD ^ = 0.02 ; 95 % CI: − 0.03 to 0 . 06 ; p = 0.46 $\widehat{\mathrm{SMD}}=0.02;\,95 \% \,\mathrm{CI:}-0.03\,\mathrm{to}\,0.06;\,p=0.46$

). The level of heterogeneity in this analysis was moderate (

I 2 = 66.79 % $I^2=66.79 \%$

), and only five studies were not rated as having a high risk of bias (Figure 18).

OPEN IN VIEWER

Figure 18

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent on domestic work. We reversed the sign for time spent on domestic work as we consider the reduction in this measure to be positive.

One study (Khandker et al. 2014) may be a potential outlier in the context of this model and could be considered overly influential. Removing this study did not change the overall result of no effect on the time spent on domestic work, as the average effect size was still very small and not statistically significant (

SMD ^ = − 0.001 , p = 0.96 $\widehat{\mathrm{SMD}}=-0.001,\,p=0.96$

).

Studies with a focus on the demand side (e.g., donating solar lamps with a mobile charger evaluated in Bensch et al. 2015) showed statistically less time spent on domestic work when compared to interventions focused on both supply and demand (e.g., training and awareness-raising activities for staff, the POs and customers in addition to subsidised prices and access to credit evaluated by Khandker et al. 2014) (

p < 0 . 01 $p\lt 0\text{.}01$

). We found no statistical difference in results based on geographic region or country. Technology type, implementation mechanism and intervention scale did not help explain differences in results either. Study characteristics such as the year of publication and the risk of bias rating were not significant moderators either. Removing all effect sizes rated as having a high risk of bias does not change the overall result of no effect on the time spent on domestic work as the result is still not statistically different from zero (

SMD ^ Low or med ROB = 0 . 01 , 95 % CI: − 0 . 04 to 0 . 07 ; p = 0 . 63 ; k = 5 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{ROB}}=0\text{.}01,\,95 \% \,\mathrm{CI:}\,-0\text{.}04\,\mathrm{to}\,0\text{.}07;{\ }p=0\text{.}63;\,k=5$

).

Time Spent on Leisure

Our quantitative synthesis is based on the effects for the overall time spent on leisure reported in 13 studies, but we also identified studies reporting the effects on specific activities such as watching TV (

k = 2 $k=2$

), radio listening (

k = 3 $k=3$

), reading (

k = 2 $k=2$

) and playing (

k = 2 $k=2$

). On average, interventions did not have a significant effect on time spent on leisure (e.g., socialising, praying, playing, reading or radio listening). The overall average effect based on the random-effects model was very small and not statistically different from zero (

SMD ^ = 0 . 03 ; 95 % CI: − 0 . 03 to 0 . 09 ; p = 0 . 36 $\widehat{\mathrm{SMD}}=0\text{.}03{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}03\,\mathrm{to}\,0\text{.}09;\,p=0\text{.}36$

). The level of heterogeneity in this analysis was moderate (

I 2 = 66.95 % $I^2=66.95 \%$

), and we rated most effects as having a high risk of bias (Figure 19).

OPEN IN VIEWER

Figure 19

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent on leisure.

One study (Wagner et al. 2021) may be considered a potential outlier. Removing this study increased the overall average effect on leisure time and made it marginally significant (

SMD ^ = 0.05 , p = 0 . 06 $\widehat{\mathrm{SMD}}=0.05,p=0\text{.}06$

). Compared to national programmes, regional programmes showed a larger positive effect (

p < 0 . 01 $p\lt 0.01$

). Pooling the effects of the six studies with a regional scale indicated that off-grid technologies implemented in a region significantly increase the time spent on leisure (

SMD ^ Regional = 0.11 ; 95 % CI: 0.05 to 0.16 ; p < 0.01 ${\widehat{\mathrm{SMD}}}_{\mathrm{Regional}}=0.11{\rm{;}}\,95 \% \,\mathrm{CI:}0.05\,\mathrm{to}\,0.16;\,p\lt 0.01$

) and there was no heterogeneity for this analysis. Other contextual and intervention design characteristics did not help explain differences in the results. Compared to effects rated as having some concerns (Arraiz and Calero 2015; Bensch et al. 2015) or low risk of bias (Mahajan et al. 2020), the effects rated as having high risk of bias were smaller (

p = 0 . 04 $p=0\text{.}04$

). The overall average effect excluding high-risk of bias studies was small, positive and statistically significant (

SMD ^ Low or med BOB = 0 . 14 ; 95 % CI: 0 . 06 to 0 . 23 ; p < 0 . 01 ; k = 3 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}=0\text{.}14;\,95 \% \,\mathrm{CI:}\,0\text{.}06\,\mathrm{to}\,0\text{.}23;\,p\lt 0\text{.}01;\,k=3$

), and there was no heterogeneity in this analysis either. However, these results should be considered with caution given the small sample size used for it.

Time Spent Resting/Sleeping

We included six studies in the analysis of the effect of off-grid technologies on resting or sleeping and found a null overall effect. The overall average effect based on the random-effects model was very small and not statistically different from zero (

SMD ^ = − 0 . 03 ; 95 % CI: − 0 . 13 to 0 . 06 ; p = 0 . 47 $\widehat{\mathrm{SMD}}=-0\text{.}03{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}13\,\mathrm{to}\,0\text{.}06;\,p=0\text{.}47$

). The level of heterogeneity in this analysis was moderate (

I 2 = 69.42 % $I^2=69.42 \%$

), and all but one of the included studies were rated as having a high risk of bias (Figure 20).

OPEN IN VIEWER

Figure 20

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent resting or sleeping.

There was no indication of outliers influencing the overall average effect. However, significant results from moderator analysis are based on two groups of studies. On one side, four studies implemented in SSA by a non-government agency used an experimental design. The other two studies evaluated interventions implemented by the government, used a QED and were not in SSA. The effects from quasi-experimental studies were on average smaller than those from experimental studies in SSA, however, the pooled effect was still not statistically significant (

SMD ^ QED = − 0 . 14 ; 95 % CI: − 0 . 31 to 0 . 03 ; p = 0 . 10 ; k = 2 ${\widehat{\mathrm{SMD}}}_{\mathrm{QED}}=-0\text{.}14{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}31\,\mathrm{to}\,0\text{.}03;\,p=0\text{.}10;k=2$

). Other study and intervention characteristics were not significant moderators.

Time Spent Studying

We included 13 studies in the analysis of the effect on the number of hours a student spends studying, either captured in school or at home. ⁶ Off-grid technologies can increase the time spent studying. The overall average effect was moderate, positive and statistically significant (

SMD ^ = 0 . 11 ; 95 % CI: 0 . 03 to 0.19 ; p < 0 . 01 $\widehat{\mathrm{SMD}}=0.11{\rm{;}}\,95 \% \,\mathrm{CI:}\,0.03\,\mathrm{to}\,0.19;\,p\lt 0.01$

). However, the level of heterogeneity in this analysis was large and significant (

I 2 = 85.48 % $I^2=85.48 \%$

), and only three studies were not rated as having a high risk of bias (Figure 21).

OPEN IN VIEWER

Figure 21

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent studying.

Factors that helped explain differences in the results included the type of technology and the intervention mechanism. Interventions using solar lamps/lanterns with mobile phone chargers showed a larger effect compared to the rest of the technologies (p < 0.01), while SHS showed a larger effect compared to micro-hydro plants (p = 0.02). Interventions that provided the opportunity to access technology through market expansion showed a significantly smaller effect on time spent studying compared to those that directly provided the technology (p = 0.03).

When we rerun the analysis only for those studies where results were not rated as having a high risk of bias, the overall average effect on time studying is larger in magnitude but less precise and only marginally significant (

SMD ^ Low or med BOB = 0 . 17 ; 95 % CI: − 0 . 03 to 0 . 38 ; p = 0 . 09 ; k = 3 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}=0\text{.}17{\rm{;}}\,95 \% \mathrm{CI:}\,-0\text{.}03\,\mathrm{to}\,0\text{.}38;\,p=0\text{.}09;\,k=3$

).

None of the studies were considered an outlier and the main result was not sensitive to leaving a study out one at a time. However, this was the only outcome in the review for which there is significant evidence of funnel plot asymmetry, suggesting an unbalanced representation of results in the published literature (

t = 2.77 , df = 11 , p = 0.02 $t=\,2.77,\,{df}=\,11,\,p=0.02$

). The presence of publication bias indicates that the true effect size could be smaller, or not significant, than the pooled effect presented above.

Time Spent Studying (Children Only)

Interventions showed a positive and significant increase in time spent studying by children. We included 12 studies in this analysis. The estimated average effect based on the random-effects model was small, positive and statistically significant (

SMD ^ = 0.14 ; 95 % CI: 0.05 to 0 . 23 ; p < 0.01 $\widehat{\mathrm{SMD}}=0.14{\rm{;}}\,95 \% \,\mathrm{CI:}\,0.05\,\mathrm{to}\,0.23;\,p\lt 0.01$

). The true effects, however, appear to be heterogeneous (

I 2 = 75.26 % $I^2=75.26 \%$

) and all but two effect sizes were rated as having a high risk of bias (Figure 22).

OPEN IN VIEWER

Figure 22

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on time spent studying by children only.

Moderator analyses here showed similar results to those for the effects on time studying, regardless of age. Again, the effects found for solar lamps/lanterns with mobile phone chargers were on average larger compared to other technologies (

p = 0 . 03 $p=0.03$

), while the opportunity to access or market expansion was a mechanism with significantly smaller effects on the time spent studying by children compared to studies that directly provided the technology (

p = 0 . 01 $p=0.01$

).

When we removed effect sizes rated as having a high risk of bias from the analysis, the overall average effect was no longer significant (

SMD ^ Low or med BOB = 0 . 32 ; 95 % CI: − 0 . 09 to 0 . 73 ; p = 0 . 13 ; k = 2 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}=0\text{.}32{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}09\,\mathrm{to}\,0\text{.}73;\,p=0\text{.}13;\,k=2$

). However, this result should be interpreted with caution, given the small sample size it is based on.

Education

We assessed the effects of interventions on education using school attendance and test scores. We found positive, but not statistically significant effects for both attendance and test scores.

School Attendance

We found six studies looking at the effects on the time a student spends attending school, absenteeism ratios, percentage attendance, or the proportion of children regularly attending school. Included studies do not show a statistically significant effect on school attendance. The estimated average effect based on the random-effects model was very small and not statistically different from zero (

SMD ^ = 0.03 ; 95 % CI: − 0.02 to 0.09 ; p = 0 . 25 $\widehat{\mathrm{SMD}}=0.03{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0.02\,\mathrm{to}\,0.09;\,p=0.25$

). There was no significant amount of heterogeneity in the true effects (

I 2 = 27.72 % $I^2=27.72 \%$

), but only two studies were not rated as having a high risk of bias (Figure 23).

OPEN IN VIEWER

Figure 23

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on school attendance.

From the moderators tested, we found no significant differences across groups. Compared to high risk of bias studies, those rated as having some concerns reported a marginally smaller effect (p = 0.01) so that the pooled effect becomes negative, but it is still very small and not statistically significant (

SMD ^ Low or med BOB = − 0 . 02 ; 95 % CI: − 0 . 06 to 0 . 03 ; p = 0 . 48 ; k = 2 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}=-0\text{.}02{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}06\,\mathrm{to}\,0\text{.}03;\,p=0\text{.}48;\,k=2$

). Finally, none of the studies were considered an outlier, nor were any of the effect sizes in the analysis overly influential.

Test Scores

We include the results from five studies in our synthesis of the effects of off-grid interventions on test scores. Off-grid technologies do not change the average test scores of students. The overall average effect based on the random-effects model was very small and not statistically significant (

SMD ^ = 0.002 ; 95 % CI: − 0.02 to 0.03 ; p = 0.86 $\widehat{\mathrm{SMD}}=0.002{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0.02\,\mathrm{to}\,0.03;\,p=0.86$

). There was no significant amount of heterogeneity in this analysis (

I 2 = 0.00 % $I^2=0.00 \%$

), but most effect sizes used were rated as having a high risk of bias (Figure 24).

OPEN IN VIEWER

Figure 24

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on test scores.

From the moderators tested, we found no significant differences across groups. Removing the effect sizes with a high risk of bias, the results did not change (

SMD ^ Low or med BOB = 0 . 04 ; 95 % CI: − 0 . 06 to 0 . 13 ; p = 0 . 49 ; k = 2 ${\widehat{\mathrm{SMD}}}_{\mathrm{Low}\mathrm{or}\mathrm{med}\mathrm{BOB}}=0\text{.}04{\rm{;}}\,95 \% \,\mathrm{CI:}-0\text{.}06\,\mathrm{to}\,0\text{.}13;\,p=0\text{.}49;\,k=2$

). None of the studies had a value larger than ± 2.576, and hence, there was no indication of outliers.

Respiratory Illnesses

Measures of respiratory illness reported in the 10 studies used in this analysis included indices and individual indicators of respiratory diseases or symptoms. ⁷ We found no effect of off-grid technologies on this outcome. The overall average effect size based on the random effects model was very small and not statistically different from zero (

SMD ^ = 0 . 04 ; 95 % CI: − 0 . 02 to 0 . 10 ; p = 0 . 16 $\widehat{\mathrm{SMD}}=0\text{.}04{\rm{;}}\,95 \% \,\mathrm{CI:}\,-0\text{.}02\,\mathrm{to}\,0\text{.}10;p=0\text{.}16$

). We observed some heterogeneity in the effects used in this analysis (

I 2 = 43.43 % $I^2=43.43 \%$

). All but one estimate were rated as having a high risk of bias (Figure 25).

OPEN IN VIEWER

Figure 25

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on respiratory illnesses. We reversed the signs of effects on respiratory illness as we consider a reduction in this outcome to be positive.

From the moderators tested, we found no significant differences across groups other than a larger effect on average for studies from Uganda (p = 0.02) Most studies in this analysis came from SSA, one from South Asia and one from Latin America. All interventions were demand-focused, with varying mechanisms and scale, which did not help explain differences in results. There was only one study not rated as having a high risk of bias. None of the studies were an outlier or overly influential.

Women's Empowerment

Based on three studies that reported comparable women's empowerment outcomes, we found a small statistically significant positive effect on women's decision-making (

SMD ^ = 0 . 11 ; 95 % CI: 0 . 03 to 0 . 18 ; p < 0 . 01 $\widehat{\mathrm{SMD}}=0\text{.}11{\rm{;}}\,95 \% \,\mathrm{CI:}\,0\text{.}03\,\mathrm{to}0\text{.}18;\,p\lt 0\text{.}01$

). ⁸ Further, there was no significant amount of heterogeneity in this analysis (

I 2 = 32.61 % $I^2=32.61 \%$

). This finding should be taken with caution, given the small sample size for the analysis and the risk of bias rating of the included studies (Figure 26). This outcome only includes decision-making indicators, and no other aspects of women's empowerment.

OPEN IN VIEWER

Figure 26

Forest plot showing estimates of the random-effects model for the effect of off-grid interventions on women's empowerment.

There was insufficient variation in the included studies to allow for moderator analyses. None of the studies could be considered an outlier nor were any of the effect sizes overly influential.

Qualitative Synthesis

This section details the evidence we identified to answer review question 3. Here, we are interested in understanding which factors may or may not have impacted programme effectiveness, as opposed to understanding the impact of the included programmes from a qualitative lens. As discussed above, we conducted an additional search to identify documents to help answer this question. Our search led to the identification of 39 documents that provided qualitative and descriptive information related to barriers and facilitators of the effectiveness of the programmes included within our review. In addition, 30 of the included impact evaluations provided details on barriers and facilitators either through a dedicated qualitative analysis within a mixed-method evaluation or through descriptions of programme implementation.

Our analysis is constructed around and builds upon the work of Sovacool (2013), which identified 10 themes that contributed to the success or failure of renewable energy access programmes in the Asia-Pacific region. These are: appropriate technology; income generation; financing; political leadership; capacity building; programmatic flexibility; marketing and awareness; stakeholder engagement; community ownership; technical standardisation. Our analysis finds that these themes are pertinent to contexts beyond the Asia-Pacific region, in addition to other factors, such as the impact of COVID-19 and societal norms, that can also act as further barriers off-grid interventions face. Given that we are unable to make causal claims from the documents identified for this analysis, the results here should be interpreted with caution and viewed as suggestive only.

Implementation Features

Our quantitative analysis showed few differences in effects across outcomes between programmes implemented at the local, regional and national levels. The exception to this was for time spent on leisure activities, where regional programmes were shown to have a larger effect than national programmes. Our qualitative analysis also supported the idea that regional programmes may lead to increased participant engagement under certain circumstances. Qualitative studies highlighted that regional and local programmes, those with greater levels of community participation and channels for community input, might lead to greater engagement amongst participants.

Quantitative analyses also found that for two outcomes, kerosene consumption and time spent resting, programmes which had government agencies as implementing partners were more effective than programmes with other implementing partners. Qualitative findings suggest there was greater engagement by participants when they saw their government promoting the programme. On the other hand, national programmes, often promoted by governments, may have issues when mechanisms for community participation are not present or when the level of trust between communities and governments is low.

Programme Characteristics

Both the quantitative and qualitative analyses highlighted different features related to programme technologies and mechanisms that may help explain programme effectiveness. Across three outcomes, energy expenditure, time spent working and time spent studying, quantitative analyses found solar lamps/lanterns with mobile phone chargers to have larger positive effects than other, larger technologies. This is in contrast to the findings from qualitative studies, which highlighted that in some programmes, participants felt that the technologies provided to them did not meet their energy needs and were not of a large enough size. In these cases, participants purchased additional solar technologies. One qualitative study also highlighted the need to ensure that technologies are appropriate to local contexts; for instance, a programme brought a mobile phone charger from a neighbouring country where the programme was already operating, but it did not fit the mobile phones of the participants.

In terms of programme mechanism, the quantitative analysis found that providing information as well as financial support was more effective at reducing kerosene consumption compared to providing financial support on its own. This finding is supported by the qualitative analysis, where multiple studies highlighted the benefits of information and marketing components. Participants expressed positivity towards programmes where focal points were working in the community, who spread awareness of the technologies and the benefits of their uptake. Participants were also more prone to maintaining systems to a higher standard when programmes included an information mechanism dedicated to system maintenance.

Programme Targeting

When assessing whether there were any differences in effects between programmes that targeted different actors (e.g., households, individuals, businesses) or populations (e.g., women, children), quantitative analyses found no significant differences across outcomes. Qualitative studies found that targeting different actors, women in particular, may act as either a barrier or facilitator to programme effectiveness, depending on the context. For example, while in one programme women were unable to travel for entrepreneurial purposes due to social norms, limiting their ability to engage with the programme, in a different context, women had significant influence over household purchase decisions and so could persuade their husbands to participate in the programme.