Residential energy efficiency interventions: A meta-analysis of effectiveness studies

What is the effect of installing EEMs on energy consumption?

Our synthesis finds promising evidence that installing EEMs bundles reduces energy consumption. On average, installing bundles significantly reduced energy consumption. In most studies, installing individual EEMs caused smaller, statistically significant reductions in consumption, but a few studies estimate larger or negligible changes and one study found an increase in consumption. The results were similar when focusing on the five low risk of bias studies, with the caveat that the high quality evidence examining any EEMs is limited to one or two studies. Currently, there is not enough evidence to formally rate EEMs effectiveness; only one or two low risk of bias studies examine each EEMs. The effectiveness of each EEMs depends on many contextual factors (such as implementation or specific EEM features), and existing studies do not rigorously compare EEMs to each other.

What is the available evidence on funding mechanisms and costs?

All the interventions were fully or partially funded by governments, universities, or a mix of them. Eight studies conducted some type of cost analysis such as cost-benefit or cost-effectiveness analysis. Whilst some studies found that the energy saved by EEMs installation was greater than the installation cost, other studies identified small or even negative cost-effectiveness results. Among the two low risk of bias studies, one found a small negative rate of return from installing an EEM bundle—primarily because the reductions in energy consumption were much smaller than expected—and the second found a large positive rate of return from installing attic insulation.

EEM installation

EEM installation includes the replacement and upgrades of heating and cooling systems, the installation of insulation, more efficient boilers and heating, ventilation, and air conditioning technologies, among others (EEM installation examples are included in Adan & Fuerst, 2016; Howden-Chapman et al., 2007; Maher, 2013). EEM installation often involves “weatherisation” which increases energy efficiency by protecting the building from sunlight, wind and precipitation (examples of studies evaluating EEM installations are Fowlie et al., 2018; Francisco et al., 2017; Pigg et al., 2018). EEMs can be further categorised by the amount of postinstallation household involvement required:

Passive measures, such as insulation, do not require households to adopt a particular behaviour once completed

EEMs are often installed after energy audits, which provide households with recommendations on appropriate REEIs, as well as information on applicable utility and state incentives that can reduce or eliminate the cost of installation (Taylor et al., 2014). By providing households with additional information, such as a simulation of benefits, audits can overcome informational barriers to installing EEMs.

EEM installation combined with information provision interventions

These bundled interventions combine EEM installation with interventions that provide information designed to change household behaviour. These interventions inform households on how to best use the installed EEMs, such as advising households on how to set thermostats or how to reduce air conditioning load (examples of studies evaluating EEM installation in combination with behavioural interventions are James & Ambrose, 2017; Zivin et al., 2015). This guidance can be provided, for instance, by energy audits or other forms of technical assistance. Such guidance can be especially impactful for semi-active and active EEMs. Behavioural interventions can be broader than information provision, but we limited this review to information provision because another systematic review (SR) published this year (Khanna et al., 2021) is focused on broader behavioural interventions to reduce energy consumption.

Moderating contextual factors

The effects of REEI installation can vary depending on the context (Russell-Bennett et al., 2019), and accordingly the theory of change includes moderating factors. These include the characteristics of the housing (such as the age of the building), climate, the applicable policies and building standards, and the income level of the households. REEIs might have different impacts for low-income households due to the correlation between household income and energy consumption, after controlling for building characteristics (Abrahamse & Steg, 2009; Santin et al., 2009).

Figure 1 presents the anticipated theory of change, but the installation of REEIs is a complex process involving many different actors (such as installers and beneficiaries), and consequently some REEIs might lead to higher energy consumption or impaired wellbeing (Bone et al., 2010; Shrubsole et al., 2014). For instance, simply adding insulation without adjusting ventilation can reduce air circulation and the additional moisture can lead to mould and increases in other indoor-generated pollutants (Pigg et al., 2018; Shrubsole et al., 2014), or lead to overheating in summer (RAND, 2020). Similarly, installing REEIs might cause increased energy usage if households feel that their “good behaviour” allows increased energy consumption in other areas, so-called moral licensing (Jacobsen et al., 2012; Tiefenbeck et al., 2013).

Finally, REEIs might increase energy consumption due to the “rebound effect” of affordability (Davis et al., 2014; Shrubsole et al., 2014). This happens when the installed EEMs: (a) reduce the cost of operating equipment, causing the equipment to be used more (direct rebound effect), or (b) EEMs save households money and households use part of the saved income to increase energy consumption (indirect rebound effect). Therefore, simply considering energy consumption might underestimate utility gains from implementing these interventions, hence it is important to understand the causes of an increase in energy consumption in each context (Allcott & Greenstone, 2012; Hong et al., 2009).

Types of studies

To answer the first, second and fourth research questions, we included counterfactual studies that use an experimental or quasi-experimental design and/or analysis method that can plausibly control for confounding and selection bias (i.e., different types of households choose to install REEIs and these differences, not the REEIs, impact outcomes).

Specifically, we included the following study types: 1.

Randomised controlled trials with assignment at the individual, household, community or other cluster level, and quasi-randomised trials using prospective methods of assignment such as alternation.

We refer to studies in categories 3 or 4 as quasi-experiments.

For Research Question 3, we also looked at additional studies related to implementation, financial mechanisms and context for the studies included in the review.

Types of participants

We included any study that involved households living in single-family or multi-family residential buildings (dwellings) regardless of income or geographic location.

We excluded studies that installed EEMs in public, commercial, office or industrial buildings because, whilst a priority of institutions such as the EIB, the EGM only identified three studies targeting public commercial, office or industrial buildings. When a study included residential and nonresidential buildings and reported separate estimates for residential buildings (e.g., Liang et al., 2018), the residential estimates are eligible for inclusion in this SR.

Types of interventions

We included studies that measure the impact of at least one of the interventions listed in Table 1. Studies that compare an EEM control group to a bundle of EEM + information provision intervention group are not eligible because they are only examining the impact of information provision rather than the impact of an EEM plus the information provision However, studies that compare EEM + information provision intervention to a control group that does not receive any or another treatment, will be included.

Types of outcome measures

Duration of follow-up

We included any follow-up duration, coding multiple outcomes if studies report multiple follow-ups.

Types of settings

We accepted studies from any type of setting and any part of the world. We only reviewed studies conducted in real-world settings (i.e., we did not include efficacy studies).

Electronic searches

We conducted the search strategy in the following academic databases:

CAB Abst

We also searched the organisational databases and evidence repositories listed in Supporting Information Appendix C.

Searching other resources

We screened all studies listed in the bibliography of the energy efficiency EGM and other relevant SRs and literature reviews. In addition, we screened the reference lists of all included studies (backward citation search) and used Google Scholar to search for studies that cited included studies (forward citation search).

To identify additional studies, we contacted key experts and organisations through our review external advisory group and internal EIB reference group.

Targeted search for studies addressing Q3

To answer Question 3 relating to implementation, financial mechanisms and context, we attempted to identify programme and project documents associated with the programmes identified in the first stage of the search. We did this by undertaking a targeted search for programme names and authors using Google, after we identified the studies included in the review. Evidence on context and mechanisms was collected from all the included studies. Information on programme mechanisms was either suggested by study authors or identified by the review team.

Criteria for determination of independent findings

Estimating standard meta-analytic average effects assumes that each included effect is statistically independent (Hedges, 2019). The statistical significance of findings can also be inflated when there are dependencies within a study. Dependent effect sizes can arise when: (1) one study provides multiple results for a similar outcome of interest, (2) one study has multiple treatment arms compared to the same comparison group, or (3) multiple studies use the same data and report on the same outcome. We therefore used the following rules to ensure that only statistically independent effect sizes were included as primary findings (other effect sizes are reported in Supporting Information Appendix H).

When a study reported multiple outcomes using similar outcome constructs (Howden-Chapman et al., 2007), to enhance the potential for meta-analysis we selected the construct that is the most similar to other estimates for the same outcome type. For example, when studies included both measured and self-reported energy consumption (Howden-Chapman et al., 2007), for consistency across studies we extracted the measured consumption. When a study included more than one energy outcome (Adan & Fuerst, 2016; Fowlie et al., 2018; Grimes et al., 2016; James & Ambrose, 2017), such as electricity consumption and gas consumption and total energy (electricity + gas) consumption, we chose the outcome that would provide the most sensitive test of the intervention (such as gas for boilers or electricity for air conditioning).

No studies included more than one outcome period.

When we identified studies with multiple treatment arms and only one comparison group (Grimes et al., 2016; Hamilton et al., 2016; James & Ambrose, 2017; Suter & Shammin, 2013), we choose the intervention that most commonly resembles other studies' interventions as the primary comparison (other effect sizes are reported in Supporting Information Appendix H).

Several studies reported multiple effect size estimates using slightly different models; here we chose the one with the lowest assessed risk of bias or most similar to other studies. If a study included different analyses with overlapping samples (Alberini et al., 2019; Fowlie et al., 2018), we chose the one with the lowest risk of bias. Where we identified several studies/publications that report on the same analysis we used effect sizes from the most recent publication.

Selection of studies

We imported all search results into EPPI-Reviewer 4¹ and removed duplicates. After testing the inclusion/exclusion criteria for operationalisability, two independent research assistants double screened all studies against the inclusion criteria using information available in the title and abstract; any disagreements were resolved through conversations with a core review team member. Where a study's title and abstract did not include sufficient information to determine relevance, the study was included for a full text review.

While undertaking title/abstract screening, we took advantage of the text-mining capabilities of EPPI-Reviewer 4, to reduce the initial screening workload (O'Mara-Eves et al., 2015). We used the “Priority” screening function to prioritise screening the studies that were more likely to be eligible and accelerate the screening process. Ultimately, all the studies were independently reviewed by two screeners during the title and abstracts screening because we kept finding some potential includable studies until the end of the screening.

Studies included for full-text screening were double screened by two independent reviewers. Disagreements were resolved by discussion with a core review team member and the input of an additional core reviewer if necessary.

The screening of studies for Question 3 took place later, after studies were identified for inclusion in the core effectiveness component of the review. The studies identified to answer Question 3 were assessed for relevance, that is, whether they (1) examined one of the programmes in an included effectiveness study, and (2) whether they provide information on the implementation processes, context or mechanisms at play.

Data extraction and management

Using a standardised data extraction form (form provided in Supporting Information Appendix A), we extracted the following descriptive, methodological, and quantitative data from each included study:

Descriptive data including authors and publication date, as well as other information to characterise the study including country, cost data, type of intervention and outcome, population, and context.

We extracted all data using Excel. Descriptive and qualitative data were double-coded and checked by a core team member.

Assessment of risk of bias in included studies

Our literature search was inclusive, and identified studies that did not undergo peer-review. We assessed the risk of bias for the eligible impact evaluations, using the 3ie risk of bias tool (Supporting Information Appendix B) which covers both internal validity and statistical conclusion validity of experimental and quasi-experimental designs (Waddington et al., 2012) and the bias domains and extensions to Cochrane's ROBINS-I tool (Sterne et al., 2016).

Two reviewers independently assessed the risk of bias. When there were disagreements, they were resolved by discussion and the involvement of a senior reviewer. We conducted the risk of bias assessment at the study level, noting any potential differences in methods and the risk of bias for different outcomes.

We assessed the risk of bias based on the following criteria:

Factors relating to baseline confounding and biases arising from differential selection into and out of the study (e.g., “Was any differential selection into or out of the study (attrition bias) adequately resolved?”);

For each criterion, we coded each study as “Yes”, “Probably Yes”, “Probably No”, “No” and “No Information” according to how they address each domain. After the risk of bias was appraised for each criterion, an overall risk of bias rating was assigned using the following approach: (1) if any domain was appraised as “no” or “probably no”, then the overall risk of bias is high; (2) if all domains were appraised as “yes” or “probably yes”, then the overall risk of bias is low; (3) if the information needed to appraise one or more domain was unclear but the rest of the dimensions were appraised “yes” or “probably yes”, then the overall risk of bias is "some concerns".

Measures of treatment effect

Studies examining similar outcomes might report effects using different metrics (e.g., some studies' outcomes are in kilowatt hours and others are in the natural logarithm of kilowatt hours). To enable a synthesis of these findings, all study effects have been converted to standardised effect sizes that express the magnitude or strength of the relationship between the intervention and outcome (Borenstein et al., 2009; Borenstein & Hedges, 2019).

For studies reporting difference-in-differences computed with means and SDs, we use the formula described in Morris (2008):

d = [ y 1 , t − y 0 , t − y 1 , c − y 0 , c ] n t − 1 s 1 , t 2 + n c − 1 s 1 , c 2 ( n t + n c − 2 ) ,

V d size = 2 1 − ρ × 1 − 3 4 N − 9 × n t + n c n t n c × n t + n c − 2 n t + n c − 4 + 1 + d 2 2 ( 1 − ρ ) ( n t + n c n t n c ) − d 2 ,

ρ

For studies reporting regression coefficients, we used formulae from Lipsey and Wilson (2001). Fowlie et al. (2018) report both an intent-to-treat (ITT) estimate and a complier average causal effect (CACE) estimated using two-stage least squares. Because roughly 95% of treatment households did not install REEIs, the Fowlie ITT estimates a different impact than the average treatment-on-treated estimated by other studies; thus to calculate effect sizes we used the CACE and backed-out the baseline SDs (Fowlie et al., 2018; table II). We used these SDs to compute the effect, instead of the outcome SD.

When the regression coefficient and the pooled SD of the outcome are available:

d = β t S ,

β t

S = n t − 1 s t 2 + n c − 1 s c 2 ( n t + n c − 2 ) .

V d = n t + n c n t n c + d 2 4 ( n t + n c ) .

When studies do not report the outcome SD, we approximate a rough effect size using the coefficient t statistic. For the regression models that include covariates or fixed effects—almost of the models included in this study—the formulas make strong assumptions to approximate the effect size. Where the pooled SD of the outcome is unavailable but the sample size information is available for each group:

d = t · stat 1 n t + 1 n c .

V d = n t + n c n t n c + d 2 4 ( n t + n c ) .

The t statistic (t-stat) is calculated by dividing the coefficient by the standard error or using the reported t-stat. If the authors do not report a t-stat but report the p value to three decimal places, we used the Excel T.INV.2T function to approximate the t statistic.

Where the pooled SD and sample size of each group are unavailable, but the total sample size information is available, we used a formula that assumes both groups have identical sample sizes:

d = 2 t N .

V d = 4 N + d 2 4 N .

For randomised trials reporting unadjusted odds ratios, we used the formula reported in Borenstein et al. (2011):

d = ln ( OR ) 3 π ,

V d = 3 1 T outcomes + 1 C outcomes + 1 T non − outcomes + 1 C non − outcomes π 2 ,

We converted d's to Hedge's g by multiplying by the following approximation:

1 − 3 4 N − 9

1 − 3 4 N − 9 2

We also calculated impacts in kWh by converting different metrics for energy consumption (m³ of gas consumed, 100 cubic feet of gas consumed, and 1,000,000 British thermal units) to kWh annually using the US Energy Information Administration's online energy conversion calculator. Some studies reported daily, monthly, or quadrimester energy consumption; we converted these to annual consumption by multiplying by 365, 12, and 3, respectively. For the five studies (Adan & Fuerst, 2016; Alberini et al., 2016; Alberini et al., 2019; Fowlie et al., 2018; Liang et al., 2018) that used the natural log of energy consumed as the outcome, we report the estimated percentage change in energy consumed as calculated by

e β − 1

Unit of analysis issues

Unit of analysis issues arise when a study's unit of allocation (assignment) is different from the unit of analysis, and the analysis does not account for the potentially correlated outcomes of units within clusters. Only one included study (Carranza & Meeks, 2016) had a unit of assignment that differed from the unit of analysis. For this study, we use the author-reported cluster-corrected standard errors.

Dealing with missing data

When studies did not provide data needed for meta-analysis (such as means and SDs), we contacted two study authors to obtain the required information. One author (James & Ambrose, 2017) did not respond to the request and we were unable to obtain the necessary data. We excluded this study from the quantitative synthesis but included it in the descriptive analysis. Another author (Howden-Chapman et al., 2007) did not respond to a request for data for three outcomes (Short Form-36 full scales: role-physical, role-emotional, and social functioning); however, there was complete data for other outcomes and those outcomes have been included.

Two other authors (Carranza & Meeks, 2016; Maher, 2013) did not respond to a request for additional information needed to appraise risk of bias. These studies were appraised as some concerns for risk of bias.

Assessment of reporting biases

We attempted to reduce publication bias by conducing a comprehensive search for all relevant studies and including grey literature in the review (see Vevea et al., 2019). In addition to the electronic searches, we performed backward and forward citation-tracking, contacted experts, and searched websites. Although we had planned to conduct a funnel analysis, we did not due to the small number of studies in each meta-analysis.

Quantitative data synthesis

Once we identified all the eligible studies, we mapped out the interventions, climates, comparisons, and outcome measures. Based on an examination of these characteristics, we chose how best to synthesise findings across studies.

Meta-analysis

We only synthesised studies using meta-analysis when we identified at least two effect sizes: (a) involving a similar intervention, and (b) implemented in a similar climate. We report separately by intervention category because we believe that funders, policymakers, installers, and households are interested in how specific EEMs impact energy consumption (not how any type of EEM impacts consumption). We analyse by climate because climate affects whether heating or cooling is mostly needed and how much heating or cooling is needed; climate thus directly determines how much energy consumption can be changed. Accordingly, we classify study climates by the average number of annual heating degree days (HDD)

Intervention characteristics, housing characteristics, and other relevant factors varied across studies, and so we conducted a maximum-likelihood random-effects meta-analysis with inverse-weighting by statistical precision using the metafor package in R (R Development Core Team 2018). The weights are based on within-study statistical precision as well as the estimated between-study variance. In case the estimates are sensitive to the estimator (Viechtbauer, 2005; Veroniki et al., 2016), Supporting Information Appendix G reports meta-analysis statistics estimated using a restricted maximum likelihood estimator (Viechtbauer, 2005) and fixed-effects meta-analysis.

When there is only one study examining an intervention, we present the effect in a table and synthesise findings narratively.

Subgroup analysis and heterogeneity reporting

For one type of intervention—EEM bundle—there were sufficient studies to conduct sub-group meta-analyses for the following categories of interest to the primary funder:

Resident socioeconomic status

We assess heterogeneity by calculating the Q statistic, I ², and τ to provide an estimate of the amount of variability in the distribution of the true effect sizes (Borenstein et al., 2009). We complement this with a graphical presentation of heterogeneity of effect sizes using forest plots that include prediction intervals as recommended by Borenstein (2019).

Sensitivity analysis

We conducted two sensitivity analyses. The first used the leave1out command in R to assess whether the results of the meta-analysis were sensitive to the removal of any single study. For the meta-analysis that included more than one low risk of bias study (EEM bundle), we also assessed sensitivity of results by removing high risk of bias studies from the meta-analysis.

Results of the search

The PRISMA flow chart (Figure 2) shows the results of the search and screening processes conducted for this SR in November 2020 (although some relevant studies were identified through the EGM; Berretta et al., 2021), we conducted a new search tailored specifically to the inclusion criteria of this SR, and to include studies after the search for the EGM was conducted in April 2020.) The initial studies were identified by searching academic databases (n = 13,589) as well as by searching grey literature in websites, online libraries and repositories of selected organisations (n = 40). After removing duplicates, we screened 12,976 studies at the title and abstract level. We excluded 12,879 studies based on the inclusion criteria and 97 studies were identified as potentially relevant studies and underwent a full-text screening. At the title and abstract screening, most of the studies were excluded because they did not include an intervention (6919), the intervention was not relevant (5015), the study design used was not one of those listed in the protocol (Berretta et al., 2021) (515), lack of empirical data (290), or they did not address effectivenss (131); the remaining studies were excluded because they were duplicates. Excluding a large number of the studies initially identified is not unusual. SRs often exclude the vast majority of studies identified through comprehensive searches (Wang et al., 2020).

OPEN IN VIEWER

Figure 2

PRISMA flow-chart

During the full-text screening stage, we excluded 81 studies for different reasons. Several studies were screened out for multiple reasons, but only the first reason was coded. The most common first reasons were: ineligible study design (14), the lack of a valid comparison group (12) or ineligible intervention (12 studies). The complete list of the studies excluded at the full-text screening stage can be found at the end of this report.

Geographic coverage

Over two-thirds of the studies (69%, n = 11) were conducted in North America and Europe (Figure 3). Of these 11 studies, five were conducted in the United States (Alberini et al., 2016, Fowlie et al., 2018, Liang et al., 2018; Maher, 2013; Suter & Shammin, 2013), two studies in Ireland (Beagon et al., 2018, Scheer et al., 2013), two in the UK (Adan & Fuerst, 2016, Hamilton et al., 2013), one in the Netherlands (Aydin et al., 2017) and one in the Ukraine (Alberini et al., 2019). Three studies were conducted in the Pacific, respectively two in New Zealand (Grimes et al., 2016, Howden-Chapman et al., 2007) and one in Australia (James & Ambrose, 2017). No studies were conducted in South America, with one study each conducted in Africa (Costolanski et al., 2013) and Asia (Carranza & Meeks, 2016) (Figures 4 and 5).

OPEN IN VIEWER

Figure 3

Geographic coverage of included studies

Thirteen of the 16 studies took place in high-income countries (n = 13, 81%) using World Bank definitions (Figures 4 and 5). Costolanski et al. (2013) conducted in Ethiopia is the only included study that took place in a low-income country and the two lower-middle income countries are Ukraine (Alberini et al., 2019) and the Kyrgyz Republic (Carranza & Meeks, 2016).

OPEN IN VIEWER

Figure 4

Number of included studies, by region. Regions based on World Bank classifications

OPEN IN VIEWER

Figure 5

Number of included studies, by country income category. Income categories based on historical World Bank classifications for the year of publication

Most of the studies (n = 9) evaluated interventions in colder places (Figure 6) (1954–2860 HHDs), four studies were located in warmer locations (300–830 HHDs), and one in a moderate climate (1000–1500 HDDs). The remaining two eligible studies are not included in this graph because they examined lighting interventions (compact fluorescent lightbulbs or CFLs) which are not affected by temperature (Costolanski et al., 2013; Carranza & Meeks, 2016) (Figure 6).

OPEN IN VIEWER

Figure 6

Number of studies, by climate

Interventions in included studies

Seven of the eligible studies examined individual EEMs, eight studies looked at bundles of two or more EEMs, and one study (Adan & Fuerst, 2016) examined both single EEMs and bundles. The most commonly examined individually EEMs were: loft/attic insulation (n = 4), replacement of a boiler or a heat-pump (n = 4), and the installation of cavity wall insulation and efficient lighting, each examined by two studies (Table 3).

In eight of the nine studies examining bundles different households received various bundles options; the specific EEMs installed were tailored for each dwelling. For instance, in the programme evaluated by Fowlie et al. (2018), an energy audit of each house determined which EEMs were appropriate and cost effective for that house. Some households replaced their furnace and installed attic and wall insulation, while others replaced their furnace and had their windows sealed, and so on.

We did not find any studies evaluating passive cooling systems, or district heating/cooling. Only one included study (James & Ambrose, 2017) looked at behavioural interventions combined with EEMs. The intervention households in this study received: a bundle of EEMs only (such as insulation, weather sealing, appliance repair and replacement, and lighting upgrades); a behavioural intervention only which included information and house operation strategies to encourage behaviour change; or both the energy efficiency and behavioural intervention (only the first and third are eligible REEIs for this review).

Some studies such as Adan and Fuerst (2016) are listed under different interventions categories because they included two or more different EEM types. For Adan et al., in particular there were four different types of EEMs, of which one was a bundle of the other three.

Additional characteristics of the 16 studies, including the 15 different programmes, are reported in Supporting Information Appendix D.

Programme take-up

In 10 of the 16 studies, households independently decided to install the EEMs or participate in the programme; while in the remaining six studies the households were selected by the research or programme team. Among the households that independently installed, there was perfect compliance—all the houses selected for the treatment group received the REEI and none of the comparison houses did. Among the researcher- or programme-chosen households, there was perfect compliance in two studies (Beagon et al., 2018; Suter & Shammin, 2013) and in four studies there was imperfect compliance—some treatment households did not install REEIs and/or some of the comparison households did install (Carranza & Meeks, 2016; Fowlie et al., 2018; Howden-Chapman et al., 2007; James & Ambrose, 2017). Among the imperfect compliance studies, only Howden-Chapman et al. (2007) reported why households did not participate.

Twelve of the studies were conducted retrospectively, and used existing datasets (n = 7) or programme records (n = 6) to measure take-up rates; two other studies conducted surveys where household self-reported installing the EEMs (Alberini et al., 2016; Alberini et al., 2019). The final study (Costolanski et al., 2013) did not clearly describe the data source.

Most studies examining EEM bundles also reported how many households installed specific EEMs.

Intervention funding mechanisms and context features

To better understand the context and the funding mechanisms, we conducted a search on Google in which we retrieved 18 additional documents on the programmes evaluated in the included studies.

In 50% of the studies, the interventions were completely or partially subsidised by governments (n = 8), in 31% by a mix of public and private institutions or households (n = 5), 13% of the studies (n = 2) were funded by the research team or universities, and finally in one study the funding was not reported (Figures 7 and 8).

OPEN IN VIEWER

Figure 7

Intervention funding

OPEN IN VIEWER

Figure 8

Number of studies providing subsidies, by household income

The government-funded REEIs include: the SEAI Better Energy Communities scheme in Ireland (Beagon et al., 2018, Scheer et al., 2013); the Carbon Emission Reduction Target and the Community Energy Saving Programme in the UK (Adan & Fuerst, 2016); the federal Weatherization Assistance Program in the United States (Fowlie et al., 2018); the Meer Met Minder programme in the Netherlands (Aydin et al., 2017); the Warm Up New Zealand: Heat Smart scheme (Grimes et al., 2016); the Energize Phoenix programme led by the City of Phoenix, Arisona State University, and the state's largest electricity provider, Arizona Public Service (Liang et al., 2018); and the Low Income Energy Efficiency Program established by the Australian Government (James & Ambrose, 2017).

Two studies were funded by a utility company (Maher, 2013) or by a utility company (the Ethiopian Electric Power Corporation) in combination with the World Bank (Costolanski et al., 2013). In three other studies the interventions were funded through a combination of private and government funding (Alberini et al., 2016, 2019; Hamilton et al., 2016). For example, in Alberini et al. (2019), 91% (351) of the households financed the EE renovations entirely themselves, and 33 households took advantage of government programmes such as government loans and the Warm Loans programme.

Finally, for two studies (Howden-Chapman et al., 2007; Suter & Shammin, 2013) the installed REEIs were funded by the researchers or a university.

REEI installation was subsidised regardless of household income. Four studies sampled predominantly or entirely low-income households, 11 studies targeted samples of households at all income levels, and the remaining study involved college undergraduates. In three of the four studies with interventions targeting low-income households, the REEI installation was fully subsidised; in the remaining study, installation was partially sub sidied (Figure 8). Six of the 11 studies that included households of any income level had fully subsidised REEIs; and REEIs were partially subsidised in five. In Alberini et al. (2019), Alberini et al. (2016) and Hamilton et al. (2016), some of the households fully funded the EE renovations themselves (Table 4) and some were subsidised. In most of the partial-subsidy programmes, the subsidy corresponded to between 20% and 30% of the total costs. In most of the cases subsidies were provided as a reimbursement rather than an ex-ante subsidy. In the study involving rental undergraduate housing, the REEIs were funded by the university which owned the housing.

A total of five studies (Alberini et al., 2016; Aydin et al., 2017; Fowlie et al., 2018; Liang et al., 2018; Scheer et al., 2013) reported including an audit before installation, where an expert visited the residence to assess energy usage and loss, and provided recommendations for reducing energy consumption. Among these five studies, two included a full subsidy and three included a partial subsidy (Figure 9).

OPEN IN VIEWER

Figure 9

Number of studies involving audits and/or subsidies

Outcomes in the included studies

To be included in the review, studies need to measure at least one of the primary outcomes (energy consumption, energy affordability, CO₂ emissions and air quality indices and pollution levels). All 16 studies included at least one measure of energy consumption, and none of the studies measured any of the other primary outcomes. A few studies, such as James and Ambrose (2017) estimated how the intervention impacts on emissions and air pollution by multiplying the change in consumption by a factor.

Two studies also reported on secondary outcomes in the health status, comfort, and wellbeing domain: Howden-Chapman et al., 2007 detailed indoor temperature as well as mental and physical health effects; Suter and Shammin (2013) measured monthly household ambient indoor temperature.

Three-quarters of the included studies (n = 12) examined only one measure of energy consumption, either electricity, gas, or total energy (Alberini et al., 2016, 2019; Aydin et al., 2017, Beagon et al., 2018, Carranza & Meeks, 2016, Costolanski et al., 2013, Hamilton et al., 2016, James & Ambrose, 2017; Howden-Chapman et al., 2007; Liang et al., 2017; Maher, 2013; Scheer et al., 2013, Suter & Shammin, 2013), see Table 5. Two studies reported impacts on two consumption outcomes (Adan & Fuerst, 2016; Grimes et al., 2016,) and two reported impacts on all three consumption measures (Fowlie et al., 2018; James & Ambrose, 2017).

No studies reported the following outcomes which were specified in the protocol: energy security, air quality index, income savings, GHG emissions, job creation, building stock value.

The measurement units for total energy and electricity consumption are typically reported in kWh except for Fowlie et al. (2018) who reported consumption in MMBtu. Gas consumption is commonly measured in m³ (Alberini et al., 2019; Aydin et al., 2017), or kWh (Adan & Fuerst, 2016; Beagon et al., 2018; Hamilton et al., 2016; Scheer et al., 2013) or Ccf (Suter & Shammin, 2013) or MMBtu (Fowlie et al.).

The energy consumption data was obtained from utility companies, gas and electricity meter operators, or in one study, household energy bills (Alberini et al., 2019). To measure indoor temperature and humidity, one study installed sensors (Suter & Shammin, 2013) and a second installed “data-loggers” (Howden-Chapman et al., 2007). Howden-Chapman et al. (2007) measured health status using interviewer-administered or self-administrated surveys with the residents or health care providers. Many of these health measures are subsets of or adopted from existing health measurement scales, such as SF-36 scales (Howden-Chapman et al., 2007).

Risk of bias summary

Of the 16 eligible studies, five were appraised as having a low overall risk of bias (“probably yes” or “yes” in the eight risk of bias domains, see Figure 10 for domains), two studies were appraised as some concerns due to incomplete reporting, and the other nine studies were appraised as having a high overall risk of bias (rated “no” or “probably no” in at least one domain). We appraised risk of bias using slightly different criteria for randomised trials and quasi-experimental designs.

OPEN IN VIEWER

Figure 10

Risk of bias for included studies using randomised designs

Three of the five randomised trials were rated as low overall risk of bias (see Figure 10 and Supporting Information Appendix Table E1 for the appraisal of each study on each criterion), and two of the 11 quasi-experiments were rated as low overall risk of bias (see Supporting Information Appendix Table E2). One randomised trial (Carranza & Meeks, 2016) and one quasi-experiment (Maher, 2013) were rated as some concerns because the study did not report information needed for the appraisal and the author did not respond to a request for additional information.

Three of the five included randomised trials were appraised as having overall low risk of bias (Fowlie et al., 2018, Howden-Chapman et al., 2007; Suter & Shammin, 2013). For the one study with unclear appraisal on selection bias (Carranza & Meeks, 2016), the authors did not report how 14 assigned but unsurveyed clusters were chosen (i.e., whether this attrition was random); otherwise, there were no serious concerns with this study. One randomised trial (James & Ambrose, 2017) was appraised with concerns in four domains: compromised random assignment (several households were assigned based on researcher perceptions of responsiveness); high attrition likely related to whether the household was assigned to treatment or comparison group; important baseline differences between groups; and the authors were more likely to have outcome data from control households for certain months. For the other studies, there was less risk of performance bias, outcome measurement bias, or analysis bias because outcomes were typically from administrative data.

Two of the quasi-experiments were appraised as having a low overall risk of bias (Adan & Fuerst, 2016; Grimes et al., 2016). The most common issues for the included quasi-experiments were selection bias and confounding, with only three of the 11 studies being appraised as low risk of bias in both those domains (Figure 11). The three studies with unclear appraisals on the confounding domain (Alberini et al., 2016; Hamilton et al., 2016; Maher, 2013) did not report the statistics needed to assess baseline equivalence. Similar to the randomised trials, the quasi-experimental outcomes were typically administrative records from utility companies, so there was less risk of bias in the other domains.

OPEN IN VIEWER

Figure 11

Risk of bias for included studies using quasi-experimental designs

Presentation of results

We report the magnitude of energy consumption impacts in two ways: (1) standardised mean difference (Hedges' g), and (2) change in kilowatt hours (kWh). Hedges' g enables impacts to be compared across studies/interventions and is thus our primary reporting metric, used in the text and forest plots. However, because Hedges' g impacts are in SD units, a nonintuitive metric, we also report impacts in kWh. Hedges' g and average difference in temperature or health are also reported for other outcomes.

To facilitate understanding of impacts reported as Hedges' g, we provide intuitive benchmarks estimated in other studies. First, one recent study (Huebner et al., 2015) estimates that adding one additional member to a household was associated with an increase in household energy consumption by 0.46 SDs (Hedges' g = 0.46). A second study (Huebner et al., 2016) estimated that houses with an electric clothes dryer consumed 0.22 SDs of energy more than those who hung their clothes to dry (although these estimates were not calculated through a counterfactual analysis that can establish causation, these studies do control statistically for important factors, such as amount of livable space.)

We estimate average impacts using random effect meta-analysis because we seek to make broader inferences and relevant factors varied across studies. We present heterogeneity statistics (Q statistic, τ, and I ²) in Supporting Information Appendix F, along with prediction intervals in the forest plots. Random-effects meta-analysis can provide unreliable estimates for the between-studies variance (τ ²) when the analysis includes a small number of studies, as in this review. As a sensitivity check, Supporting Information Appendix G presents overall averages estimated using a fixed-effects meta-analysis. Because of the small number of studies and the diverse interventions, we do not conduct a meta-regression to systematically explore sources of heterogeneity. Instead, we describe possible explanatory factors in the text.

Forest plots

When there are two or more studies, results are presented in forest plots (such as Figure 12, a common graphic that presents estimated impact for each study, variation between studies, and average impact across studies.

OPEN IN VIEWER

Figure 12

Impact of highly subsidised compact fluorescent light bulbs. For individual studies, the rightmost column and the horizontal lines indicate the 95% confidence intervals (we can be 95% confident that this interval captures the actual impact). For the average impacts, the confidence interval is displayed in the rightmost column and represented by the width of the diamond, while the dashed horizontal lines indicate the prediction interval (the range in which the future impact will likely fall)

Studies are first categorised by climate subgroup for the heating/cooling REEIs, and each study is presented in a separate row (such as Figure 13). When there are multiple studies in a given climate, the last row for the climate presents the average impact for the climate subgroup. The final row in the plot presents the average for all studies across all climates.

OPEN IN VIEWER

Figure 13

Impacts of attic or loft insulation only, by climate subgroup. Because of the large impact estimated in Suter et al., the scale extends to −2 rather than −1 as in the other forest plots. For individual studies, the rightmost column and the horizontal lines indicate the 95% confidence intervals (95% of the time this interval will capture the actual impact). For the average impacts, the confidence interval is displayed in the rightmost column and represented by the width of the diamond, while the dashed horizontal lines indicate the prediction interval (the range in which the impact will fall for 95% of the population)

The first column in the plot lists the study's first author and publication year. The second column graphically presents the study's Hedges' g on a scale from −1 to +1, with a dotted vertical line indicating 0 Hedges' g (no impact). The point estimate for each study is represented with a box and the size of the box reflects the study sample size. The point estimate is the best estimate for the actual impact, but there is variation due to the randomness of the assignment/sampling process; we report this uncertainty using CIs and predictions intervals. The 95% CIs for each study are indicated with horizontal lines—95% of the time this interval will capture the actual impact. The average impact across studies is displayed with a green diamond and the width of the diamond indicates the CI—95% of the time this interval will capture the actual average impact. Finally, the prediction intervals for the average impact are indicated with horizonal dashed lines—if we predict the REEI impact for a randomly selected population, there is 95% chance the impact will fall in this interval (Borenstein, 2019). The third column in the plot numerically reports the impact and 95% CIs.

Results from one study not presented

One eligible study (James & Ambrose, 2017) did not report sufficient information to calculate an effect size and the lead author did not respond to a request for this information. Accordingly, findings from this study are not reported in this section (this was the one study that occurred in a moderate climate).

Supplemental findings

Because some of the studies examined different interventions using the same comparison group, creating dependent comparisons, these supplemental impacts are reported in Supporting Information Appendix H.

Source of data

All the data used for the following analysis were obtained from the published studies included in this review.

Lighting interventions

Two included studies examined providing households with up to four CFLs. Each study estimated that providing CFLs to households significantly reduced electricity consumption, with the average impact equal to Hedges' g = −0.29. The Costolanski et al. (2013) intervention allowed Ethiopian households to exchange incandescent bulbs for CFLs, while the Carranza and Meeks (2016) intervention enabled Kyrgyz households to buy CFLs at highly subsidised prices.

Impacts varied between the studies, with Carranza and Meeks (2016) estimated impact roughly one-third of the impact estimated by Constolanski et al. (2013). One possible reason for the lower impact in the Carranza and Meeks study was somewhat lower participation (households received 3.2 CFLs on average and 12% of households received no CFLs compared to 98% of households in Costolanski et al. receiving four CFLs). Moreover, although neither study was appraised as low risk of bias, selection bias and confounding were much larger risks to the Costolanski et al. findings (see Supporting Information Appendix E). However, the meta-analysis weights the larger sample Costolanski et al. study (3998 households included in the analysis) more when computing the average impact than the Carranza and Meeks study (899 households) (Table 6).

Heating and cooling interventions

Because climate directly determines how much heating or cooling is needed, we synthesise impacts for heating and cooling interventions separately by climate as determined by annual HDD—specifically, the colder and warmer climate subgroups as reported in Figure 6—as well as an average across all studies and climates. In the forest plots, the meta-analytic average for colder climates is reported using a blue diamond, for warmer climates using a red diamond, and an overall average impact across climates using a green diamond (when there is only one study in a climate, we do not report a meta-analytic average for that climate). Presenting studies separately by climate does not mean that any differences are due to climate; differences between climates might be due to other factors.

Attic insulation without any other EEMs

Four studies examined the impact of attic insulation on energy consumption. Three studies took place in colder climates with gas consumption as an outcome (Adan & Fuerst, 2016; Hamilton et al., 2016; Suter & Shammin, 2013), and one study took place in a warmer climate, with the electricity used to run air conditioning consumption as the outcome (Maher, 2013). The average impact across all climates was statistically insignificant (Hedges' g = 0.04).

Two of the three studies examining attic insulation in cold climates estimated impacts less than Hedges' g = 0.05, while the third study estimated a much larger impact; the average impact in colder climates was not statistically significant (see Figure 13). Adan and Fuerst (2016), a low risk of bias study, found a smaller reduction in energy consumption (Hedges' g = −0.03), although the sample was so large (over 150,000 households) that this impact was statistically significant. Hamilton et al. (2016) also estimated a smaller impact for a large sample of roughly 105,000 households (Hedges' g = 0.01). However, Suter and Shammin (2013) another low risk of bias study, found a larger impact on reducing energy consumption (Hedges' g = –1.35), so large that the forest plot scale is different.

The descriptions of attic insulation are incomplete, but Suter et al. might have installed thicker attic insulation than households in Hamilton et al. and Adan et al. Specifically, Hamilton et al. noted that most households installed insulation between 5 and 75 mm thick. Although Adan et al. do not report the amount of insulation, like Hamilton et al., the study took place in the UK and involved some of the same energy efficiency programmes and might have also installed similarly thin installation. Limited additional insulation would not be expected to change consumption much. In contrast, the Suter et al. homes were roughly 100 years old, “with minimal amounts of prior insulation… and with occupants that typically heat their homes to approximately 72 degrees F” (p. 559), and the attic insulation cost roughly $1000 indicating a significant amount of insulation was installed.

There are other possible explanations for the high Suter et al. finding. Suter et al. note that the sample houses were “relatively homogeneous in their size and characteristics” (p. 554); this similarity will lead to less variation in consumption between houses (i.e., a smaller SD) and a larger Hedges' g. Table 7 indicates that although the Suter et al. impact was large, in absolute terms it was less than twice that of Maher. Moreover, as the small box in the plot indicates, this study involved only 24 households and so the impact is not precisely estimated (because the random-effects meta-analysis weights by precision, this impact does not have much effect on the average impact). Finally, the houses in Suter et al. were rentals that included utilities with the rent, and these dwellings typically use more energy (Leth-Petersen & Togeby, 2001). Thus, households in Suter et al. might have been using more energy at baseline enabling the insulation to have a larger impact.

A fourth study (Maher, 2013) examined the impact of attic insulation in a warmer climate (7526 households) and found that insulation significantly reduced electricity consumption. This study occurred in a hot, humid area (Gainesville, Florida, United States), a location with high need for air conditioning.

Electric heat pumps without any other EEMs

Two studies examined the impact of installing electric heat pumps. The average impact across studies was -0.11 and not statistically significant. However, there was substantial variation between the two studies. Replacing older pumps with more efficient versions significantly reduced electricity consumption (335 households) in a colder climate (Alberini et al., 2016), but Grimes et al. found a smaller, significant increase in energy consumption from installing new heat pumps (24,164 households). The authors note that part of the “increased energy use occurred at warmer temperatures, when heat pumps were likely used as air conditioners” (p. 165). This represents new energy consumption because traditionally New Zealand houses did not have air conditioning (French et al., 2009). Thus, in contexts where older heat pumps are being replaced by more efficient models, the Alberini et al. estimates are likely more relevant.

Although Grimes et al. was appraised as low risk of bias, the study results do not fully capture heat pumps' impact on total energy consumption. Specifically, the authors note that they did not measure solid fuel consumption, such as wood and coal stoves. Yet solid fuel was used to generate roughly 56% of home heating energy in New Zealand at the time of the study (French et al., 2009). Thus, installing heat pumps to replace wood or coal stoves would increase measured electricity consumption, but also reduce consumption of unmeasured solid fuels.

An earlier version of the paper (Grimes et al., 2011) reports two supplemental analyses consistent with the possibility that the installed heat pumps reduced total energy consumption. In the first analysis, the authors estimated impacts among subgroups of households that did and did not use unmeasured fuels before installing heat pumps. While households using unmeasured energy sources before installation increased measured energy (electricity + piped gas) consumption after installation, households that used measured sources preinstallation typically reduced total? energy consumption at colder outside temperatures after installation. Although these differences were not statistically significant, the authors note the analysis was underpowered as they did not have data for most households' preinstallation energy sources. In a second subsample analysis, comparing households that did and did not have access to piped gas before installation, the authors find that households with gas—who presumably replaced gas heaters with heat pumps—significantly reduced energy (gas + electricity) consumption at colder temperatures while households without gas—presumably more likely to use unmeasured energy sources—increased total measured energy consumption at colder temperatures (as only 13.6% of sample households had access to gas, this subsample had a small effect on overall impacts) (Figure 14 and Tables 8–10).

Other individual EEMs

One included study examined the impact of installing a more efficient boiler or installing cavity wall insulation (Adan & Fuerst, 2016). The samples in these analyses were independent (i.e., did not share a comparison group) and involved roughly 360,000 households (boiler) and 103,533 households (cavity wall). Another study, involving 7526 households, examined the impact of replacing central air conditioning with a more efficient system (Maher, 2013).

Adan et al. find that the impact of cavity wall insulation was roughly three times larger in magnitude than their estimated impact for loft insulation; the authors do not explain why wall insulation was more impactful. Maher finds a large impact of replacing central air conditioning, likely caused by two features: (1) the study was conducted in Gainesville, Florida, a hot and humid climate with a strong need for air conditioning, and (2) the subsidy programme required that the replacement air conditioning system be rated highly efficient by the US Environmental Protection Agency.

EEM bundles

The previous sections have synthesised and/or reported the impact of installing one EEM, but households often install more than one EEM at a time. Eight studies examined the impact of these so-called bundles of multiple EEMs, and we classify these interventions into two categories:

Bundles where the EEMs installed in each household varied because each household chose which EEMs to install (i.e., households within each study received different bundles).

Bundles where each household installed the same bundle (one study (Adan & Fuerst, 2016) involving four independent comparisons).

Five of the seven studies in the first category examine bundles installed after an energy audit; for these bundles, the specific EEMs installed in each household varied based on the audit (i.e., households often installed tailored interventions based on dwelling needs assessed by a professional). A supplemental analysis for these studies is reported in Supporting Information Appendix I.

Each category is relevant for a different research question, and accordingly we analyze each category separately.

OPEN IN VIEWER

Figure 14

Impacts of electric heat pump only, by climate subgroup. For individual studies, the rightmost column and the horisontal lines indicate the 95% confidence intervals (95% of the time this interval will capture the actual impact). For the average impacts, the confidence interval is displayed in the rightmost column and represented by the width of the diamond, while the dashed horisontal lines indicate the prediction interval (the range in which the impact will fall for 95% of the population)

EEM bundles, where EEMs installed in each household vary

The included studies uniformly found that installing bundles reduced residential energy consumption (Figure 15), with an average impact of 0.36 SD units that was statistically significant. Although the bundles and contexts were diverse, most estimated reductions in energy between −0.29 and −0.64. The two exceptions were Fowlie et al. (2018) (−0.15 and statistically significant) and Alberini et al. (2019) (−0.08 and not statistically significant). The number of households included in the analysis ranged from 45 (Beagon et al., 2018), 136 (Howden-Chapman), 231 (Liang et al., 2018), 429 (Alberini et al., 2019), 5198 (Aydin et al., 2017), roughly 25,000 (Fowlie et al., 2018) to over 150,000 (Scheer et al., 2013).

OPEN IN VIEWER

Figure 15

Impacts of EEM bundles that vary by household, by climate. For individual studies, the rightmost column and the horizontal lines indicate the 95% confidence intervals (95% of the time this interval will capture the actual impact). For the average impacts, the confidence interval is displayed in the rightmost column and represented by the width of the diamond, while the dashed horizontal lines indicate the prediction interval (the range in which the impact will fall for 95% of the population)

Overall, the two low risk of bias studies indicate an impact of between −0.15 to −0.29. Unlike the other studies that estimated an average treatment-on-treated effect, the Fowlie et al. estimates are for compliers, those participants who complied with their assignment, roughly six percent of the sample (the intervention encouraged and helped eligible families to apply to the REEI programme). In addition, the Fowlie et al. effect size calculations relied on additional assumptions (see Section 4.3.5) that might have affected the estimated effect size.

The Alberini et al. estimates are lower than others, but most study households only installed one or two EEMs, typically wall insulation by itself or with new windows (the authors note that instances where a sample “household does more than one efficiency upgrades are not numerous”; p. 22). Given the limited nature of the intervention bundle, the impact of the intervention (-0.08) is similar to the impact of wall insulation only estimated by Adan and Fuerst (2016) (−0.11).

All bundles: Identical EEMs installed in each household

Unlike the previous studies that examine bundles where the EEMs differed by residence, using a retrospective analysis, Adan and Fuerst (2016) examined the impact of identical bundles installed in each residence. Specifically, they study all four possible combinations of attic insulation, cavity wall insulation, and boiler replacement (see Table 11).

The authors estimate that each bundle reduced gas consumption, although the effects were always less than Hedges' h = −0.11. All of the samples have more than 10,000 households, with precisely estimated impacts, thus even when the reduction in consumption is small it is statistically significant.

The impact of installing additional EEMs in the bundles does not appear to be additive. For example, the impact of installing wall insulation only was −0.11 (see Table 11) and the impact of installing a boiler only was −0.04, but the impact of installing a boiler and wall insulation was −0.11, not 0.15. Similarly, the largest reduction in Table 11 occurred when households installed both cavity wall insulation and a boiler, and this reduction was larger than when the household installed cavity wall insulation, a boiler, and loft insulation. The authors label this pattern “less straightforward” and believe the lack of additivity is due to the prebound effect—households in the least efficient residences consume the least energy (prebound effect) and were most likely to install multiple EEMs; thus, households that consume less energy through energy behaviour were more likely to install bundles but their behaviour meant that they were also least likely to benefit from bundles.

Indoor temperature and health outcomes

Few studies that estimated REEI impacts on energy consumption also examined other outcomes. Two included studies, examining different REEIs, reported indoor temperature outcomes (Howden-Chapman et al., 2007; Suter & Shammin, 2013). (In addition to the randomised experiment included in this synthesis, Fowlie et al., 2018 also used a quasi-experimental design to estimate impacts on indoor temperature outcomes; however, this design was ineligible.) One study reported impacts on several self-reported health outcomes (Howden-Chapman et al., 2007).

Indoor temperature

Both studies estimated that households installing an REEI slightly increased indoor temperatures during the winter, although the impact was only statistically significant for Howden-Chapman et al. (Hedges' g = 0.35). These increases are consistent with a small rebound effect. Howden-Chapman recorded main bedroom temperatures during the winter in 140 randomly selected houses. Suter et al. installed temperature and humidity sensors in each home, recorded temperature every 10 min (the data were aggregated by month) and focused on indoor temperatures during winter months (November through to March). Smaller impacts in Suter et al. are expected—households did not pay for utilities, and thus had little incentive to keep their house at a lower temperature to save money (Table 12).

Health outcomes

Of the included studies, only Howden-Chapman et al. (2007) examined health outcomes, and this study included outcomes in the following domains: self-reported mould (allergen) in the house, self-reported low vitality, self-reported low happiness, self-reported fair or poor health, self-reported winter colds or flu, self-reported wheezing, and self-reported phlegm. We do not include health care usage outcomes (such as visits to general practitioners) in this SR because the desired direction is unclear (e.g., a visit to a primary care doctor could be either a good or bad outcome).

For each outcome, Howden-Chapman et al. found that installing the EEM bundle improved self-reported health (Table 13) and the impacts were statistically significant.

Household socioeconomic status

Two studies examined interventions implemented with only low-income households (Irish social housing in Beagon 2018; US Weatherization Assistance Program in Fowlie et. al. 2018) and one study included households that mostly lived in lower-income areas (Howden-Chapman et al., 2007). The intervention in each of these studies was an EEM bundle, and all of the studies estimated that bundles reduced energy consumption (see Figure 16).

OPEN IN VIEWER

Figure 16

Impacts of EEM bundles for low-income households only, by climate. For this analysis, low-income refers to relative income status within a country, not based on absolute income level. The confidence interval for the overall average, displayed in the rightmost column and the width of the diamond, will include the actual population mean 95% of the time. The narrow prediction interval, the bars around the average for all climates, indicate the interval in which 95% of the population will have an impact. EEM;

The small number of studies and the much larger sample size in one study limit the generalisability of the subgroup analysis. Specifically, because the meta-analysis weights studies by precision, when rounded, the average impact for low-income household studies (0.16) is almost the same as the impact estimated by Fowlie et al., a study with 28,790 households (the other two studies have 181 households in total). Two of the three studies (Fowlie et al. and Howden-Chapman et al.) were appraised as low risk of bias. Given the small number of studies involving low-income households, further investigation of differences between these households and other households was not possible.

Region of residency (European Union-27 and the UK)

Five studies examined interventions conducted in EU-27 or the UK: two in Ireland (Beagon et al., 2018; Scheer et al., 2013), one in the Netherlands (Aydin et al., 2017) and two in the UK (Adan & Fuerst, 2016; Hamilton et al., 2016). The studies that occurred in Ireland and the Netherlands examined a similar intervention—bundles that vary by household—and are synthesised (Figure 17).

OPEN IN VIEWER

Figure 17

Impacts of EEM bundles for EU households only, by climate. EEM; EU;

All of these studies took place in colder climates, and the average impact is large (0.61) and precisely estimated, but none of these studies was appraised as low risk of bias. Because the meta-analysis weights studies by precision, the Beagon et al. impact, based on 45 households is not heavily weighted.

Remove individual studies

For the two meta-analyses with three or more studies (attic insulation and EEM bundles that vary by household), we assessed whether the results of the meta-analysis were sensitive to the removal of any single study (see Supporting Information Appendix J). This analysis found minor differences for the meta-analysis of bundle interventions and always in the expected direction (i.e., removing studies with impacts below the average increased the average and vice versa). For the attic insulation meta-analysis, removal of Adan and Fuerst (2016), Hamilton et al. (2016), and Maher (2013) led to unreliable estimation with accordingly large standard errors.

Remove high risk of bias

When limiting syntheses to those studies appraised as low risk of bias (n = 5), only one meta-analysis is possible—EEM bundles that vary by household. This meta-analysis included two studies: Fowlie et al. (2018) and Howden-Chapman et al. (2007).

The average effect size is −0.16 and statistically significant across the two low risk of bias studies is similar to the Fowlie et al. estimate of −0.15 (see Figure 18); the Fowlie et al. estimate dominates the meta-analysis because it is much more precisely estimated as the study has a sample size more than 200 times larger than Howden-Chapman et al.

OPEN IN VIEWER

Figure 18

Impacts of EEM bundles for low risk of bias studies only, by climate. EEM;

Cost analyses

Eight studies also conducted some type of cost analysis, estimating the cost of saving energy, reducing pollution, or calculating the rate of return on investment (Table 14). These calculations are based on study estimates of how much REEIs affect energy consumption and other assumptions. Thus the larger the estimated reduction, holding other assumptions constant, the lower the cost estimate or the higher the rate of return.

Consistent with the approach taken in this review, we focus on the two studies that report cost analyses and were appraised as low overall risk of bias.

These two studies estimated substantially different rates of return. One of the studies (Fowlie et al., 2018) examined EEM bundles in a large sample of low income homes and found a small negative rate of return and a cost per ton of CO₂ eliminated that was several times higher than typical estimates of the social cost of CO₂. The other study (Suter & Shammin, 2013), found a high positive rate of return from installing attic insulation in a small sample of college residential houses. These estimates are consistent with the impact estimates—Fowlie et al. found that installing these relatively expensive bundles led to smaller reductions in energy consumption, while Suter et al. found that installing relatively cheap attic insulation led to substantial reductions in energy consumption.

Characteristics of included studies

Our database search and grey literature search retrieved 13,629 studies. Independent screening by two reviewers identified 16 eligible studies evaluating 15 different programmes or REEIs. Most studies were conducted in high-income countries, including five in EU-28 countries and five in the United States. Three studies were conducted in Australia/New Zealand, and three studies in lower-middle and low-income countries (Ethiopia, Ukraine, and Kyrgyz Republic).

The studies evaluated diverse interventions. Half of the studies (n = 8) evaluated a bundle of two or more EEMs, that it is, a combination of EEMs that typically varied by household with a single effect size estimating the impact of the whole bundle. The remaining eight studies either evaluated one type of EEM (Carranza & Meeks, 2016; Costolanski et al., 2013; Suter & Shammin, 2013), or reported separate effect sizes for each EEM evaluated (Adan & Fuerst, 2016; Hamilton et al., 2016).

To answer Research Question 3, we collected information on funding mechanisms. In most of the studies REEI installation was subsidised (or partially subsidised), regardless of whether the intervention targeted low-income households (n = 4) or mixed-income households (n = 11); the remaining study (Suter & Shammin, 2016) fully subsidised the renovation of university student housing. Most of the subsidies were provided by governments (n = 10) or private organisations (n = 4), while the other two studies were funded by the university and researchers who led the evaluation. In 9 of the 15 studies involving households (Suter & Shammin, 2016 involved university students) the subsidies fully covered the REEI cost; in the remaining studies partial subsidies usually covered between 20% and 30% of the total cost. We did not find much information on programme implementation.

Roughly a third of the included studies (n = 5) reported that the intervention required an energy audit before the installation of the upgrades. Some of the households in the other studies might have conducted an audit, but this information was not reported.

To answer Research Question 4, we collected data on costs. Nine of the studies provided some information on intervention cost, which for heating/cooling REEIs was $3089 on average with a median of $2348. In Costolanski et al., 2013 the consumer cost was around $0.40 for an 11 W CFL bulb (the market price for the same bulb was $1.42); in Carranza and Meeks (2016) the programme cost was on average $8.81 per household.

All of the included studies reported at least one energy consumption outcome. In most of the studies the data were collected from administrative records or obtained from utility companies and gas and electricity meter operators, including household level energy bills.

Quantitative synthesis

Our quantitative synthesis finds promising evidence for subsidising EEM bundles—the average reduction in energy consumption was statistically significant. The evidence examining individual EEMs is more limited and studies typically find smaller, statistically significant reductions in consumption; a few studies estimate large impacts or negligible impacts. One eligible study (James & Ambrose, 2017) was not included in the synthesis, as the authors did not report enough information to calculate an effect size, and did not respond to a request for this information.

The results were similar when limited to the five low risk of bias studies, with the caveat that the evidence examining any REEI was only one or two studies. This limited evidence and the importance of contextual factors means we could not formally rate the effectiveness of individual REEIs.

To be most helpful for policymakers, funders, agencies, and households who are often choosing between different types of EEMs, we grouped similar interventions together when synthesising findings. Because climate directly determines the amount of energy needed for heating/cooling REEIs, for these REEIs we report effects separately for warmer/colder climate subgroups as well as provide an overall effect across all included studies.

The two included studies evaluating subsidisation of CFLs both found a statistically significant reduction in electricity consumption. The estimated impacts in Carranza and Meeks (2016) had a lower risk of bias and were roughly one-third of Costolanski et al. (2013), possible due to fewer installed CFLs and less selection bias.

Overall, the four included studies found that installing loft/attic insulation had mixed impacts, with an average impact close to zero. One low risk of bias study conducted in a colder climate—and likely to have installed thicker installation—reported a large reduction in energy consumption (Suter & Shammin, 2013) but had a small sample size (24 households). Another study (Maher, 2013), conducted in a warm, humid climate, reported a statistically significant reduction in consumption. The other two studies, the first of which was low risk of bias, reported smaller effects, but the average amount of additional insulation installed in these studies appears to be minimal.

Two studies examined the replacement or installation of heat pumps (Alberini et al., 2016; Grimes et al., 2016). Replacing older pumps with more efficient versions significantly reduced electricity consumption in a colder climate (Alberini et al., 2016). However, in warmer climates where electric heat pumps were typically installed to replace other heating appliances, Grimes et al. found smaller, positive effects, indicating an increase in energy usage, partially due to households using the heat pumps to cool the residence in warmer weather. However, the Grimes et al. results did not accurately capture heat pumps' impact on total energy consumption, as the authors did not measure coal and wood energy consumption, which are primary energy sources in New Zealand. Supplementary analyses indicate that the installed heat pumps actually reduced overall energy consumption across all sources (i.e., households used more electricity but less unmeasured wood and coal). Thus, the Alberini et al. study impacts are likely to provide more relevant estimates of how heat pumps impact energy consumption.

Three other EEMs were examined by a single study. For instance, Maher (2013) found that in a warm, humid climate, replacing central air conditioning unit with a high-efficiency unit significantly reduced electricity consumption. A low risk-of bias study, Adan and Fuerst (2016) conducted independent evaluations of both cavity wall insulation and gas boiler replacement finding significant but relatively smaller reductions in gas.

Eight studies examining EEM bundles—combinations of two or more EEMs, with the specific EEMs installed differing by household—typically found promising results. There was significant variation in impacts, possibly due to variation in risk of bias, population and the EEMs installed. Reductions in energy consumption were statistically significant in seven of the eight studies. Focusing on the two low risk of bias studies, conducted with mostly low-income households, the impact on residences with installed bundles was statistically significant.

One low risk of bias study (Adan & Fuerst, 2016) examined the impact of installing identical bundles in each dwelling and found that the impact of installing additional EEMs does not appear to be additive. The authors attribute the nonadditivity to the prebound effect—households in the least efficient residences used behaviour to consume less energy (prebound effect) and were most likely to install multiple EEMs, and their low-consumption behaviour continued and minimised impacts.