Using big data for evaluating development outcomes: A systematic map

The problem, condition or issue

Policymakers need access to reliable data to evaluate development outcomes and decide on future resource allocation. Governments, multilateral organisations and other development players in low and middle income countries (LMICs) use censuses, nationally representative household surveys, other household surveys and administrative data to evaluate development programmes and policies. With the increasing complexity of development programmes, there is a need to collect a vast array of output, outcomes and contextual variables to robustly assess impact. However, significant data collection challenges remain. Data challenges for IEs include limitations on sample size and power due to budget constraints, inaccessible or difficult-to-reach sections of target populations, measurement errors due to recall bias, inadequate frequency and level of aggregation, inadequate information on controls and covariates, data collection lag times and difficulties in measuring long-term impact

Data sources

Big data offers great potential for answering some of these data needs. More importantly, it answers the causal questions around which policies or interventions work, including in contexts where traditional methods of data collection are challenging. The UN Global Pulse (2013) defines big data as being digitally generated (as opposed to digitised manually), passively produced (a by-product of digital services, transactions and interactions), automatically collected and geographically and temporally trackable. Although there is no formal definition for big data, currently the term is characterised by the three Vs: high volume, velocity and variety. Satellite images, sensors and drones, mobile phone CRDs, commercial transactions data, online searches, social media, citizen reporting or crowdsourced data are the sources of big data.

Integrating big data with traditional household surveys and administrative data can complement data availability, quality, granularity, accuracy and frequency, as well as help measure development outcomes temporally and spatially in a number of new ways (BenYishay et al., 2018; Lokanathan et al., 2017; Salganik, 2017; UN Global Pulse, 2016; York & Bamberger, 2020). For example, satellite images and mobile CRD have been used in mapping poverty (Blumenstock et al., 2015; Jean et al., 2016), disaster response (Lu et al., 2012; Wilson et al., 2016) and food security (Decuyper et al., 2014). Web searches and social media were used in predicting unemployment and crime instances (Gerber, 2014; Xu et al., 2013).

While big data is increasingly used for tracking indicators and monitoring development progress on SDGs (Lokanathan et al., 2017; UN Global Pulse, 2012; Vaitla, 2014), available data are less often utilised to address causal questions about the effects of specific policies and programmes. Big data can contribute to answering some of the causal questions around which interventions work. Big data prediction models can generate proxy estimates for key development outcomes such as wealth, human development, infrastructure quality, forest cover and more, which can be used in experimental (Jayachandran et al., 2016; Pellegrini, 2019) and quasi-experimental studies (BenYishay et al., 2018; Jaiswal et al., 2020). Satellite images such as night light, crop intensity, water availability, land use, proximity to services and physical attributes such as elevation or slope can be used in IEs as a direct measure of outcomes or as covariates. Furthermore, big data can be used for measuring and evaluating the long-term impacts of policies and programmes, conducting ex-post evaluations and estimating spatial heterogeneity. For example, satellite data are available at least as far back as 1993 for all places (high-resolution pictures are available for the entire globe at a granularity as low as 1 × 1 metre), allowing measurement of long-term impacts. This can help fill the gaps in evidence that cannot be addressed by traditional data sources.

Why it is important to develop the systematic map

The potential of big data to answer causal attribution, however, is still not widely understood or used, especially in LMICs (York & Bamberger, 2020). In this context, a systematic collection of various sources of big data and ways of measuring and evaluating development outcomes will be a great value addition to the development community's contribution to evidence-informed policymaking.

In this paper we look at IEs, SRs and measurement studies

Types of study design

Table 1 summarises the criteria we used for searching, screening and including the studies for the map.

Report structure

We use the SDGs as the basis for identifying the outcome categories, similar to Phillips et al. (2017).

Filters for presentation

We have added filters such as population, challenging contexts, geographic coverage to provide more details about the data sources and outcomes of interest. The population filter provides information on whether the studies in the map identify subgroups such as rural, urban, refugees, conflict affected persons, ethnic minorities. With the challenging contexts filter we show if the studies were in areas which were remote, conflict affected, affected by disease outbreak, and so on. We also identified different regions where the studies were conducted.

Dependency

Each unit of analysis was a report, peer reviewed journal publication, working paper. We included one study per unit of analysis. In cases where we had multiple reports we included the latest one with preference given to peer review versions and/or based on ease of access to a particular version. When multiple studies were covered by a single report, we included the report and then mapped the data sources and outcomes accordingly.

Screening and study selection

We first employed Evidence for Policy and Practice Information (EPPI)-Reviewer's built-in text mining, an ML technique, to sort the studies based on the inclusion and exclusion criteria at the title and screening stage (see Appendix D for more details on how text mining was used for screening). This reduced the number of studies to be screened at the title and abstract level to 9720. At the second stage, three researchers screened the studies for eligibility based on inclusion and exclusion criteria defined in the study protocol. At the end of stage two, we had identified 1348 studies to be screened at the full-text level.

At stage three, four researchers coded the studies at the full-text level. We followed a single screening with safety first approach,

Data extraction and management

The three-stage process of screening resulted in a final list of 437 studies including 48 IEs, 381 measurement studies and 8 SRs. Through a consultation process, we identified the metadata to be extracted using a standardised data extraction tool and defined them in the study protocol. During the final studies coding, we collected metadata such as outcomes studies, outcome subcategories, data sources used, geographical location of the intervention, country, evaluation design, target population, data transparency, ethics and other bibliographic information from the included studies (see Appendix D for the data extraction tool). We also critically appraised the SRs (see Appendix E for the SR appraisal tool and summary of the included SRs). Due to the size and the nature of included studies we did not conduct critical appraisal of IEs or measurement studies.

Tools for assessing risk of bias/study quality of included reviews

All SRs have been critically appraised using 3ie's critical appraisal checklist, adopted from the SURE checklist. Appendix F summarises all the SRs included in our sample and Table G1 provides details on the critical appraisal of the included studies.

Methods for mapping

The studies were mapped using 3ie's evidence gap map platform, which is organised into rows and columns. Various big data sources are placed in the rows and the development themes are placed in the columns. Any intersecting cell represents the development outcome measured or evaluated using the particular type of big data. Different colour bubbles represent the type of study: grey bubbles denote IEs, blue bubbles denote measurement studies, green bubbles denote high-quality SRs and red bubbles denote low-quality SRs. Hovering over the bubbles will show the links to studies. There are also filters for different regions, countries, study design, fragile context and the target population of the studies.

Development themes (such as environmental sustainability, economic development and livelihoods) contain a large number of studies. We have provided maps within the main map to show how the studies are distributed across the subthemes under these broad themes. In the submaps, the big data sources are mapped against level 2 or level 1 subclassification provided in the SDGs as relevant. For example, the submap for economic development and livelihoods has been coded against the level 2 indicators of eradicating poverty (SDG 1) and employment and economic growth (SDG 8), and the submap for environmental sustainability is coded against the level 1 classification of SDGs 12, 13, 14 and 15. The following development themes have submaps:

Economic development and livelihoods

Results of the search

We systematically searched academic databases with the help of an information specialist (see Appendix C for the list of academic databases and search strings) and manually searched specialist organisational websites and grey literature sources. The initial searches resulted in 17,393 studies, of which 17,008 studies were identified through bibliographic databases search and 385 studies were identified through hand-searching specialist databases and IE and grey literature repositories (Figure 1). All the results were uploaded onto EPPI-Reviewer 4 for screening and coding. Screening of the studies was done in three stages.

OPEN IN VIEWER

Figure 1

The PRISMA flowchart.

Excluded studies

This report includes papers written in English and published between 2005 and 2019. This map does not include studies that develop algorithms or methodologies for using big data without explicit application to measuring or evaluating a development outcome. This map does not include studies that describe how big data and ML have been used in development programming to help programme implementation, coordination and management for designing and scaling new development solutions; neither does it include studies that show how big data methods are used in randomised controlled trials (RCTs) to identify the differential impact of subgroups and in improving survey data collection, such as defining sampling frames.

Studies awaiting classification (if applicable)

If applicable, list the characteristics of any studies that have been identified as potentially eligible but have not been incorporated into the map (ER40).

Distribution of studies across data sources

As discussed in Section 2.1.2, big data can be generated by human interaction on social media, process-mediated data recorded by governments and the business and machine-generated data that is recorded by the automated systems. Figure 6 shows that machine-generated data are used the most. Of the total number of studies, close to 84% (n = 380) of the studies used some form of machine-generated data, while 12% (n = 53) of the studies used human-generated sources and 17% of the studies (n = 77) used process-mediated data.

OPEN IN VIEWER

Figure 5

Number of studies by income classification.

OPEN IN VIEWER

Figure 6

Number of studies per different type of big data.

Table 4 below provides a detailed breakdown of the number of studies per data source.

Distribution of studies across development themes

Section 4.2 identifies 10 broad development themes based on SDGs. Figure 7 and Table 5 show the number of studies across the development themes. About 50% of studies (n = 217) focus on environmental sustainability, which includes sustainable consumption and production, climate change, underwater life, and life on land. Economic development and livelihoods accounts for about 26% of the total studies (n = 114). Urban development and health account for 16% each (n = 68). Governance and human rights (7%, n = 30) and energy, industry and infrastructure (7%, n = 30) account for the remaining studies.

OPEN IN VIEWER

Figure 7

Number of studies against development outcomes.

While the distribution of IEs and measurement studies across the development themes remains the same as the overall distribution, there are a substantial number of IEs on economic development and livelihoods (17%) and energy, industry and infrastructure (17%). While most of the SRs looked at cross-sectoral themes, health is the most-studied sector (n = 5), followed by urban development (n = 3) and environment sustainability and economic development (n = 2). See Appendix F for critical appraisal of the SRs.

Units of observation

The unit of observation (or unit of analysis) is the class of elemental unit that constitutes the population and the units of measurement. Typically, in IEs, the units of observation are individuals, households, facilities (in facility surveys) or various level of administrative units such as villages, counties or districts. We have classified the unit of observation as population (including both individuals and households) or administrative units (villages, land parcels or any other units with a spatial element). The unit of observation seems to be an important element in analysing the use of big data in measuring development outcomes. Figure 8 (Panel 1) shows that about 70% of the measurement studies (n = 267) and 65% of the IEs (n = 31) have administrative units as their unit of observation. There is a clear distinction between different sources of big data, as shown in Figure 8, Panel 2. The unit of analysis for satellite data-based studies is predominantly administrative units (n = 259, 83%), while CRD-based studies are usually based on population units (n = 53, 82%). This difference shows that satellite data are more applicable when the outcome of interest has some spatial dimension such as local economic development, agricultural land productivity, forest cover or urban development.

OPEN IN VIEWER

Figure 8

Units of observation.

Studies using mixed methods

Mixed-methods IEs that combine qualitative and quantitative analyses help assess the quality implementation and reliability of data and understand the mechanism of programme impact (Bamberger, 2012). Big data IEs can be combined with qualitative methods. However, only three IEs and five measurement studies reported using mixed methods.

Studies with a rural or urban focus

Figure 9 shows the proportion of studies focused on rural areas or urban areas, or both. Most studies looked at both rural and urban areas (74%, n = 325). About 9% of the studies (n = 41) focused on rural areas; 12% (n = 51) focused on urban areas. Among the remaining studies, 14 studies looked at conflict affected population, 4 were studies of ethnic minorities and 2 studied refugees.

OPEN IN VIEWER

Figure 9

Distribution of population subgroups.

Studies by fragile context

We used the OECD definition of fragile context that includes conflict, institutional, social fragility, environmental, health and climatic risks (OECD, 2018). Figure 10 shows that 91 studies included on the map (21%) are from countries considered to be fragile. About 39 studies were conducted in a conflict or humanitarian crisis context; 22 studies each were conducted in contexts of difficult terrain and natural disasters; and 15 studies were conducted in the context of epidemics or disease outbreaks. There was one measurement study in the context of a chemical/radio-nuclear disaster. IEs follow the same pattern, except for one notable gap: there are no IEs in the context of epidemics or disease outbreaks despite a reasonably good number of measurement studies.

OPEN IN VIEWER

Figure 10

Number of studies in fragile contexts.

Table H4 in Appendix G shows that satellite data are the most used in fragile contexts, followed by CRD data and then the sensor data. The table also shows that almost all the big data sources have been used in one or two fragile contexts, indicating the importance of big data in fragile contexts.

There is a considerable potential for measuring various development indicators using big data

We identify a significant and growing evidence base of measurement studies that use some form of big data to measure a development outcome. Some outcomes are more amenable to the use of big data than others; environmental sustainability, economic development and livelihoods, health and well-being and urban development are where the majority of studies are concentrated. Education, sanitation, governance and human rights seem to be less responsive to big data use.

Multiple entries for most development theme indicate the potential of big data in contributing to measuring development indicators. Identifying measurement studies will be a valuable addition to development evaluators who look for innovative ways to measure a development outcome that was difficult to measure at all required spatial and temporal scales using conventional data collection methods.

There is potential for more IEs using big data on development interventions

The map contains 48 IEs. Use of big data measures in IEs as main outcomes or for controlling key covariates is fast-growing, but the IEs are fewer in number compared to measurement studies as well as in terms of the extent of their thematic and geographical coverage. IEs seem to be concentrated more around environmental sustainability, economic development and urban development. This complements existing efforts to build the evidence base in international development, as these sectors have much less rigorous evaluations (Sabet & Brown, 2018). The IEs also concentrate on using satellite data.

Satellite data are used most

The map shows that 71% of the measurement studies and 81% of the IEs used satellite data. This is also one of the sources that has been used since the early 2000s. The prominence of satellite data are primarily due to the fact that satellite images offer unique possibilities for measuring and evaluating development outcomes. Given the vast number of satellites covering almost every location on earth, it is possible to collect data at a high granularity (spatial resolution) and for multiple temporal frequencies for the past 30 years. Satellite data are freely available from several sources (such as NASA's Landsat and MODIS and the European Space Agency's Sentinel); more importantly, several preprocessed databases are available (such as AidData's Geoquery,

Spatial dimension matters

One of the key findings of the map is that most of the big data studies are applied in the context where the phenomenon studied has a spatial dimension, meaning the outcome and other covariates are measured on a spatial scale. Close to 70% of the studies on the map report using administrative units as their unit of measurement. This is particularly true for satellite and sensor data-based studies, as 82% have administrative units as their unit of measurement (such as local economic development, agricultural land productivity, forest cover or urban development). This is referred to as geospatial IE (BenYishay et al., 2017). However, there is considerable difference across data sources as CRD data are used to measure changes at the population level.

CRD data has great potential for measuring and evaluating development outcomes but is not yet used in IEs

CRD data are one of the most widely used sources in measurement studies. This is used for measuring population movement, migration, disease spread, and so forth. Despite a number of high-profile measurement studies, our systematic search did not find even one IE that used CRD data for rigorously evaluating a development outcome. This is a notable gap and a potential area for future exploration. It should be noted that CRD data are also fraught with multiple methodological challenges (such as nonrepresentativeness, lack of completeness, etc.) and ethical challenges (such as consent, unintended exclusion, etc.). Further, CRD data has been difficult to obtain as it is proprietary and hence it is difficult to maintain data transparency.

Other big data sources such as human-sourced and process-mediated data have good proof-of-concepts

Human-sourced data (such as social networks, internet searches, mobile data content citizen reporting or crowdsourced data) and process-mediated data (such as data produced by public agencies and by businesses) have a good number of measurement studies as proof-of-concept for using these sources to measure various development outcomes, but not many IEs use these sources. This also shows the possibility of potentially using these sources in future IEs. Similar caveats on methodological and ethical challenges discussed above in relation to CRD data will apply.

East Africa is well-represented, but not the rest of Africa

The geographical distribution of measurement studies and IEs show that the studies are evenly spread across the continents. However, Ethiopia, Kenya, Rwanda, Tanzania and Uganda are well-represented in terms of number of measurement studies and IEs, but the only non-East African country that seem to have well-represented on the map is Nigeria. There are very few studies in the rest of Africa. This gap is particularly serious given Africa's data challenges (Serajuddin et al., 2015).

Big data holds great potential for conducting IEs in fragile contexts, including during conflicts, humanitarian crises, epidemics and natural disasters

Conducting rigorous evaluation in fragile contexts (such as natural disasters, disease outbreaks and other crisis contexts) can be costly, risky to the beneficiaries and the evaluators, and in some cases outright infeasible. We identified 73 measurement studies, 17 IEs and one SR in such fragile contexts. Measurement studies are spread evenly across conflict or humanitarian crisis, disease outbreaks or epidemics, natural disasters and difficult-to-reach terrain. However, the IEs are concentrated around conflict and difficult terrain. The number of measurement studies indicate the potential for more IEs in fragile contexts.

There are potential sectors and themes where SRs will be useful

Though the number of IEs are fewer, the map highlights a few potentials thematic areas where SRs will help answer key questions on policy and research methods. For example, there is a concentration of IEs using satellite data, referred to as geospatial IEs, but we know little about how satellite data can help evaluate development programmes better, where it can add value, what type of interventions could be better evaluated and the technical challenges involved in using satellite data. An SR of all geospatial IEs that have used satellite data across the sectors will help understand the potential and challenges in using satellite data for IEs.

Similarly, there is a concentration of IEs on environmental sustainability and within that climate action and forest management. Though there a few SRs and evidence gap maps on forest management (Pelletier et al., 2016; Puri et al., 2016), a new review with reference to innovative, new data sources used in rigorously evaluating forest cover and the advantages and challenges thereof may be useful.

Using satellite data in IEs

In a 3ie funded evaluation conducted by the Institute for Financial Management and Research, Pande and Sudarshan (2019) evaluated the recent environmental clearance (EC) reforms in India. Before the 2006 reform, mines of area over 25 hectares were required to hold a public hearing before approval. The new EC reform required mines of area between 5 and 25 hectares to hold a public hearing as well. This study exploits this historical discontinuity in clearance requirements to evaluate the impact of public hearings on mines' environmental compliance. Apart from rigorously evaluating the EC process in India, this study also provides a proof-of-concept for the use of remote sensing data and other publicly available data to monitor mines' environmental compliance. Using satellite data to assess the impact of EC process requires data on the timing of the intervention, the geographical scope of the intervention (ie the individual mines in this case), the outcome of interest (such as air pollution, land cover and water quality for the corresponding intervention) and control areas for the years before and after the intervention.

The following were the key steps involved in the big data IE.

Step 1: the researchers used web scraping techniques to collect information on the mines from their EC application for the years from 2006 to 2016, available online in a database published by Ministry of Environment, Forests and Climate Change. These are all mostly scanned PDF documents. The researchers scrapped for the information on project name and location (the tehsil and village where the mine is located); the dates of key EC stages of submission, review and approval; and mine characteristics such as minerals mined, mine production capacity and size of the mine. They also scrapped the clearing letters available in the same database for cross checking the data. They collected information about all 934 relevant mines and used 134 of them in their regression discontinuity analysis. Finally, the researchers hired ML Infomap, a local company, to geocode all the mines identified.

Step 2: Satellite data on various key environmental outcomes such as air pollution, land cover and water quality were collected from different sources.

Step 3: The researchers then linked the geocoded mine sites to the corresponding cells of environmental outcomes. Of the total 934 mines in their database, they could link 889 of 1 km cells of EVI data and 882 of 250 metre cells with corresponding geocoded mine sites. They could also link 538 site monitors to the mines.

Step 4: The new EC reform required the mines of between 5 and 25 hectares in area to hold a public hearing that had not been considered big enough to hold public hearings during the previous regime (ie only the mines of above 25 hectares in area were required to hold the hearings). This study exploited the discontinuity around the 25 hectare mark and compared the mines marginally above 25 hectares with the ones marginally below 25 hectares. The final sample included 134 mines, of which 68 were treatment mines (<25 hectares) and 66 were control (>25 hectares). Using data before and after the EC applications, they estimated a difference-in-difference model.

This study, utilising web scraping to collect data on project characteristics and various sources of satellite data for measuring the outcomes of interest, is an excellent example of innovative data collection methods in a sector where the evidence base is very small (Rathinam et al., 2019b).

Using CRD analytics to inform disaster management

In this section, we briefly outline the process for undertaking CRD analytics to measure, characterise and predict population displacement and returnee/resettlement patterns in post-disaster settings. While applications of CRD data analytics to date have lacked an evaluative component, their potential in this regard is evident. We draw on a recent project at Flowminder, which revisited three sudden-onset disaster events to investigate drivers of displacements (individual and contextual) and the feasibility of predicting displacement locations from CRD data and data on disaster intensity and damage, on population density and on the humanitarian response. The three events were the 2010 earthquake in Haiti, the 2015 Gorkha earthquake in Nepal and the 2016 Hurricane Matthew in Haiti.

Flowminder has long-established partnerships with the major mobile phone network operators in Haiti and Nepal. Historically, data access has been a major barrier to the scale-up of CRD analytics for humanitarian and development applications. Mobile network operators (MNOs) are justifiably hesitant to authorise third-party access, given the need to safeguard subscribers' personal data.

Prior to analysis, MNO data underwent a long series of cleaning and preprocessing steps as part of quality assurance and to support the generation of standardised metrics. A first stage of analysis was undertaken to structure the data in a usable format and to detect data anomalies. Once data was cleaned, quality assured and converted into an analysable format, a number of preliminary processing steps were undertaken, including:

Clustering of cell tower locations

MNO data: Preprocessing steps

Here are some commonly occurring issues in MNO data. Once identified, corrections and/or accommodations can be performed prior to and/or during the preliminary processing and analysis phases. 1.

Standard data quality issues applicable to MNO data:

The processing steps undertaken to discern at individual level disaster-induced displacements from pseudonymised, time series CRD data are presented below in Figure 13.

OPEN IN VIEWER

Figure 12

Number of studies reporting on ethics issues

OPEN IN VIEWER

Figure 13

Steps in CRD processing for displacement and return/resettlement/recovery pattern analysis. CRD, call record detail

The analysis disclosed striking commonalities in internally displaced person (IDP) return/resettlement rates, with the fraction of IDPs who remain displaced exhibiting a common rate of decay across all three post-disaster settings studied.

In a further step, the team developed new mobility and social network metrics to permit analysis of the relationships between contextual and individual variables and displacement duration, distance and trajectories, controlling for the severity of impacts and humanitarian response. The results suggest that the dispersal of an individual's social contacts and travel history predisaster are highly predictive of their post-disaster displacement trajectories. Individuals with localised travel patterns and social contacts were more likely to be displaced in the vicinity of their usual residence compared with those with more dispersed travel patterns and social contacts. A majority of IDPs remained within a 10 km radius of their usual place of residence. Across the three disasters, 60–70% of long-distance displacements (in excess of 100 km) involved travel to a familiar location and/or proximate to one or more contacts discernible in the predisaster CRD data. This pattern holds controlling for the severity of impacts at local area level and is consistent across all three disasters.

Results were validated via comparisons with reports retrospectively quantifying population displacements produced by the International Organisation for Migration, as well as with reference to data on the intensity of each disaster's impact on affected areas. The results indicate that CRD data analysis can be used to predict the estimated number and spatial distribution of IDPs at different time points based on initial estimates of the number of persons displaced in the immediate wake of a disaster, as well as to predict recovery/resettlement timelines. This has important implications for post-disaster humanitarian response and resettlement efforts. The same methods can support disaster resilience assessments and planning and provide a means to compare recovery and resettlement rates across different disaster events.

Satellite data also presents misclassification problems

Researchers, however, point to several technical challenges in using satellite images that may provide misleading conclusions. For example, Jain (2020) argues for the need for ground validation of satellite data as sometime the images could be misclassified (e.g., flood irrigation may be classified as flooded area); often this misclassification is systematic (ie forest cover is almost always misclassified as agriculture, which will bias the study results). This can be rectified with a field visit. Further, there could be differences in the data coming from different satellites and from the same satellite constellation but using different sensors (e.g., Normalised Difference Vegetation Index varies for different satellite sources and for same satellites across different versions, such as Landsat 5 and Landsat 8). Another example of what qualitative field visits could contribute to improving the interpretability of satellite data are the indicator “quality of roof construction” as a proxy for economic development. Straw roofs from satellite images are generally classified as low-quality; zinc and other hard roofs are classified as a sign of development. However, several factors may bias this classification: in hot climates, straw roofs may be preferred for their ability to keep cool compared to other hard roofs; poor families may have received donations of high-quality roofs; or lack of tenured security may discourage people invest in immoveable roof-tops despite increase in income.

Data quality and transparency is paramount

The map also points to the need to set standards for better reporting, as about 87% of the measurement studies did not report on data quality issues, representativeness, construct validity and generalisability. This would lead to questioning the internal and external validity of the findings. There is also a need to set standards for data transparency, taking into consideration the challenges in sharing proprietary data, data storing and the capacity of the Dataverse (see Box 1 for more details on data transparency).

Ethical concerns are substantial

Ethical challenges such as consent, data privacy, data security and unintended exclusion are well documented in the literature (Lokanathan et al., 2017; York & Bamberger, 2020). A brief analysis of the studies on the map shows that very few studies report on any of these ethical challenges. However, the challenges are different for different sources of big data. For example, satellite data that involves little human interaction may not need an IRB review but most other big data source that use human-generated data without explicit consent for secondary use should be reviewed by IRBs. We also recommend more mixed-method big data evaluations to mitigate the potential disconnect between development stakeholders and big data researchers. Any mixed-method research needs to be reviewed by IRBs.

The map shows that most IEs have done well on reporting data quality issues but not on ethical issues. Since big data involves ethical issues (such as consent for secondary data use and unintended exclusion) that are new to conventional ethical standards, there is a need to update the current ethical standards practice to include big data use as well.

Big data may be growing in use and popularity, but the need for independent auxiliary data for “ground-truthing” remain

Many sources of big data are partial in terms of coverage and prone to biases that are difficult to measure, control and correct for in the absence of secondary data. Despite growing awareness and acknowledgement of its limitations, the household sample survey remains the dominant source of development policymaking. Big data often require survey data as “ground-truth” data to validate the findings. Demographic Health Surveys and Living Standards Measurement Studies are the two main surveys used in ground-truthing. There is considerable scope for merging the income and expenditure surveys, and food surveys conducted in several developing countries with big data to assess food shortages, poverty hotspots, and so forth.

Some capacity constraints are acute

Development organisations need to build staff capacity in order to use big data as a strategic asset (Perera Gomez & Lokanathan, 2017). They need to build multidisciplinary teams consisting of data experts and subject matter professionals, and also compete with the private sector to recruit the staff. Other major costs involve scaling up the technical infrastructure to enable data storage and processing on a large scale and data accessibility costs. The latter can be more difficult to predict considering that big data sources that are currently public may involve licensing in the future. Besides, ensuring the sustainability of data can be a cause of concern. As suggested by Hammer et al. (2017), as most of the big data is produced as a by-product by the private sector, continuity of data provision cannot be guaranteed in this age of evolving technology and market conditions. These concerns will call the wisdom of committing resources upfront to build capacity into question.

Need for better coordination between data scientists and evaluators

Big data analysts and evaluators use different framework and analytical tools. In particular, the big data measurement studies look for hidden patterns in the data with little support from theory and aim at prediction rather than causality (York & Bamberger, 2020). Further, the expertise needed to analyse big data remains largely localised and siloed. Outside of a small and highly specialised group of data scientists, there is uncertainty about how best to carry out large-scale big data analysis. The degree of technical specialisation combined with strict access restrictions to many types of big data has hindered big data applications in development evaluation. Hence, there is a need to promote interaction between development evaluators and data scientists for better cross-learning and adoption of big data in measuring and evaluating development outcomes.

The cost of collecting, analysing, storing and reporting big data is largely unknown

There is very little publicly available information on the cost of collecting, analysing and reporting big data. Blumenstock et al. (2015) reported that the phone survey for ground-truthing the CRD data costed USD 12,000 and took 4 weeks to administer. This is, however, only the variable cost of data collection in this study. There are multiple hidden costs such as staff costs and the cost of the necessary computing infrastructure (including storage); in addition, the opportunity cost of time involved in developing partnerships with data providers in some cases is not known. BenYishay et al. (2017) report that the cost of geospatial IE is around USD 150,000. One of the 3ie funded studies using satellite data are reported to have spent USD 3300 on data collection, which is about 1% of the total study budget, but have spent USD 103,864 on data analysis and reporting (about 32% of the total cost

Evaluation of potential methodological issues

While big data can help resolve many data related challenges, there are considerable methodological, analytical, logistical and ethical challenges in the way of using it in measuring development outcomes (Letouzé, 2016; Lokanathan et al., 2017; Olteanu et al., 2019; Salganik, 2017). This section briefly discusses some of the prominent methodological challenges. As big data is varied in type, quality and composition, we also discuss if the challenges are specific to any particular type of big data.

We have grouped all the big data challenges that may affect measuring development outcomes credibly and lead to questionable internal and external validity of the studies.

Evaluating reporting on privacy and ethical considerations

There are concerns over data access, privacy, consent and ethics in using big data. Although these are foundational issues for both small and big data studies, the challenges posed by big data have greater repercussions.

When using big data sources such as mobile data, most mobile operators have “inform and consent” policies that mandate disclosure of all relevant information to potential participants who can then evaluate this information and give explicit permission. However, these policies often contain legal language that is generally not discernible, and it is not clear if explicit consent is obtained to repurpose the data. This kind of informed consent may be completely absent in research leveraging social media data due to the impracticality of obtaining consent from millions of users.

Mobile phone user data and social media data are some of the most used sources of big data that can inform researchers about individuals' behaviour. Even if the data are de-identified, concerns still remain over the consent and ethics of sharing such data with researchers. It is thus imperative to have an ethics approval process in place that lays down the conditions under which such research can take place. There is a need for clear ethical standards for big data research and studies should be monitored by the IRBs.

Another ethical criterion when using big data can be concerned with the assessment of risks, the most common being privacy breaches leading to identity theft or other cybersecurity risks. The possibility of the reidentification of any individual user from poorly anonymised datasets adds to the concerns over anonymity of subjects. When combined with other sources, such datasets can be used to gain detailed insights about people without their knowledge. Such precise inferences may create the capacity for discrimination or mass manipulation. Sometimes data obtained for one purpose in social data research is used for secondary analyses, but the associated risks may not be well understood. For example, Facebook data in the past has been used for ad targeting, as well as for tailoring propaganda (Horowitz et al., 2018).

Big data may also inadvertently exclude certain sections of the population. For example, this bias can be observed in the case of “Street bump,” a mobile app that notifies the Boston City Hall whenever the user hits a bump on the road (Carrera et al., 2013). The data includes information only from the app users who often use both their cars and the app; this might inadvertently exclude poorer parts of the city that app users may not frequent. Policy based on such big data sources may have unintended consequences for the people who are excluded.

We assessed the studies on the following:

Ethical approval obtained

Reporting on privacy and ethical challenges

Figure 12 shows that most studies (81%) do not report on ethical challenges and privacy issues. Of the few that do discuss such challenges, the most frequently discussed issue is consent for data use.