Sustainable and inclusive – Evaluating urban sustainability indicators’ suitability for measuring progress towards SDG-11

Challenges of urban indicators

Indicators are defined in a myriad of ways, but they represent the “past or projected performance of different units,” and are generated through a “process that simplifies raw data about a complex social phenomenon” (Davis et al., 2015: 4). They have been applied in diverse contexts to measure environmental performance and sustainability, and are typically aggregated from multiple data sources using statistical methods to form indices, or complex indicators (Ness et al., 2007). Indicators can provide a powerful basis for political decision-making, building public awareness, and assessing the ways in which fundamentally environmental problems are conceptualized and their solutions determined (Davis et al., 2015). In urban areas, indicator systems have increasingly resonated with decision-makers and managers, in large part due to what Kitchin et al. (2015) refer to as the rise of “new managerialism” within cities. This trend emphasizes the instrumental nature of indicators that are often used to guide practices, provide evidence of performance, or inform policies or decision-making (Hezri and Dovers, 2006; Kitchin et al., 2015). This type of “dashboard governance” allows urban managers to collect and integrate statistical information, including traffic control, weather conditions, and municipal and utility services (Mattern and Mattern, 2015).

Indicators in cities are now a commonplace means of tracking performance and guiding policy formation, to the point where they have become “normalized as a de facto civic epistemology through which a public administration is measured and performance is communicated” (Kitchin et al., 2015: 7). One study suggests that there are now 200 city benchmarking initiatives that aim to compare hundreds of cities globally (Moonen and Clark, 2013). These include indices that strive to rank the economic competitiveness of cities, such as the Global City Index (Kearney, 2019) and the Global City Power Index (Mori Memorial Foundation, 2018), to others that are more specifically focused on environmental performance, including the Siemens Green City Index (EIU, 2012). These indices typically rank different aspects of cities that are weighted and then aggregated into a single composite score. Despite the growth in indicators and indices in urban areas, meaningful and comparable indicators are difficult to achieve due to the complexities embedded in defining sustainability (Bohringer and Jochem, 2007 Cooper, 1999) within the urban boundary (Alberti, 1999; Bagliani et al., 2008; UNHSP, 2004). Mori and Christodoulou (2012: 95) suggest that existing indexes do not allow for the ability to “assess and compare cities’ sustainability performances for understanding the local and global impact of cities on the environment and human life.”

Another challenge facing urban indicators, and in particular SDG-11, is with respect to their generalizability or comparability across, at times, vastly different contexts. While all SDGs face monitoring challenges, SDG-11’s subnational and spatial focus adds complexity because residents’ experience of SDG-11 indicators can vary significantly within the city, and are often shaped by neighborhood-specific socio-spatial factors (Ulbrich et al., 2019). Many cities that do attempt to collect data for the SDGs, by leveraging their existing monitoring systems or applying SDG indicators to local contexts, capture the information most relevant to local priorities and politics, which does not necessarily allow for comparability across cities (Pfeffer et al., 2019). In Germany, for instance, Koch and Krellenberg (2018) assessed three initiatives focused on translating and tracking SDG-11 and found limited comparability between each initiative’s indicators; difficulties in data disaggregation; and a lack of sustainability indicators by German cities themselves. Similarly, Simon et al. (2015) assessed pilot SDG-11 indicators in Bangalore, Cape Town, Gothenburg, Greater Manchester, and Kisumu. Each city struggled to access sufficient data to track the indicators, and all proposed changes that would make them more relevant to their unique context. In other words, it is difficult not only to track a city’s progress towards SDG-11, but to compare and assess progress towards this goal at a global scale.

Reflecting environmental equity in urban sustainability measurement

SDG-11’s focus on the distribution of environmental harms (e.g. exposure to air pollutants) and access to environmental benefits (e.g. urban green spaces and transportation) across different demographic categories suggests that this goal primarily measures the distributive dimensions of environmental equity. Distributive environmental equity emphasizes the allocation of social, economic and political goods, costs, and privileges among demographic groups, including groups defined by age, income, race, education, gender, employment status, or social groups and districts (Agyeman and Evans, 2004; Rawls, 2009; Schlosberg, 2009: 12–13).

Researchers also often emphasize the importance of two additional dimensions of environmental equity – procedural equity, or the fair access of different demographic groups to the decision-making processes (Agyeman and Evans, 2004; Reckien et al., 2017; Schlosberg, 2009) and justice as recognition (Reckien et al., 2017; Young, 1990), the influence of pre-existing political, economic, and social conditions on the crafting and implementation of environmental policy. Like SDG-11, however, many existing urban environmental indexes primarily consider the distributive dimension of environmental equity, likely because it is often easiest to quantitatively measure (Reckien et al., 2017). We focus on our analysis on this dimension of environmental equity, while noting procedural equity and justice as recognition are also important elements of understanding environmental in/equity in urban areas (see, for instance, Kopnina (2016) for a critical discussion of whether the SDGs successfully engage with global unsustainability challenges).

Although attempts to measure distributive equity are more common, they carry their own challenges and complications. The notion of distributive equity, by definition, implies that environmental benefits and burdens should be placed equally among individuals. In practice, however, any marginal exposure to an environmental harm (i.e. pollution) for a vulnerable population could be detrimental (Maguire and Sheriff, 2011). For example, low-income populations often have less access to medical care, alternative water sources, or housing options that can enable individuals to avoid or reduce the impact of exposure to environmental harms (Maguire and Sheriff, 2011). Distributing environmental burdens equally across all communities may still disproportionately impact marginal populations. Fainstein (2014), in her comparison of spatial justice in the highly unequal cities of New York, Amsterdam, and Paris, argues that inherited advantages mean that a just distribution of public benefits should favor the disadvantaged. While we note these caveats around the application and measurement of distributional equity, we adopt this concept as our primary way of evaluating if and how existing urban environmental indicators account for equity in ways that could be operationalized to track progress towards SDG-11.

Index and indicator selection

To examine the landscape of existing urban sustainability indicators and the extent to which they could be applied to assess equity and consequently SDG-11, we reviewed 41 indexes (Table S2) – that is, composite urban sustainability frameworks that include multiple indicators – that contained a combined total of 484 indicators. We conducted a “horizon-scanning” (e.g. Joss, 2011: 270) exercise to identify and characterize indexes. We used keywords such as “urban sustainability index” and “urban environment index” in Google searches to identify candidate indexes. We conducted the search between January – July 2016 and repeated it in January 2018, searching for urban indexes available in English that included indicators of environmental assessment or sustainability. This analysis is not intended to be a comprehensive review; instead, we stopped collecting indexes when we felt we had reached a wide diversity of concepts covered in the indicators. The selected indexes assess geographic areas ranging from neighborhoods to mixed urban-rural regions. They were created by organizations including: NGOs, academic institutions, governments, development agencies, and consulting companies (see Table S2 for a full list of the indexes reviewed).

The goal of reviewing these indexes is ultimately to understand what individual urban sustainability indicators measure in terms of environmental issues; if they address equity and at what scale; and the data sources they rely upon. Urban sustainability indexes cover a wide range of topics and often include economic and social indicators alongside environmental or ecological indicators. The analysis focuses on individual indicators included in the composite indexes, rather than the indexes as a whole. In cases where indexes included a mix of environmental, social, and economic indicators, we selected indicators that measure environmental quality (e.g. air pollution), or human activities that affect environmental quality (e.g. vehicle miles traveled) in urban areas. Social and economic indicators, such as population density and unemployment rate, that did not directly measure an environmentally related dimension of SDG-11 were thus excluded from our final analysis. However, we did consider social and economic indicators when deciding whether environmental indicators are “able to assess equity” or not, even if the index authors did not explicitly calculate equity distributions. This process is further discussed in the section on qualitative coding below. We further aimed to identify indexes with adequate metadata (e.g. information or data about the data itself) to allow us to analyze the characteristics (e.g. target type, presence or absence of baseline) of the sampled urban sustainability indicators.

Qualitative coding

We applied qualitative coding to identify key aspects of each indicator, including: the origin and geographic scope, an indicator’s environmental focus issue, the target type, whether the indicator related to equity, the unit of analysis, and the type of data source. Four different researchers evaluated indicators across these categories and each of their resulting lists were compared. Discrepancies between the lists were resolved by group consensus.

Geographic origin and coverage

Figure 1 maps the corresponding country of the index creators, as well as the country where the cities assessed by these indexes are located. The points on the map are shaded according to the United Nations’ 2016 classifications for their World Economic Situation and Prospects report (UN, 2016), based on 2014 GNI (these dates were chosen to correspond with the timing of our data collection). Only 24 of the 40 indexes publish the countries where they have been applied, and these applications are run by organizations located in just 9 high-income countries. These include multiple indexes from the United States (14), Canada (3), and one each from Australia, Belgium, Korea, Netherlands, Sweden, United Arab Emirates, and the United Kingdom. These indexes cover 114 countries, primarily developing countries (47 countries included), followed by developed countries (34), least developed (23), and countries in transition (10). The most prevalent types of index creators vary by location: government indexes often measure urban sustainability in Europe; consulting firms more often include developed country cities in their indexes in Europe, Latin America and Asia; and NGOs include cities from Africa and Southeast Asia. As Figure 1 shows, urban sustainability in developed countries and countries in transition is often assessed using indicators developed by a mix of governments, consulting companies, and NGOs. Our sample did not include any government-created indexes from developing and least developed countries; instead, NGOs and consulting companies primarily track urban sustainability in these regions.

OPEN IN VIEWER

Figure 1.

Country classifications for the index creators. The size of each node (i.e. degree) represents the number of connections between countries, with larger nodes representing those with more connections. The direction of the arrows indicates the countries included in indexes, while the origin is the headquarters country of the index organization. The color of the edge between two points indicates the sector of the index organization.

Environmental issue areas

Our analysis found that several core issues occurred most frequently across the reviewed indicators. Figure 2, which shows the frequency of indicator issue areas in the sample, demonstrates that six issue areas are observed much more frequently than others. We label these core issues for urban sustainability indexes: water and sanitation, climate and energy, land use and built environment, transportation, waste management, and air quality. The least frequent core issue area – air quality – was still nearly twice as frequent in our sample compared to the next most frequent issue area, urban food, and agriculture. For some of the analyses described below, we focused only on the core issue areas.

OPEN IN VIEWER

Figure 2.

Indicators by issue classification and index creators. The most prevalent categories include: water and sanitation (n = 82), climate and energy (n = 77), land use and built environment (n = 60), transportation (n = 56), waste management (n = 40), and air quality (n = 37).

We also reviewed the extent to which each indicator could measure one of the SDG-11 targets or indicators, summarized in Figure 3. Out of 484 indicators we surveyed, about 40% (194) could potentially be leveraged to measure an SDG-11 target or indicator. The indicators were most relevant to Target 11.6, “By 2030, reduce the adverse per capita environmental impact of cities, including by paying special attention to air quality and municipal and other waste management” (77 indicators) and Target 11.2,

OPEN IN VIEWER

Figure 3.

Frequency of SDG indicators and targets as represented in reviewed indicators. (see UN DESA n.d.) for full indicator and target names.).

“By 2030, provide access to safe, affordable, accessible and sustainable transport systems for all, improving road safety, notably by expanding public transport, with special attention to the needs of those in vulnerable situations, women, children, persons with disabilities and older persons” (47 relevant indicators).

The indicators were also more applicable to SDG-11 targets (157) than to the more detailed and specific SDG-11 indicators (37).

Targets and baselines

We use targets and baselines, which are often used for performance management and goal-setting, to assess the degree to which urban sustainability indicators could be used for management. Only 43% of reviewed indicators had either specific or directional targets, leaving the remaining 57% with ambiguous management objectives. Figure 4 shows how the presence of different kinds of targets and baselines varies across different issue areas. Core issue area indicators have a similar pattern: many indicators with no target and no baseline, or indicators with only a directional target. Water and sanitation (8), land use (7), and air quality (6) indicators have the largest number of targets with a baseline. The perceptions and participation category has the largest percentage and absolute number of indicators that fall into the “Has Baseline Only,” category. For example, the Central Texas Sustainability Indicators project includes a series of indicators that gauge citizens’ knowledge about water consumption, including indicators such as, “percent of people who ‘definitely know’ the source of their drinking water” and report a baseline percentage of survey respondents who answered positively. The index, however, does not specify what a target is, whether 100% of respondents answering positively is a goal, or otherwise (see Table S2 for additional examples of indicator targets and baselines).

OPEN IN VIEWER

Figure 4.

Comparing the frequency of target type by issue area reveals that many of the indicators reviewed do not include a target. A visual examination of the table suggests that there are different patterns for some of the issue areas.

Equity indicators

Of the 484 indicators we evaluated, only 61 (13%) attempted to assess distributional equity, the only form of equity we assessed. Figure 5 illustrates the proportion of indicators in each issue area that was determined to be an equity-related indicator because it either disaggregated the indicator by a demographic group, or set a goal to ensure equity in distributional impacts for all populations (e.g. clean air quality for all). The environmental health category has the greatest proportion of equity indicators; 64% of the assessed environmental health indicators also relate to equity. Air quality (28%) and water and sanitation (18%) had the second and third highest proportion of equity-related indicators. The climate and energy, biodiversity and habitat, and natural hazard vulnerability categories had the fewest equity-related indicators.

OPEN IN VIEWER

Figure 5.

Percentage (count) of equity indicators by environmental issue area.

Of the equity indicators that specify a unit of analysis, 40% relate to the neighborhood level, making neighborhoods the most common unit of analysis for equity indicators. Twenty-four percent (23) of neighborhood-focused indicators relate to equity. Equity indicators tend to have a smaller unit of analysis than other indicators that are not related to equity, supporting the conclusion that spatial disaggregation enables measuring equity.

Data sources

Very few indicators included clear information about their underlying data sources. Many of the data sources for the indicators we could assess (67%, 114) were collected and published by government agencies (Cluster A indicators), while 33% (56) could be identified as using alternative methods (Cluster B indicators). The remainder (314) could not be assessed regarding the type of source data (Cluster A or B). Of the indicators we reviewed, 90 (19%) used open data sources. Regardless of Cluster A or B sources, publicly available data (i.e. open data) are more likely the source for equity indicators. Over half of the equity indicators were derived from open data sets, even though open data account for only a quarter of the indicators.

Looking ahead

Despite the rapid increase of urban indexes, it is clear that greater investment in data collection for SDG-11 will be needed for cities to track progress towards SDG-11. Our findings are consistent with Klopp and Petretta (2017)’s conclusions that (1) the poor availability of standardized, open, and comparable data; (2) the lack of strong data collection institutions to support monitoring at the city scale; and (3) the challenges of “localizing,” the SDGs in widely different cities all hinder monitoring for SDG-11. At least 10 out of the 15 SDG-11 indicators require new monitoring approaches and tools for collecting, analyzing, and using information (UN Habitat, 2018), and even for cities in the Global North, capacity for data collection is low (Simon et al., 2015). Cities are also likely to face considerable challenges collecting data disaggregated by age, sex, and ability, as proposed in SDG-11, and acquiring more detailed datasets may be unrealistic given the current state of data collection and management administration. This challenge is especially true in developing countries, where institutional capacity and poor transportation and information technology infrastructure, among other factors, limit data collection (Simon et al., 2015). Spatially-explicit, disaggregated data will be particularly critical to locate spatial patterns and inequalities and identify policy priorities that can be more effective in targeting specific areas and social groups (Pfeffer et al., 2019; Ulbrich et al., 2019).

Where not available through official government sources, sub-city or neighborhood-level data could be developed from alternative data sources, such as citizen science, crowdsourcing, low-cost sensors, or commercial/private sector data (Fritz et al., 2019; Hsu et al., 2014). Open datasets such as Open Street Maps and earth observation data represent promising sources of spatially-explicit information that could be adapted to measure urban-scale environmental performance, though they require technical skill and capacity to obtain and analyze (Andries et al., 2019; Kuffer et al., 2018). Technology-based citizen engagement efforts, however, also need to carefully weigh potential ethical and privacy considerations (MacFeely, 2019). The Urban Environmental and Social Inclusion Index (Hsu et al., 2018), for instance, represents emerging research to measure progress on SDG-11’s urban environment and equity dimensions in a spatially-explicit, neighborhood-disaggregated way using open data sources and satellite remote sensing.