Abstract
Quantitative information is one of the means used to interface science with policy. As a consequence, much effort is invested in producing quantitative information for policy and much criticism is directed toward the use of numbers in policy. In this paper, I analyze five approaches drawn from such criticisms and propose alternative uses of quantitative information for governance: (i) valuation of ecosystem services, (ii) social multicriteria evaluation, (iii) quantification of uncertainty through the Numeral, Unit, Spread, Assessment, Pedigree approach, (iv) Quantitative Story-Telling, and (v) the heuristic use of statistics. The analysis shows the varied ways that numbers are conceptualized and how different conceptualizations matter for the science–policy interface. Alternative conceptualizations of numbers are used to challenge the model of science-speaking truth to power. Uncertainty, complexity, pluralism, malpractice, and values are mobilized to redefine the relations between science and policy. Alternative quantification may produce alternative facts, but reflexive approaches that use numbers to discuss the relevance of equity, positionality, and quality in science for policy may offer a remedy.
The use of quantitative information is central to the science–policy interface and has been extensively studied. According to Hacking (2006), the emergence of statistics shifted the understanding and governing of events from causal determinism to the understanding and governance of statistics. Asdal (2008) illustrates how statistics has been a central technology in the accounting of pollution. Policymaking has relied on quantification as a means of governance, and quantification has influenced the means and objects of governance.
The use of numbers gives an illusion of certainty and precision that is often at odds with the uncertainty that characterizes many complex and multidimensional policy issues, such as sustainability, poverty alleviation, and the regulation of global finance, to name but a few examples. A flourishing literature denounces a variety of issues associated with excessive faith in numbers (Porter 1995; Saltelli and Funtowicz 2014), failures in making predictions about the future (Pilkey and Pilkey-Jarvis 2007; Taleb 2007; Savage 2009), the limited usefulness of idealized mathematical models in dealing with complexity and uncertainty (Fine and Milonakis 2011; Chu 2013), and the misuse and misinterpretation of quantitative information in policymaking (Funtowicz and Ravetz 1990; Reinert 2009).
Porter (1995) points out that the ideal of objectivity associated with numbers has created the perception that quantitative science can be trusted. Saltelli et al. (2013) argue that the authority of science is associated with the rhetorical use of quantitative information, which is used as Latin was used in the Middle Ages to confuse rather than to clarify. This criticism points to cases in which models are used because they are mathematically sound and not because they are useful. Fine and Milonakis (2011) criticize economics for being “useless but true.” They argue that economic models are mathematically valid but lack any practical relevance because of the large number of assumptions on which those models are based, such as perfect knowledge, perfect competition, complete preferences, ceteris paribus, and so on. Pilkey and Pilkey-Jarvis (2007) also speak of useless arithmetic in relation to environmental science. The object of contention is the use of quantitative information in guiding policy, building trust, and constructing authority, not the technologies or the scientific knowledge itself.
Different reactions have emerged from criticisms of the use of quantitative information for policy, which for the purposes of this paper are divided into approaches that move away from quantification and approaches that seek alternative ways of engaging with quantification. Decision-making models that move away from quantification are, for example, the precautionary approach, participation models, and social, legal, and ethical assessments. These approaches suggest alternative criteria upon which decisions should be based when quantitative information is not conclusive, reliable, or certain. The second type of approach seeks alternative ways of using quantification that do not dismiss the use of numbers but redefine their role in decision-making processes.
I use the term science–policy interface to refer to how science and policy relate to each other. I analyze scientific activities aimed at guiding, informing, or assessing policy, and governance processes that use scientific information of some sort. As I will argue throughout the paper, the character of numbers plays a constitutive role in this relationship. What is at stake in the conceptualization and use of numbers is the ideal of science speaking truth to power (Wildavsky 1989). Science and technology studies (STSs) have long taken a critical stand toward the positivist conception of numbers as representations of the world and have shown how numbers often constitute the objects of governance and are as much part of policymaking as values. I add to this discussion by asking how nonpositivist uses of numbers matter for governance.
In this paper, I analyze five alternative approaches to quantification that have emerged from STS-inspired criticisms. The approaches considered are (i) valuation of ecosystem services, (ii) social multicriteria evaluation, (iii) the Numeral, Unit, Spread, Assessment, Pedigree (NUSAP) system for quantification of uncertainty, (iv) Quantitative Story-Telling, and (v) the heuristic use of statistics. The five approaches have been chosen because they engage, in their practice or in their theoretical background, with some of the recurrent themes of STS such as the fact-values duality, the concept of expertise, and the issue of symmetry and demarcation.
I argue that different approaches to quantification give a different character to numbers, which affects how numbers are discussed and used. Particularly, I show how the character of a number can open or close the policy option space. The positivist use of numbers perpetuates the modus operandi and authority of existing institutions, as pollution accounting remains linked to technical expertise (Asdal 2008) and the prominent role of experts in science advice. The approaches analyzed in this paper opt for a heuristic use of numbers, and in so doing they invite a reflexive use of scientific information and a more humble and care-oriented engagement with policy processes.
This paper has two objectives: to review and analyze five approaches to quantification from the literature and to compare what bearing these approaches have on the numbers produced and how they influence the relationship between science and policy. I first give an overview of alternative quantifications used for policy. I then assess the conceptualizations of numbers and the science–policy models that emerge from the different uses of numbers presented. The conclusion summarizes how new definitions of numbers emerge from alternative quantifications and discusses the uses of quantitative information for the science–policy interface. When the debate is limited to values, alternative quantification methods may give rise to alternative facts. The quantification of pluralism, uncertainty, and complexity does not associate numbers with facts. In so doing, the debate shifts from the question of which type of facts should be used for policy to the questions of how to deal with issues of equity, positionality, and quality at the science–policy interface.
Alternative Approaches to Quantification
This section analyzes the conceptualizations of numbers that emerge from alternative practices of quantification specifically aimed at the production of quantitative information for governance. Five approaches to quantification are analyzed: valuation of ecosystem services, social multicriteria evaluation, the NUSAP system for the quantification of uncertainty, Quantitative Story-Telling, and the heuristic use of statistics. The first four approaches are applied mainly to environmental and sustainability issues, while the heuristic use of statistics is explained in reference to votes tallying. For each approach, I give a practical example and analyze (i) the methodology and the use of numbers, (ii) the main challenge that the approach aims to solve, and (iii) the theoretical basis of the approach and its relation to the conceptualization of the science–policy interface.
Valuation of Ecosystem Services
Valuation of ecosystem services has been widely studied in STS (Turnhout, Hisschemoller, and Eijsackers 2007; Asdal 2008; Verran 2013). One of the many practical examples of this approach is the use of valuation for reducing emissions from deforestation and forest degradation (REDD+). REDD+ was proposed by the United Nations Framework Convention on Climate Change and is an interesting example of economic valuation of ecosystem services aimed at influencing public policy with regard to forest management. There are five types of activities that are eligible for funding: (i) reducing emissions from deforestation, (ii) reducing forest degradation, (iii) conservation of forest carbon stocks, (iv) sustainable forest management, and (v) increase of forest carbon stocks. The latter three elements constitute the + in REDD+. Quantification is central to REDD+, as emissions, carbon stocks, and forest quality all need to be measured and monitored to implement the policy.
In general terms, different methods are used in the valuation of ecosystem services with the purpose of assigning them monetary value including the travel cost method, the contingent valuation, the use of Pigouvian taxes, and the creation of markets through “cap and trade” policies (de Groot, Wilson, and Boumans 2002; Pirard 2012). Markets are seen as the most efficient means of allocating scarce resources because they use prices to mediate the willingness to pay of different economic agents. STS scholars have highlighted how valuation becomes the object of governance itself (Asdal 2008; Helgesson and Muniesa 2013).
The use of economic tools for valuation assumes that values can be expressed or conveyed in monetary terms, but differences emerge about whose values are taken into account. In the travel cost method and contingent valuation, values are determined at the individual level, expressed as willingness to pay and individual preferences. In cap and trade schemes and REDD+, ethical values related to the need to protect the environment are used to determine limits to pollution levels and deforestation. Caps are used to determine the socially desirable level of pollution.
Emission accounting in REDD+ is a fundamental tool for advancing awareness of the urgency of climate change and for monitoring measures taken to mitigate and adapt to climate change. The valuation of ecosystem services tries to solve the challenge of problem framing in science–policy interactions. In the example of REDD+, climate change is mobilized as the most pressing sustainability issue and rendered as a policy priority. Framing may refer to the problem definition (climate change), to the approach and measures used to solve the problem (carbon accounting and monitoring), or to the scientific approach used to produce knowledge about the problem (valuation of ecosystem services).
The valuation of ecosystem services questions the choice of facts used to inform or guide policy and argues for an explicitly normative approach to the production of scientific facts, based on values such as environmentalism, sustainability, conservationism, and so on. This approach can be associated with standpoint epistemology in philosophy of science. Standpoint epistemology engages with the idea of value-laden knowledge to defend a certain set of values (e.g., feminism or environmentalism). Standpoint epistemology argues that the sciences have been blind to their own framing biases (Harding 1993). In this case, revealing such blindness is seen as a way to restore the appropriate use of scientific knowledge, where appropriate is often cast as ethical. In the REDD+ example, climate change discourse is permeated by ethical claims. Following Verran (2013), numbering is a means of ordering in REDD+.
The relationship between science and policy is interpreted as one in which scientific truths carry ethical guidance for power to act. This approach does not challenge the concept of “power” and may thus stabilize existing power relations and policy institutions. A recurrent criticism of REDD+ is that it may lead to the commodification of nature, thereby stabilizing market mechanisms and neoliberalism as means of governance (Kosoy and Corbera 2010; Gomez-Baggethun and Ruiz-Perez 2011). Values enter the science–policy interface in the form of universal values, defined in ahistorical, apolitical, and asocial terms.
In comparison to the linear model of science-speaking truth to power, this approach introduces the novel idea that the choice of policy-relevant facts is explicitly determined through value judgments. Science and value judgments are collapsed into one inextricable activity. Science retains its role of fact producer and provider of expertise, albeit with an increased awareness of the ethical consequences of the use of its facts.
Social Multicriteria Evaluation
Munda (2005, 2004a, 2004b) introduced social multicriteria evaluation as a decision support tool in ecological economics. The approach consists of an iterative learning process through which different policy alternatives are identified and assessed against a range of criteria and in relation to a set of social actors, which are identified through institutional analysis and which may be expanded in successive iterations (De Marchi et al. 2000). Different stakeholders are asked to rank the alternatives presented according to multiple criteria. The ranking can be done either in quantitative terms, by assigning scores, threshold values, or quantitative indicators, or in qualitative terms, by expressing preferences. The multicriteria assessment consists in pulling together the plurality of scores given to the different criteria in a multicriteria matrix. The assessment of the different alternatives depends on the weighting factors used to aggregate the criteria. Social multicriteria evaluation uses two main tools for assessment: a multicriteria impact matrix for policy options (Figure 1) and social equity matrix for the assessment of power relations and possible coalitions among social actors (Figure 2).
Example of multicriteria impact matrix.
Example of social equity matrix.
Social multicriteria evaluation has been applied to the assessment of water use in the town of Troina, in Sicily, Italy (De Marchi et al. 2000). Water management was associated with unfair use of water resources and lack of transparency in management. Social multicriteria evaluation was deployed to assess possible policies including an information campaign, changes in management, and changes in water pricing. The rankings derived from the multicriteria impact matrix yielded different results from those of the social equity matrix, indicating that policy solutions were not just a matter of managing preferences but also of understanding power relations among actors.
Social multicriteria evaluation addresses the incommensurability associated with the plurality of values that may guide decisions. Incommensurability is defined at two levels: with regard to the multiple criteria that social actors may use to assess the desirability of policies and with regard to the plurality of social actors (Munda 2004b). This approach highlights the impossibility of defining what is good and what is desirable in universal terms and drives attention to the need to reconcile the interests and needs of a plurality of social actors in policymaking. Impossibility is defined with reference to Condorcet’s impossibility theorem, as the existence of not perfectly ordered preferences in social aggregates. The theorem refers to situations in which three individuals (1, 2, 3) have to decide over three options (A, B, C) with their preferences being 1(A, B, C), 2(B, C, A), and 3(C, A, B). In a majority voting system, aggregate preferences would lead to a situation in which A is preferred to B, B is preferred to C, and C is preferred to A.
The participatory focus of social multicriteria evaluation challenges the privileged role of the expert in the production of scientific knowledge and argues for the inclusion of a plurality of legitimate perspectives. Munda (2004a) favors “extended peer review” (Funtowicz and Ravetz 1993) as a means to include a plurality of social actors in decision-making. The participatory focus echoes the literature on the democratization of science (Wachelder 2003) and the social contract between science and society (Lubchenco 1998) within STS.
The approach problematizes expertise and unpacks the general notion of power through institutional analysis. The analysis of social actors and of the possible coalitions and conflicts that different policy options may create offer a more nuanced understanding of power as a relational issue. The use of iterative evaluations as a learning process signals awareness of the dynamic character of science–policy relations. In the case of Troina, one of the most relevant outcomes is the increased understanding of power relations: “as a feedback of the process of generation of alternative options, it was clear that additional interest groups outside Troina also had to be considered” (De Marchi et al. 2000, 272).
The NUSAP System of Uncertainty Assessment
The NUSAP system was developed by Funtowicz and Ravetz (1990), and later by van der Sluijs et al. (2005), and is aimed at the characterization of the uncertainty not explicitly communicated by numbers. The NUSAP system combines quantitative and qualitative assessments of the uncertainty present in scientific information. Numbers are classified in terms of Numeral (the quantity), Unit (the type of measure), Spread (the statistical error) on the quantitative side, and Assessment (the quality of the information), and Pedigree (the quality of the process producing the information) on the qualitative side. The NUSAP system has been widely applied to a variety of case studies, such as the uncertainties surrounding climate change predictions (Wardekker et al. 2008), groundwater modeling (Refsgaard et al. 2006), and the monitoring of emissions and environmental assessment in the Netherlands (van der Sluijs et al. 2005). The Netherlands Environmental Assessment Agency has adopted NUSAP as part of its Guidance on Uncertainty Assessment and Communication (Saltelli and Funtowicz 2014).
The NUSAP system highlights the uncertainties associated with the production of quantitative representations, both at the level of measurement and at the level of model uncertainties. The various categories make it possible to assess the quality of the estimates used and the possible trade-offs between, for example, reliability (assessment) and significance (spread; Funtowicz and Ravetz 1990). The pedigree category gives an additional measure of the confidence of the experts involved about the data and measurement schemes adopted (van der Sluijs et al. 2005).
NUSAP analysis yields a series of scores given by different experts to the numbers, indicators, or models under study. Experts assess the assumptions behind the numbers analyzed in order to reveal different dimensions of uncertainty such as technical uncertainty relating to the exactness of quantification, methodological uncertainty relating to the reliability of the models used, and epistemological uncertainty relating to the limits of disciplinary knowledge. The measure of uncertainty is given by the degree to which experts’ scores diverge rather than by the actual scores themselves (van der Sluijs et al. 2005). This makes the vagueness and ambiguity associated with numbers explicit and easy to communicate to policy makers.
Figure 3 offers an example of a NUSAP assessment. Pedigree relies on expert elicitation. The role of experts is thus analyzed reflexively, as those producing the scientific information are called to assess the quality of their own assessment.
Example of Numeral, Unit, Spread, Assessment, Pedigree. Source: Adapted from Funtowicz and Ravetz (1990).
The NUSAP system addresses the challenge of uncertainty associated with quantification. Uncertainty challenges the idea that numbers are unambiguous and precise means of conveying information. Funtowicz and Ravetz (1990) offer many examples of misuses of quantitative information. Mathematical accuracy is meaningless in the presence of high uncertainty, as illustrated by the fossil joke: a museum attendant tells visitors that some fossils in the collection are 56,000,012 years old. He knew this because, when he started the job twelve years before, he was told that the fossils were 56,000,000 years old. As Funtowicz and Ravetz (1990) explain, in expressing the age of the fossil, the zeros indicate the order of magnitude at which paleontological time is estimated, not an exact measure. The confusion about the role of zeros illustrates how the use of numbers does not eliminate vagueness or ambiguity. Additional knowledge is required about the meaning and intended use of mathematical notation in different contexts.
In the context of irreducible uncertainty, complete information cannot be obtained. For example, uncertainty plays a central role in the response to an earthquake, which entails depletion of natural resources, emergence of new diseases, and so on. Wynne (1992) identifies four different levels of uncertainty, starting from risk (uncertainty due to the probability of occurrence of known events such as casino bets), strict uncertainty (possible outcomes are unknown such as the financial crisis before 2007), ignorance (a situation in which one is faced with unknown unknowns such as the emergence of new diseases), culminating with indeterminacy (a situation in which the system as whole may change such as the Industrial Revolution following the invention of the steam engine). Different levels of uncertainty require different approaches to quantification.
Funtowicz and Ravetz (1993) hold that the get-the-facts-then-act model is only pertinent in a situation of low uncertainty, in the realm of Kuhnian normal science, where the so-called facts are certain. As uncertainty and stakes increase, facts become soft. Post-normal science is defined as a situation in which uncertainty is irreducible, the stakes are high, and decisions are urgent.
The uncertainty focus of post-normal science can be related to the line of research within STS developed by Callon and Law (2005) on the blurred boundary between quantity and quality. As Callon and Law point out, the definition of what can be quantified depends on the definition of what cannot be quantified, that is, on the definition of the act of quantification as distinct from nonquantification. In their discussion of the role of uncertainty in quantification, Funtowicz and Ravetz show how the definition of what can be quantified changes depending on context. In the context of high uncertainty, quantification cannot be distinguished from nonquantification, and the analysis turns to the quality of the scientific information.
This approach undermines the idea of “science speaking truth.” The concept of science is replaced by a more situated notion of expert opinions, and the limits of expert knowledge are highlighted. The reflexivity exercise required by the pedigree assessment unpacks the concept of truth and reveals a multiplicity of truths, or rather knowledge claims, that are performed and negotiated among experts.
Quantitative Story-Telling
Quantitative Story-Telling involves a participatory analysis of the quality of policy narratives. The participatory aspect places emphasis on the fact that different storytellers use different narratives. Quantitative Story-Telling uses numbers to represent narratives. Instead of looking for precise quantitative measurements formalized in a given scale and dimensions of analysis, the goal is to provide a richer characterization of the system under study as a remedy against “hypocognition” (Lakoff 2010). Hypocognition is defined as the limited understanding of a problem due to the tunnel vision effect caused by the use of a single problem framing, which implies a filtering on alternative explanations and on the definition of relevant aspects. Quantification is a means to acknowledge complexity. Complexity in this context is defined as the coexistence of multiple nonequivalent observations of the same system.
The approach focuses on the analysis of the feasibility, viability, and desirability of policies (Saltelli and Giampietro 2017). Feasibility is defined as compatibility with external constraints, viability is the compatibility with internal constraints, and desirability is the compatibility with normative values of a specified society. For example, Quantitative Story-Telling has been used to assess the feasibility and viability of the export-led growth strategy based on the production of sugarcane pursued by the Republic of Mauritius (Serrano-Tovar et al. 2014). Feasibility refers to the compatibility of the economy with the ecosystem in which it is embedded. In this case, the feasibility of sugarcane production can be tested against the availability of land, water, and the climatic conditions. Viability refers to the compatibility of the policy with the structure and organization of the socioeconomic system. In the case of Mauritius, the viability of increased sugarcane production can be tested against the availability of labor, the economic productivity of the agricultural sector, and the available technology. Feasibility and viability are assessed in relation to external and internal constraints, defined with regard to the focus of the analysis, in this case, the island of Mauritius. Desirability is discussed through participatory processes.
The focus on narratives highlights the role of the observer in the production of scientific information. The observer uses narratives to organize information about perceptions of external events. Storytelling implies agency: the choice of narrative and of what is observed depend on what the storyteller defines as relevant. The reference to external and internal observations recalls the need to consider normative choices embedded in quantitative information.
Feasibility and viability assessments are context-specific and use quantification to identify general trends, relations of congruence, and emerging properties of the system. Instead of using quantitative information to determine what should be done, quantification is used to define what cannot be done. Falsification is interpreted in this case as the identification of constraints, rather than the testing of scientific claims in the Popperian sense.
In Quantitative Story-Telling, the main challenge addressed is the simplification of complexity and its consequences for policy. In the context of science for policy, simplification is necessary to produce scientific information that can be used for policy. Producing a representation that is as complex as the issue that is described is not useful for action.
A literary illustration of the tension between the completeness of the information and the usefulness of the representation is given by Jorge Luis Borges. As the tale On Exactitude of Science (1658, in Borges 1975) narrates, the cartographers of an ancient empire made a map of the size of the empire in order to represent its entirety. However, the map was considered useless by the following generations and destroyed. A map of the size of the empire does not give any additional information, time saving, or guidance than traveling the whole empire. Simplification thus makes it possible to produce what Pereira and Funtowicz (2009) call “policy-relevant science.”
However, in the use of science for policy, some simplifications may do more harm than good. Reinert (2009) talks of “terrible simplifiers” to refer to simplifications that are not useful for guiding policy. Reinert uses the Millennium Development Goals as an example of terrible simplifiers, criticizing such goals for curing only the symptoms and not the causes of poverty. The Millennium Development Goals set goals in general terms for the whole world, overlooking the differences in the structure of the economy of individual countries, the qualitative differences between the activities of the countryside and those of the cities, and the differences between individual economic agents (Reinert 2007). Reinert argues that such differences are the cause of inequality and underdevelopment and that a policy based on averages is ineffective in tackling these causes. The problem of terrible simplifiers thus consists in the application of models based on simplifying assumptions to complex issues.
The focus on falsification of narratives draws on the distinction between complexity and relativism. In the complexity view, the existence of multiple representations is not to be confused with an “anything goes” approach (Mitchell 2009). As a consequence, it is possible to assess the compatibility of a policy with respect to different points of observation, such as the external–internal observation duality. This approach does not provide information to guide policymaking but rather assesses the information used and the consequences of different analytical and framing choices in opening or reducing the option space for policy.
The participatory aspect of Quantitative Story-Telling invites a reflexive and iterative approach to science–policy interactions, whereby the consequences of scientific inputs to policy are assessed with the aim of informing deliberative processes. Quantitative Story-Telling offers the tools to operationalize reflexive quality assessment of science for policy. The understanding of the relationship between science and policy that is embedded in this approach engages with the notion of power as the ability to choose a narrative, which defines the causal relations, representations, and points of observation in the production or validation of knowledge and recalls Foucault’s work on power and knowledge.
Heuristic Use of Statistics
Statistical tools are used in this approach as heuristic tools for quality assurance. The application of statistical methods is subject to the constraints posed by data rather than to the assumptions of the analyst. For example, statistics can be used as an instrument in setting up risk limiting audits for tallying election results (Stark and Wagner 2012). In this case, the use of statistics may guarantee that samples are audited randomly and minimize biases in the auditing. It should be noted that the use of statistics in this context is nondeterministic. As Lindeman and Stark (2012) argue, “risk-limiting audits do not guarantee that the electoral outcome is right, but they have a large chance of correcting the outcome if it is wrong” (p. 1). The heuristic use of statistics for quality assessment focuses on the reliability of mathematical representations and on the appropriateness of the statistical models used.
The heuristic use of statistics highlights the ambiguity associated with statistical tools such as probability assessment, quantification of uncertainty through statistics, random sampling, and so on. The approach consists mainly of adopting an assumption hunting attitude (Saltelli and Funtowicz 2014) and assessing the pertinence of the assumptions made and of the models used in relation to the issue at hand. Examples are the assessment of the probability models used to forecast the occurrence of earthquakes, the use of Bayesian statistics, economic forecasts, and so on. For example, in the case of earthquakes, it is argued that probability cannot be known, as earthquakes do not follow a known distribution in time and space. Stein and Stein (2013) compare estimating earthquake probability to “inferring the contents of the urn and the sampling process from the samples that have already been drawn” (p. 134).
This approach tackles the problem of the misuse of statistical concepts and methods: when faced with high uncertainty or ignorance, misplaced metaphors liken uncertainty to known situations such as casinos. As a consequence, haphazard systems are treated as if they were random systems. The same mathematical models used to assess risk in a casino scenario are applied to situations in which the probabilities and the outcomes are unknown (Taleb 2007; Albrecht et al. 2015). This approach addresses the ambiguity associated with the use of statistical methods.
The misplaced use of statistical concepts can be associated with what Kahneman (2011) has called cognitive ease: when faced with a question that is difficult to answer, people tend to substitute the difficult question for an easier one for which there is an answer. According to Stark (2012), Bayesian statistics assume that the probability distribution of occurrence of a given event can be known, as if claiming to know how nature generates its events. Assuming a known probability distribution makes it possible to estimate the probability of, for instance, earthquakes or terrorist attacks—which in frequentist terms are unknowable. The Bayesian can be interpreted as substituting a difficult (frequentist) question for an easier one.
This criticism recalls the demarcation model of science–policy (Funtowicz and Strand 2007), which distinguishes between pseudoscience and good science. This model assumes that the problem lies in malpractice in the application of the scientific method and in a skills deficit on the part of scientists (Funtowicz and Ravetz 1990). Saltelli et al. (2013) argue that the impression of rationality, prediction, and control is misleading when applied to policy. Pielke (2004) distinguishes between good science policy as the use of science to inform policy makers about a variety of different policy alternatives and bad science policy as the use of scientific information to endorse one alternative in order to influence decision-making. Saltelli and Funtowicz (2014) rephrase demarcation as the juxtaposition of evidence-based policy and policy-based evidence.
The heuristic use of statistics can be associated with the line of research developed by Asdal (2011, 2008) on the relationship between the use of numbers or, in this case, of statistical methods and probability distributions, and power. In his discussion of Bayesian statistics, Stark (2012) highlights the prominent role of trust and authority given to the statistician in determining probability distributions. The analysis of the role of the statistician resonates with the analysis of the role of bureaucrats (Asdal 2011). The focus on the demarcation model of science–policy reveals an affinity between this approach and the line of research on the constitutive role of numbers, and statistics, in policy.
Similar to Quantitative Story-Telling, the heuristic use of statistics is an ex post assessment of the quality of quantitative information and statistics used in policy and does not provide statistics or guidance for policy. The literature on this approach does not explicitly engage debates about the role of science in policy and about power. The reference to the demarcation model can be interpreted as a criticism of the linear model because scientific tools such as statistics are easily misused.
The Constitutive Role of Numbers in Science–policy Models
In this section, I unpack the conceptualizations of numbers that emerge from the nonpositivist use of numbers and show how the character of numbers impacts the relation between science and policy. The use of numbers has to be understood rather than assumed away. The concept of numbers as a description of the world-as-it-is stands at the basis of the view of science speaking truth to power (Wildavsky 1989). Alternative conceptualizations of numbers can be used to challenge this science–policy model.
A summary of the main characteristics of each approach is given in Table 1. The classification used in this paper to distinguish between the different approaches overemphasizes the differences. In practice, there are overlaps between these approaches and between the criticisms they pose to the use of quantitative information for policy.
Overview of the Five Approaches.
The types of challenges tackled by the alternative approaches to quantification considered in this paper are framing, pluralism, uncertainty, complexity, and malpractice. Some of these challenges have been amply discussed in academia and in the media. The issue of framing is central to advocacy groups, both within and outside academia. The replicability crisis in psychology and statistics (Baker 2015; Baker 2016; Open Science Collaboration 2015) has brought the issue of malpractice to the forefront. I argue that the approaches analyzed question the idea that there is something special about numbers as opposed to other kinds of scientific analysis or argumentation in the interface with policy. For example, Quantitative Story-Telling suggests that numbers are stories no different from qualitative argumentations.
The conceptualization of numbers that emerges from the approaches analyzed is varied and nuanced. The valuation of ecosystem services consists of value-based problem framing. Numbers are conceptualized as value-driven facts. The conceptualization of numbers as bearers of information does not change. What is different is the framing that determines which type of information numbers should bear.
In social multicriteria evaluation, numbers are used to rank the criteria according to which policy alternatives are measured and to rank preferences of different actors. The approach can be described as ranking of incommensurable values. In this case, numbers are used as a cardinal ranking tool in the definition of social preferences. Numbers are means to deal with pluralism, through the multicriteria matrix, weighting factors, and the social equity matrix.
According to the NUSAP approach, numbers are things to be studied, they are the object of investigation, and their use changes according to the level of uncertainty. The approach offers a quantification of uncertainty in the so-called facts. At low levels of uncertainty, numbers can be used as facts, while in the context of irreducible uncertainty numbers themselves must be assessed as sources of uncertainty or even ignorance (Ravetz 1987).
In Quantitative Story-Telling, numbers are used to test the usefulness of simplifications in guiding policy rather than as sources of information. Quantitative Story-Telling is based on the assumption that numbers are not descriptions of a reality that is out-there but play an important role in the organization of one’s perception of reality. Numbers describe the perception of a particular storyteller. Counting cells in an organism reflects the analytical choice of the observer but does not exhaust all there is to know about that organism. Quantification in this approach is used to acknowledge complexity.
Lastly, numbers have a secondary role in the heuristic use of statistics. The focus is on the pertinent use of statistical methods, as a way of ensuring that data are analyzed according to the information they provide (or fail to provide) rather than according to the assumptions imposed on data. The approach used is prominently quantitative, but numbers do not carry any information per se. The heuristic use of statistics is based on assumption hunting. Numbers become useful only once the constraints are known and the pertinence of the statistical methods can be established.
Collectively, these conceptualizations move away from the positivist use of numbers. Numbers are used as heuristic tools, as means to tell a story. Heuristics are understood as opposed to blueprint procedures for the use of quantitative information. They can be seen as a strategy to deal with uncertainty and complexity. Heuristics should not be confused with improvization but rather with the ability to adapt to the irreducible singularities of the object of study (Serrano and Romero 2014).
These conceptualizations of numbers matter for the interface between science and policy. Different conceptualizations of numbers reflect the role of science advice in opening or closing the policy option space and make it possible to rethink the type of engagement with power.
Three variations can be observed in the role of science envisaged by the approaches analyzed: advocacy, reflexivity, and quality assurance. In the evaluation of ecosystem services, science provides a value-based problem definition upon which policy is expected to act. The separation between experts and nonexperts is maintained and used to make authoritative claims over the values that should inform policy. Science thus plays an advocacy role. Social multicriteria evaluation and the NUSAP notation system pursue reflexivity in the use of numbers, preference rankings, and expert advice in policy processes. Both approaches refrain from giving substantive advice and offer analytical tools that lead social actors and experts to reflect upon their own positionality in the definition and assessment of policy options. Science and policy are understood as intertwined processes rather than as separate activities. Quantitative Story-Telling and the heuristic use of statistics retain a role of quality assurance. The former approach is used to assess the quality of policy narratives, and the latter is used to ensure the appropriate use of statistical methods.
The production of quantitative information is not just outward looking (with regard to the type of policy that science wants to inform) but also inward looking. Reflexivity is practiced by analyzing uncertainty, pluralism, and complexity and using numbers as heuristic tools. Reflexivity leads to humble science, which offers decision support tools rather than authoritative claims. By contrast, value-based problem framing takes the insights of science studies and uses them to advocate for specific policies. The advocacy function of the evaluation of ecosystem services creates a paradox, expressed by Jasanoff (1996) as “how…can a sceptical and reflexive stance in relation to scientific knowledge be reconciled with making authoritative recommendations for social policy?” (p. 193). With regard to the treatment of policy options, there is a clear difference in the evaluation of ecosystem services and the other approaches, which open the option space.
Reflexivity invites also reflection on science’s blind spots. Both Quantitative Story-Telling and the heuristic use of statistics adopt via negativa use of numbers. Quantitative Story-Telling uses quantification as a means to identify and test the constraints associated with different narratives. The heuristic use of statistics is used to determine when statistical methods cannot be used. This analysis suggests that instead of describing the world through numbers, numbers can be useful in defining what cannot be known: for example, uncertainties, constraints, probabilities. The relevance of negative information is also discussed by Taleb (2012). Taleb warns that absence of evidence is not the same as evidence of absence. In the case of new pharmaceutical drugs, side effects may not be evident in the beginning, which should not be mistaken for absence of side effects.
These approaches also have different implications for the type of political engagement they enable. Pluralism is not resolved to inform policy makers but is explored in iterative participatory processes to improve social learning. Communicating uncertainty aims at creating context awareness in policymaking. Complexity is mobilized to engage with multiple perceptions and avoid giving privilege to one level of analysis.
These approaches pursue the logic of care (Mol 2008; Puig de la Bellacasa 2010). Mol (2008) distinguishes between logics of care and logics of choice in health care. The same distinction can be made in science–policy engagements. The logic of choice is based on the understanding of the science–policy interface as one-off engagements in which scientific information is passed on to decision-makers (Völker et al. 2017). Choice may imply a confrontation of some sort between science and policy/politics. Logics of care acknowledge the need for continuous engagement between a multiplicity of actors, with purposes that may go beyond decision-making and include social learning. Care suggests that science and policy are interlinked as theorized by coproduction theory (Jasanoff 2004). The evaluation of ecosystem services turns on the logic of choice and provides value-driven facts that close the option space. By shifting away from the advocacy role, the other approaches allow for the logic of care in science–policy engagements.
The valuation of ecosystem services is an example of what Funtowicz and Strand (2007) have defined as the framing model. The framing model can be expressed as science-speaking valuable facts to power, where valuable can be defined in terms of ethics, relevance, urgency, and so on.
Social multicriteria evaluation, NUSAP, and Quantitative Story-Telling make use of iterative and participatory processes, suggesting that science is done with policy, rather than for policy, reflecting the logic of care. The relationship between science and policy in social multicriteria evaluation can be characterized as science speaking pluralism with power, in the NUSAP approach as science speaking uncertainty with power, and in Quantitative Story-Telling as science speaking complexity with power.
Similar to the demarcation model (Funtowicz and Strand 2007), which establishes which science should be used for policy, the heuristic use of statistics establishes in which cases statistics should not be used for policy, bringing attention to context. While in the case of low uncertainty quantification and statistical analysis are useful tools, in the context of irreducible uncertainty and complexity, the limits of quantitative information become the object of study. This approach aims to create awareness about constraints and does not give any substantive advice to policy. The heuristic use of statistics can be thought of as science speaking context to power.
The significance of the type of engagement pursued resides in the dynamics legitimated by these approaches. The recent debate about the role of STS, if any, in the use of alternative facts in politics (Fuller 2016; Collins, Evans, and Weinel 2017; Jasanoff and Simmet 2017; Sismondo 2017) draws attention to the unintended consequences of showing the continuity between science and politics. Valuation of ecosystem services and similar approaches may be seen to provide grounds for the use of value-driven facts. In the REDD+ example, scientific facts about climate change are mobilized to give scientific legitimacy to ethical and political claims for action against climate change. I do not use this example to criticize the cause for which evidence about deforestation is used but to argue that the use of value-driven facts makes “facts” liable to criticism if the values associated with them are questioned. Fuller (2016) suggests that the use of alternative facts becomes a problem when the “wrong” value-driven facts are mobilized. That is, value-driven quantification may legitimize alternative facts.
In the Troina study, social multicriteria evaluation was used to understand why the status quo was maintained, even though the business as usual policy option was not favored by most actors and deviations from business as usual were not perceived to cause social conflicts (De Marchi et al. 2000). Results showed that the situation was caused by a political impasse, not a conscious strategy to avoid conflict. Also in this case, one could ask whether the valuation exercise stabilized power relations or tilted them, and which actors were favored by this exercise. The discussion of unintended effects brings attention to the need to reflect on how power is conceptualized and on the sort of power that the different approaches target.
The distinction between the approaches analyzed is less sharp when it comes to the sort of power that is addressed. Social multicriteria evaluation and the NUSAP approach engage with a democratic type of power, in which decisions, expertise, and policy options are negotiated; may face opposition; and be influenced by coalitions. Evaluation of ecosystem services seems to engage with a more political understanding of power and seeks to influence decision-making. Quantitative Story-Telling and the heuristic use of statistics are less explicit about the type of engagement they pursue. The role of quality assurance could feed into both a democratic and political sort of power, or could be interpreted as introducing a different understanding of power as bureaucratic, as attention turns to procedural quality.
Conclusion
This paper introduces a definition of numbers that offers more nuance than the positivist view. I analyze the variety of practices that use quantification to inform policy and the conceptualizations of numbers that emerge from these practices. In the examples analyzed, numbers can be defined as value-bearing information, as means of dealing with social preferences and pluralism, as instruments in the governance of uncertainty, as representations of subjective perceptions, and as means to quantify what is not.
These varied uses and conceptualizations of numbers have important consequences for the science–policy interface. The model of science-speaking truth to power is based on the definition of numbers as bearers of objective information. If the definition of numbers changes, the role of science, the role of policy, and their interactions also change. Just as the positivist conceptualization of numbers creates new objects and modes of governance through statistics and indicators, and alternative conceptualizations of numbers have to be studied in relation to the processes they may challenge or stabilize.
Uncertainty highlights the limited capacity to produce facts and thus abolishes the privileged position of science as provider of facts. Storytelling highlights the subjectivity present in quantitative representations and characterizes science as one among many possible representations. Multicriteria evaluation flags the inconsistencies that may emerge from preference ordering at the social level, which science is unable to solve. The definition of what numbers are not, or what they do not say, makes it possible to identify the limits of abstraction and to adapt the use of statistics to the context.
Some of the approaches analyzed redefine the science–policy interface based on the logic of care, while others refrain from giving substantive advice to policy and assume a role of quality assurance. I have argued that valuation of ecosystem services stabilizes power relationships between science and policy, in which science provides a value-based problem definition upon which policy is expected to act. Alternative quantification is used to make authoritative claims over the values that should inform policy.
The other approaches offer a variety of means toward a more reflexive use of numbers, in which quantification engages with pluralism, complexity, and uncertainty. Reflexivity consists of the observation of science’s own assumptions, subjectivities, and ignorance. The effects of such alternative approaches on power relations and science–policy processes remain to be seen. What distinguishes the heuristic use of numbers is that uncertainties and contradictions are not swept under the scientific rug but are part of the interface with policy and with society. These approaches attempt to build trust from a humble, rather than an authoritative, position.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
