Abstract
Under ever-increasing pressure to provide more with less, to justify budgets and to earn public trust, official statistics has long been concerned with how to better prove and communicate its value. Being statisticians, our inclination has been to express the value of our offer in quantitative terms. But an ONS-led Task Force under the Conference of European Statisticians (CES) argued that before we can quantify ‘the value of official statistics’ we need to understand what this really means. This entails first articulating our own central goals as providers of a public good and then working outwards from these goals to formulate the means of fulfilling them. Only then can we start to define measurable indicators of achievement to assess how far we are creating this intended value. This is the reverse of the process often followed, which starts out by identifying already-available indicators and tries to determine the aspects of value of which they are indicative. Future international work should focus on developing tools for better understanding the pathways from goals to value indicators; sharing experiences of efforts to prove and improve the value of official statistics; and developing a core set of measures using the methods outlined in the article.
Keywords
Can we measure the value of official statistics?
Building on proposals made in a 2018 Conference of European Statisticians (CES) publication, Recommendations for Promoting, Measuring and Communicating the Value of Official Statistics [1], a CES Task Force chaired by the United Kingdom’s Office for National Statistics (ONS) was established to pilot test a measurement framework comprising a set of possible indicators for quantifying the value of official statistics. This work was viewed by CES members as increasingly pressing, in the face of declining budgets, broadening mandates, and growing challenges in staking their claim to a privileged position as providers of something uniquely valuable for society. National Statistical Offices (NSOs) felt that while they were themselves convinced that their products and services were of high value, they needed measurable evidence to make this case to governments, funders, users and taxpayers, who are often relatively unaware of the distinction between official statistics and other potential sources of statistics and data.
The initial measurement framework, proposed within the 2018 report, was divided into three elements:
Observable or ‘objective’ indicators. These are indicators which reflect actual use of official statistics, and/or which reflect adherence to one or more of the Fundamental Principles of Official Statistics, including well-established quality indicators. ‘Subjective’ indicators. These are indicators derived from general or specific user surveys, covering perception, trust, client support, satisfaction etc. Monetary valuations of the impact of statistics. These are indicators which attempt to place a monetary value on the ways in which statistics affect the outcomes for which they are employed, and/or which weigh the value of outputs against the cost of inputs.
The CES Task Force reviewed a wide range of potential measures under each of these headings; a brainstormed set of possible indicators proposed in the 2018 work, plus others suggested by the new task force. They examined whether and how these measures are currently employed by NSOs; how they are or could be produced; and what they could tell us about the value of official statistics.
The last of these aspects proved to be a turning point in the work. ‘What they could tell us about the value of official statistics’ turned out, in many cases, to be not much at all.
Many of the proposed measures were already available, or easily produced, including standard quality metrics, engagement and usage statistics (especially those derived from online platforms such as website and social media analytics), and measures of adherence to the Fundamental Principles of Official Statistics [2]. Hence it is relatively easy to gather information on the magnitude of these measures, explore trends over time, and see how they vary with other factors, such as how online engagement changes when new statistics are released. But on close inspection, it became apparent that this kind of information – while undoubtedly extremely useful for many purposes – tells us little if anything about the value that the statistical products and services are adding to society, at least when viewed in isolation.
For example, a measure of visits to a web page containing a new statistical release does not, by itself, tell us whether the user is finding what they were looking for; whether they have retrieved the information or statistics and made further use of them, or for what purposes. If the magnitude of that measure goes up when we make a change to the web page, does this tell us that the value of the information on that page has increased? What if in fact it shows that more people are accidentally ending up there when they were looking for something else? Or that the same people keep on visiting the page multiple times because they haven’t yet found what they were hoping for? What about if they do indeed find what they need, but then do nothing with that information? Can we still say that it is valuable?
Similarly, quality metrics, on their own, cannot be directly interpreted as indicative of value – at least not for all users and uses. For some, rapid error correction might be very important but for others the time lag between collection and release is of much greater interest, and is what makes the statistics valuable to them. A tendency to conflate quality metrics with value metrics speaks volumes about our own biases as official statisticians.
‘Measuring value’, then, turned out to be much more complex than it initially appeared. This was especially true when it came to the monetary valuations, which attempt to quantify value in terms of prices, revenues and ‘willingness to pay’ – an easy-to-understand view of value that corresponds with how we usually determine the value of market goods. Official statistics is by no means alone in trying to develop methods for assigning a monetary value to a public good, but, as with these other attempts, it faces many challenges. Unlike a tangible, consumable market good such as a pair of jeans, we can’t simply ask people what they would be willing to pay for, say, a quarterly GDP figure. Only one person gets to wear the jeans at any given moment, while the GDP figure, once out there, can benefit an entire society, including people who do not knowingly or directly ‘use’ the figure but who are affected by decisions made based on it. Similarly, while monetary valuation of a market good can compare it with other, similar goods (‘which brand of jam would you prefer to buy?’), it is far less apparent what should be the comparator in the case of official statistics – if we want to determine the value of a population count from the census, should we compare it with the absence of any population figure at all? Or with an out-of-date count from an earlier census? Or with a population count produced by a non-statistically-representative survey conducted by an entity outside official statistics? Even if and when we’ve managed to make these kinds of calls, we still have to grapple with the much deeper philosophical questions about how to assign monetary worth to each of the benefits identified. If monetary valuations are reduced to purely economic matters (say, the amount of money saved in planning the recruitment and training of teachers if it is based on demographic data rather than guesswork), then they disregard the many benefits that have no clear mapping to an economic outcome yet are nevertheless very real. This is not to downplay the potential of monetary valuations, but rather to emphasize that no matter how meticulously carried out, a monetary valuation would not be able to fully capture and quantify all the ways in which official statistics can generate value for society. Put simply, not everything has a price tag.
These manifold challenges in even conceptualizing what ‘value’ means, let alone actually measuring it, led the CES group on a renewed quest: to aid National Statistical Offices not merely in measuring the value of official statistics, but in understanding and assessing that value.
NSOs have a variety of reasons for wanting to measure the value of what they do. These fall into two groups: they want to prove that their work is worth it, and to improve what they are doing – for which they need benchmarks against which to measure such improvement.
Proving value means providing evidence to government, other funders, and society that the money invested in official statistics is a good use of public resources and offers a good return on investment. This is ever more critical in an environment of tightening budgets, where every claim on funds is heavily scrutinized. Where budgets are being cut and NSOs are asked to prioritize among activities and outputs, proof and quantification of value could enable them to select which activities are candidates for reduction and which are so valuable that they absolutely must remain. Beyond measures of value alone, NSOs hope to develop cost-benefit analyses which would enable them to see which activities and outputs generate the most value relative to the resources invested therein.
Proof of value also holds promise as a communications tool, to create and maintain public trust – which in turn is essential to the operations of official statistics in the first place, since without a public who trust that they will gather, store, analyze and interpret their data correctly, they will not willingly provide the data which is the lifeblood of official statistics. A simple sound bite such as “for every $1 invested in the Census, $6 of value was generated to the Australian economy” [3] can be powerful in getting the public on board.
Improving value means better meeting needs, doing more with less, and monitoring the effectiveness of efforts so that we can see what works and what doesn’t.
Whose perception matters?
When devising and producing measures for assessing the value of official statistics, it is important to ask, whose value is being measured? A core argument put forward by the task force is that value is determined by the customer. Hence any attempt to quantify how valuable official statistics are must begin with an investigation into what it is that people, users, stakeholders, and others, actually value. People will value what fits their needs. Therefore, value is inherently subjective and cannot be determined without reference to those needs, and to the perception of the one doing the valuing. As the CES task force wrote in its 2022 report [4], “value, like beauty, is in the eye of the beholder”.
How is value different from quality?
Value and quality are very closely linked and overlapping. In everyday language, indeed, they are often taken to mean more-or-less the same thing. In official statistics, quality is already a well-defined concept with clear and agreed dimensions. It amounts essentially to ‘how good our statistics are’. Quality, in essence, is the ‘degree of excellence’. Value, on the other hand, is the subjective assessment of that quality that makes something desirable. Some people might place a high value on one quality dimension but not care at all about another, while a different user might have entirely the opposite view. Hence, in spite of the many ‘objective’ measures of quality potentially available to us, we cannot simply assess a statistical output along the various dimensions of quality and declare that, if it scores well, it is necessarily of high value. A meticulously crafted, top-end racing car is of low value to a parent of small children who needs space in the back for pushchairs and shopping; and official statistics that check all the boxes for clarity of concepts, representative samples and small confidence intervals is not necessarily of high value to someone whose principal interest is timeliness.
Conceptual map of key areas of thought in defining, understanding and measuring value.
The tendency to conflate value and quality is not the only cause of misunderstanding. The value of what we do and the values to which we adhere are closely – and sometimes confusingly – linked. Values are the things that drive and motivate us, the reasons we do what we do. Value is what our work is worth, the degree to which it’s doing what people want or need and its usefulness in the eyes of our users and for society. We mustn’t assume that they are the same: just because something is important to us, doesn’t mean it should be important to others. Of course they’re not completely unconnected. Our values – such as a firm belief in the essential importance of the Fundamental Principles – push us to create, we hope, valuable products. But we should not fall into the trap of assuming that this cause and effect is inevitable and automatic. The value we place on political independence of official statistics, for instance, is absolutely crucial and non-negotiable. Yet we cannot assume that this matters as much to others as it does to us.
In sum: we measure quality to ensure that we are producing the best statistics we can; we measure value to ensure we are doing what people want; and our values underlie and motivate what we do, and, we hope, result in features of our work that society will value.
Conceptual framework: Shifting the perspective from producer to customer
Having established that the value of official statistics is a much more nuanced concept than it might at first seem, and that understanding it requires a closer look at the perspective of the one doing the valuing – how can we move towards adopting that perspective? It’s not enough to just recognize that we’ve been looking at things the wrong way; we need some guidance to help us look at them the right way.
A first step is to begin by mapping out the various concepts entailed in the idea of value, and seeing how they relate to one-another. When the CES task force did this, it emerged that there is a bifurcation between those aspects that relate to the production of statistics, and those that relate to the consumption of statistics or statistical products and service. In short, ‘producer-based’ and ‘consumer-based’ aspects of value (Fig. 1).
These two high-level categories of value concepts lead to correspondingly different sets of measures that might be used to quantify value. Both are useful and important. On the production side, there is a whole host of measures such as the availability of metadata, the rate of error correction, or the punctuality of releases. On the consumer side, there are measures of trust, satisfaction, and willingness to pay.
When NSOs begin trying to develop ways to measure their own value, they often start out with a strong focus on the production-based measures, many of which are relatively easy to produce internally without embarking on new dedicated data collection. NSOs may even feel that they are incorporating a user perspective, augmenting the measures relating to characteristics of the products and services themselves (e.g. availability of metadata, measures of timeliness, punctuality and accuracy) with measures of usage, access, and interaction with their products such as data downloads, specialized requests and citations. But even these are only proxies for value, based on an assumption that if a user accesses our products then that means they find them valuable. To really get to the heart of what our customers value and why, we need shift the balance towards the consumer-based understanding of value, and the subjective, externally-driven measures to which this perspective gives rise.
DARE to be different: a new proposal for understanding what customers value
Armed with this new realization – at once breathtakingly obvious and curiously radical – that if we want to know what people value, we need to ask them; what do we do next? How can NSOs know if they are producing what people need?
‘DARE’: The four attributes of value.
The CES Task Force proposed a model, called ‘DARE’ (Fig. 2), which builds on a framework developed by Statistics New Zealand in 2018–19 [5]. The model puts forth a customer-centric approach, based on four (not mutually exclusive) interlinked attributes: “Dependable, Applicable, Relationship and Ease of use”. Collectively, these four attributes provide a set of lenses through which to contextualize a customer’s perspective of value for a given product or service.
There are many possible value criteria within each dimension. Within ‘ease of use’, a school student might value easy website navigation, while a civil society activist might value clearly explained visualizations. Within ‘relationship’, some may find rapid customer service responses to be valuable while others are impressed by outreach sessions on planned census dissemination products. Dimensions may overlap, for instance where the extent to which a statistical product meets a user’s needs (applicability) is seen by that user as resulting from effective NSO-stakeholder consultation (relationship).
Maximum value is delivered where all four attributes are present, but the importance or weight of each of the four dimensions will vary between different users and uses. For example, a corporate customer might attach more value to dependability than to relationships, whereas an ordinary citizen might favour ease of use. The more the customer’s specific needs are met in a product, the higher the value proposition for the customer. Customer-perceived value will differ across customers, specific uses and over time.
The model suggests, therefore, that thinking about value must be flexible, outward-looking and inclusive of these dynamic subjective perspectives.
The DARE model is not a measurement framework offering indicators or scales. Rather, it provides a framework to guide thinking, to help NSOs identify areas where the value proposition can be improved and target their efforts towards maximizing the value of their work in the eyes of their customers.
So how can NSOs develop measures to assess their own value and ensure that only meaningful metrics are being used to drive decisions?
Structure of a Results Map.
The CES task force proposed that NSOs develop their own value-measurement frameworks in a results-oriented manner. One way to do this is to use a ‘Results Map’ tool inspired by Barr [6], which starts by placing an agreed organizational goal or mission statement at the centre. From there, pathways are traced outwards, through a hierarchy of layers from the more general to the more specific, ending in measurable indicators in the outer layer. In Fig. 3, the four attributes of the DARE model are overlaid on the Results Map to help see how the various pathways are distributed across the four dimensions of value (another potentially helpful means of identifying gaps where success is not being sufficiently monitored).
This approach turns value “inside out”. This contrasts with an “outside in” approach, so commonly practiced by NSOs (and other industries), in which measures are produced based on whatever information is readily available and easy to analyze, rather than being specifically crafted to measure what needs measuring.
All too often, organizations come up with key performance indicators without linking them directly to an overall strategic goal. This can result in indicators that don’t actually measure things that would lead to progress towards a goal.
By employing this “inside out” approach starting with a central goal based around creating and improving value, the NSO can meaningfully formulate the actions needed to achieve that goal, and then move to ways in which the effectiveness of those actions can be tracked and monitored. Instead of starting with ‘what can we measure and what does it tell us?’, we ask ‘what do we need to know, and how can we measure it?’. The goal comes first and the measures come last.
Much remains to be done to get to the stage where the official statistics community might have a common set of indicators of value. Whatever these indicators might eventually be, they will need to adhere to some general principles if they are to be understandable and useful.
Measures should be clearly indicative of some aspect of the value of official statistics. As discussed earlier, this doesn’t simply mean measuring quality or adherence to Fundamental Principles of Official Statistics; there needs to be evidence that whatever is being measured is something that is valued by someone whose needs we are trying to meet. Measures should be (at least theoretically) quantitative and have a monotonic relationship with the aspect of value being measured, i.e., a greater measurement indicates more value and a lower measurement indicates less value. Measures should be able to lead to the formulation of actionable targets. It should be evident what a ‘good’ level of the measure would look like. And there should be some route by which this evidence could inform actions or behaviours by the NSO to improve the aspect of value being measured. So, measuring something which is entirely beyond the control of the NSO would not be helpful, as it would not be within the power of the NSO to act on it and change it. The potential unintended consequences of producing a measure should be considered. The act of measuring anything is never neutral. By measuring something we imply that we find it important, and whether deliberately or not we may turn it into a goal or target, so that the measure becomes and end in itself. In official statistics, this could manifest as taking longer to release figures if error correction time is adopted as a key indicator of accuracy (thereby reducing timeliness); or increasing the output of popular social media-friendly material if the frequency of reposts is adopted as a key indicator of engagement (potentially at the expense of more detailed content).
If we adopt a consumer-based understanding of value, look through the DARE lens, and use a Results Map approach to formulate measures that follow these guiding principles, we may find ourselves with a range of possible measures that is broader and potentially more helpful for informing targeted endeavours to improve the value of official statistics. Some of these may not be easily-available ‘off the shelf’ measures. We may not have the data to hand in our standard compilations of metadata, web analytics and user survey responses. But difficult is not impossible. Just like so many other topics in statistics that start off seeming next to impossible to measure, we can find ways to make them possible and bring them into the mainstream.
Several areas of work remain outstanding and should be taken up by the international community.
Propagate and foster practical application of the core message of the work. It is hoped that this special theme in the SJIAOS will play a part in triggering broader and deeper thinking about this central message: that ‘value’ is a dynamic concept which must be customer-defined. A wide-ranging discussion and debate around this topic is needed to encourage NSOs to embrace the view that this really necessitates a full paradigm shift in thinking and practice at all levels. When the secretariat to CES conducted a consultation on the draft task force report in 2022, a common refrain among respondents was ‘yes, we agree, we already take user needs into account’. Yet the authors of the 2022 report, and we, the authors of this present paper, argue that there is a big difference between consulting users about individual products, and bringing society as a whole (active users, passive users, non-users and other beneficiaries) into the endeavour of defining the role and purpose of official statistics and assessing whether it lives up to the standards thus defined. A welcome growth in national and international work focusing on methods for assessing the impact of communication tools and methods of communicating statistics is evidence that NSOs are increasingly taking the needs of a range of different users and uses into account. Yet it is crucial to realize that this is not only about improving communication of statistics. Meeting user needs does not just mean presenting statistics in better ways or to a wider variety of user groups (although this is an essential part of it). Rather it means taking society’s needs into accounts at all stages in the statistical business process. It would be unfair and unrealistic to lay this all at the feet of our dissemination and communications departments. Equally, regular conduct of user satisfaction surveys is a highly commendable endeavour – but on its own, it is not sufficient to get a true handle on the value being created by our work. Rather, it is a brick in a much larger wall. Continue international collaboration to share and improve. The various models and proposals described in the CES report, and introduced briefly in this paper, need to be tested in real-world settings and refined. We issue an open invitation to any NSO willing to trial the use of the Results Map approach described in the Task Force report [1] to share their findings with the international community and help develop it as a tool for future use. Continue to gather, share and analyze examples from countries. In preparing its report, the CES task force gathered and analyzed a range of case studies contributed by the NSOs of many countries. These case studies detailed ways in which NSOs have attempted to assess and understand the value they create. Some of these examples pertained to assessing the value of specific products or projects, while others dealt with the perception of the NSO more generally. Many of the case studies were found, on closer inspection, to be more focused on measuring one of more aspects of quality and/or of adherence to the Fundamental Principles than on value. In the years since these examples were gathered, no doubt there have been new attempts to gain a better picture of how official statistics serves society. These should be shared internationally, extracting lessons learned and distilling best practices for understanding and enhancing value. Develop a new core set of measures for understanding and assessing value. It is clear that there is no one set of measures that all countries could or should employ in all their diverse contexts. Yet, there are enough similarities across countries that it may be possible to develop a set of measures that serve as a core upon which other country-specific ones may be added. The key is that this core set needs to be developed through the ‘goals to strategies to outcomes to indicators’ methodology outlined above, not through the ‘brainstorming a list of possible measures’ approach previously employed.
We hope that readers of this article may be inspired to re-invigorate the discussion of value in their own offices, prompting some NSOs to become frontrunners in putting the conversation about value back on track, and pushing for the change in perspective that is so
badly needed if value is to be anything more than an overused buzzword.