Abstract
Public libraries in the Netherlands face growing scepticism about their value to communities and society at large. The digitization of media, information and communication gives rise to questions about the library’s function, as it still is based mainly on physical service provision. Furthermore, the current economic recession causes local, regional and national governments to critically question every euro spent on cultural institutions. In this climate, there is a growing need for public libraries to show their worth – not only in an economical, but also in a more sociological sense. As standardized measurements for these values are still lacking, a research program was started to develop these outcome measurements. In this paper, we sketch the rationale behind this program and the steps the Netherlands Institute of Public Libraries is taking to develop a measurement instrument geared at validly and reliably demonstrating the societal value of public libraries. Results from the first stages of the research program will be presented: a theoretical framework of the (possible) impact of libraries on Dutch society, based on the findings of a literature study and qualitative research. The findings help us identify and conceptually enrich five domains of possible impact: cognitive, social, cultural, affective and economical. This outcome framework will guide the development of a measuring instrument.
Introduction
The value of public libraries in the Netherlands to communities and society at large is less taken for granted than it was in the past. The societal context in which libraries operate is rapidly changing. They are confronted with various challenges in the field of digitization, changing usage patterns and changing expectations of patrons. These challenges give rise to questions about the library’s function, as it is still based mainly on physical service provision. Furthermore, the current economic recession causes local, regional and national governments to critically evaluate every euro spent on cultural institutions. Libraries are therefore more and more urged to document their value and demonstrate their relevance to citizens, commentators and politicians. Dependent on the prevailing public management theory at the local level – either new public management or public value strategic management 1 – this takes the form of (only) economical or (additional) sociological frames of reference.
Demonstrating relevance can take the form of output and outcome measurement. Hitherto, much attention has been paid to the outputs of the library (numbers of materials, loans, visits, etc.) in addressing the importance of public libraries for Dutch society. What is considerably less well known and documented are the outcomes of the library, or, in other words, its real social worth to society (Institute of Museum and Library Services, 2000 2 ; Johnson, 2010 3 ; Marless and Streatfield, 2001 4 ). Attempts to capture the meaning of libraries to the users are being made with user surveys, but these measures are mostly aligned to assessing or evaluating library performance (in the context of user needs, preferences and satisfaction). These output and performance measures, however, do not shed enough light on the value of the library to the user and the impact on his or her life (Durrance and Fisher, 2003 5 ; Debono, 2003 6 , Vakkari and Serola 2012 7 ; Poll 2003 8 ; Poll and Payne, 2006 9 ).
A further step, measuring outcomes, is made in several studies abroad. Most of the research is, however, aimed at university libraries and research libraries. 10 However, in some countries efforts to conceptualize and measure the impact of the public library have also been made. These measures are mostly aimed at specific domains or types of outcome. The various approaches and results will be discussed below. These studies do not always depart from a theoretical framework or at least an inventory of possible societal domains where impact of public libraries is expected. Where they do, approaches diverge and it is not clear which approach is to be preferred.
To help improve this state of affairs, the aim of this research project is therefore a methodological one, combining the development of both a theoretically inspired framework for studying the societal value of public libraries, and a combination of measurement instruments which constitutes a valid and reliable operationalization of that framework. In so doing, in the long term, we:
contribute to the level of knowledge (from the branch, stakeholders and politics) about the outcomes of the public library in society hereby stimulate critical reflection of what we do and how we do it which contributes to maximization of the impact of libraries on society and helps us show the impact of public libraries to the outside world (e.g. stakeholders and politicians).
The main research question reads as follows:
How can we develop a valid and reliable research instrument (or combination of instruments) that makes it possible to measure various aspects of the societal value (returns or benefits) of the public library on the national level?
Outputs versus outcomes in public libraries
For long, public libraries have collected statistics on their performance. In the Netherlands, more or less reliable statistics are available starting from the sector’s birth years at the beginning of the 20th century. 11 These statistics consist largely of units that could easily be registered or counted, e.g. the number of organizations, branches, registered users, loans, loan extensions and opening hours. Numbers like these are valuable for mapping the sector’s expansion and having crude measures of its societal impact. However, if one’s aim is to demonstrate the public library’s value to critical observers like subsidy providers, these measures quickly fall short. A book having been borrowed does not equal a book having been read or consulted. One would want to know what reading a book has contributed to a person’s worldview or knowledge. Here, we touch upon the distinction between outputs and outcomes.
Outputs can be described as
a product directly resulting from a program (be it an activity or a service); typically measured in numbers to demonstrate the productivity of a program; hence output = count / event (entity that can be registered) usually a measure of volume (expressed in numbers, counts): i.e. number of products / services that are provided, people who are helped, activities that are organized the results of inputs (resources) and activities (programs or services) to be objectively quantified by neutral observers.
Outcomes take the importance of the library for individuals and society one step further in relating to the differences that are being made through the goods and services offered:
they reflect the changes or improvements brought about in people’s lives, showing that your program has (or has not) been successful (effective) they are measures of impact or benefit they are usually reported in amount of change in skills, knowledge, attitude, behaviour, or condition (life situation / social status) they are the success stories of outputs they are moving away from “what did we provide” to “why do we matter?”
Output statistics can demonstrate the “capacity utilization” of library services, which is only one dimension in the determination of the effectiveness of the library. On the other hand, outcome measurements can demonstrate how well a library is meeting the (information) needs of its users (Curry Lance et al. 2001; Dugan and Hernon 2002; Dugan et al. 2009). 12
In Table 1, the distinction between outputs and outcomes is clarified with two examples. The leftmost column contains the library’s story as it is usually told in policy documents and mission statements. ‘Mission statements reflect the desire of librarians to show that libraries show a vital role in their community’ (Durrance and Fisher, 2003: 543 13 ), ‘for many organizations it is written in such general and utopian terms that have little substance’ (Marless and Streatfield, 2001: 170). Mission statements show that libraries seek to strengthen their communities by offering guidance; inform, culturally enrich and empower citizens; contribute to democracy, promote cohesion and inclusion, support local identity, etc. 14
A further step to better illustrate this story is to des-cribe the products, services and activities employed/performed to realize or support the mission. In themselves, these do not shed much light on whether or not the mission is actually realized. The first thing one should know is the extent to which the public is actually ‘reached’ by these services. But what matters most in solidly substantiating or testing the mission, is the extent to which services have factually brought about an effect in people’s lives.
A methodological research program
The research path is divided into two phases, each comprising several steps. The first phase comprises a preliminary investigation that consists of a literature study, a qualitative study and a quantitative survey research. The results of the first two steps (literature and qualitative study) will give a first glimpse at the possible domains that libraries have an impact on, at least in Dutch society. In the third and last step of this phase these domains will be converted into a questionnaire in order to get a first grip on the user’s experience on outcome: what (general) benefits that we found in the qualitative and literature study do they actually recognize and/or experience? The findings stemming from these first three research steps help us identify and conceptually enrich domains of possible outcome and will result in a theoretical framework that will guide the development of a measuring instrument.
The development of the actual instrument(s) is the central focus of the second stage of the research project. Insights from the literature study, qualitative study, survey research, as well as from an expert meeting that will be organized in the autumn of 2012, will be used to develop one or multiple questionnaires geared at validly and reliably demonstrating the societal value of public libraries.
This questionnaire offers the operational definitions of the outcome concept that we choose to study and offers ways to measure it. For this purpose the concept will be split up into research dimensions (the qualities of the concept), indicators (measurable evidence, i.e. actions or behaviours) and, finally, questions. During the process of formulating the questionnaire, it will be elaborately pretested before it is ready to be administered. Every test phase consists of a preliminary test to determine its effectiveness and problems (administered to friends or acquaintances) and a formal pretest to examine patterns of response (administered to a trial group that is as similar as possible to the actual survey administration). 15 On the basis of the preliminary pretest and formal pretest revisions will be made and, if regarded necessary, tested again.
A first glimpse on the results of Phase 1, stages 1 and 2
Phase 1, stage 1: literature review and constructing theoretical framework – results
There is a growing body of national and international studies that have attempted to measure the value of public libraries to their communities, often as an instrument in advocacy efforts on behalf of public libraries. These studies are connected with various projects, in various nations, across different research populations. The populations under study range from national, to statewide, to a few libraries and their communities. A wide range of methodologies is used: often questionnaires were selected as a tool to assess library impact, but qualitative methods (interviews and focus groups) were also applied to look for evidence or anecdotal stories to support conclusions.
Furthermore, the literature varies from describing general or overall studies, covering multiple societal domains, to studies that focus on one specific field of impact (e.g. social / economic) in particular.
Finally, outcomes were established among different target groups (users / non-users; library staff; stakeholders; local / national community leaders) and the ones reported / found / expressed vary from very abstract or intangible (e.g. contribution to democracy) to more concrete (tangible / manifest / observable) ones (e.g. supporting language skills).
Next we will introduce some major studies dealing with the outcome of public libraries conducted in different countries. We do not aim to offer a comprehensive or even exhaustive review of the broad literature, but to shed light on the breadth of the subject under study.
US Impact study (USA) 16
In the US the Global Libraries Programme of the Bill & Melissa Gates Foundation systematically evaluates the effects of Internet PCs in the library on its users. Findings of the so called ‘US Impact study’ were based on nearly 50,000 completed surveys from patrons of over 400 public libraries across the country and 319 interviews with users, non-users, staff, administrators, funding agencies, and other community agencies in four case study sites from all over the country. Results show that Internet access is now one of the most sought-after public library services, and it is used by nearly half of all visitors. The overall purpose for using library computers is to perform both life-changing and routine tasks, such as finding work, applying for college, securing government benefits, and learn about critical medical treatments. The variety of fields where library influence was perceived was reduced to seven categories: social connection (maintain personal connections); education (e.g. using library computers to do schoolwork and taking online classes); employment (e.g. search for job opportunities; submit an application online or work on a resumé); health and wellness (learning about medical conditions, finding health care providers, and assessing health insurance options); e-government (e.g. learn about laws and regulations, find out about a government program or service); community and civic engagement (e.g. learn about politics, news, and the community, keeping up with current events); personal finance (e.g. manage personal finances, online banking and making purchases online). Although many different types of residents use public library computer and Internet services, libraries appear to be particularly effective in addressing the needs of families who still lack access in their homes or elsewhere. But also when they do have an Internet connection at home, they use the library, because they either wanted technical help from a librarian, they competed with each other for access to the computer at home, or simply wanted to work somewhere more peaceful and inviting than a crowded coffee shop or a hectic unemployment office.
Perceptions of public libraries in Africa
In six African countries, eIFL has asked users and non-users, librarians, local and national government officials about benefits from, and impact of, public libraries. 17 The study was conducted from December to July 2011 in Ethiopia, Ghana, Kenya, Tanzania, Uganda and Zimbabwe. The goal of the study was to understand the perceptions of national and local stakeholders (municipalities, ministries, public agencies, media, etc.) and the public (including non-users) on the potential of public libraries. Findings show that all groups surveyed seem to agree that libraries are essential to individuals as well as communities in general and that they have the potential to contribute to community development in important areas such as health, employment and agriculture. The main fields of impact as perceived by the respondents correspond to a large extent with the ones in the US impact study: education (e.g. children’s learning, literacy), economic development (productivity enhancement, cost savings, access to new ideas), health (e.g. access to health information, space for health-related events), communication (build new (online) social relationships, reach distant friends and family); culture (i.e. collect and promote local content, provide access to resources); social inclusion and community development (i.e. provide meeting space, serve needs of disadvantaged populations); citizen empowerment, democracy and e-government (i.e. access to and dissemination of government information, civic space for discussions, opinion sharing and exchange); agriculture outcomes (e.g. information on weather or pricing, planting and maintaining crops); information society and digital divide (technology skills and free access to information technologies).
Libraries Building Communities study (Australia) 18
In Australia, the Library Board of Victoria and the Victoria Public Library initiated an extensive research project aiming at increasing community awareness of the range of public library services and showing how public libraries can help achieve government policy goals. Several reports were written in which the results of interviews, surveys and focus groups with almost 10,000 people were presented. Information was gathered from a diverse range of stakeholders – including library users, non-users, library staff and co-mmunity leaders such as local councillors, bureaucrats, business people, school principals and teachers, and people working in key community organizations such as maternal and child health and religious gro-ups. The study shows that libraries and librarians make a fundamental contribution to the communities in four key areas: overcoming the digital divide; creating informed communities; convenient and comfortable places of learning and building social capital. More specifically, findings show that on the individual level, the library contributes by providing access to information (stimulating and supporting information discovery; providing access to multilingual services; helping culturally and linguistically diverse communities; mediating between the user and the information available and maintaining local history and culture, and helping individuals to develop their skills (promoting learning in infancy and childhood; supporting young people/students; supporting basic literacy and encouraging reading; providing access to new technology and skills; stimulate thinking). On the community level, libraries are perceived to add value in the fields of: social interaction; promoting social inclusion; bridging the generation gap and providing a focal point for the community.
Participants in the Libraries Building Communities project were also asked to place a monetary value on the library services available to them. The findings suggest that a large library with about 150,000 registered borrowers would be adding value to users of about $730 million each year and a small library with about 20,000 users somewhere in the order of $10 million. This far exceeds the annual expenditure on public libraries – which varies from a minimum of $350,000 to a maximum of $11 million.
Enriching communities: The value of libraries in New South Wales (Australia) 19
Another Australian research project was conducted more recently by the Library Council of New South Wales. A variety of methodologies was utilized to adequately explore the subject, including: a survey inviting input from the library managers of all public library services; ten case studies for a more in-depth evaluation of stakeholders’ views and interviews with representatives from nine external organizations to better understand how public libraries benefit other institutions. The project has demonstrated that public libraries sustain the community and contribute positively to several fields reflecting four types of wellbeing in society: social (offer safe, harmonious, welcoming and inclusive environment; promote acceptance and understanding of others; ensure free and equitable access to collections; address the needs of specific target groups; contribute to develop, maintain and improve literacy levels; and preserve the past through extensive local and family history collections); cultural (play an active role in local cultural coordinating committee; participate in literary events; celeb-rate cultural diversity; work with local theatres to promote events; host local artists and travelling exhibitions; cooperate with other cultural institutions; keep alive the names and work of significant Australians); economic (enable users to avoid or reduce expenditures, enlarge job opportunities; support local businesses; build programs to establish new libraries or extend/refurbish existing libraries; assist small to medium-sized enterprises to maintain high professional standards and compete with larger organizations; contribute to tourism); environmental.
Concerning monetary benefits, the study found that public libraries generated an economic benefit equivalent to $4.24 per invested dollar (which means freeing the same amount of funds for use elsewhere) and that NSW public libraries generated $2.82 of economic activity for each dollar expended on public libraries.
Public Libraries: Arenas for Citizenship (Norway) 20
In Norway, a research project called Public Libraries – Arenas for Citizenship (PLACE) has started in 2007 to gain more understanding in the role of the public library in the making of social capital.
The project aims at investigating the possibility of developing public libraries in multicultural local communities into arenas where different kinds of meetings can take place. Two quantitative surveys were carried out in 2006 and 2011 (no publication to date) in three communities in Oslo and Tromsø to measure the role of the library as a meeting place as reflected in actual use.
Results indicate that the library is a complex meeting place with a range of meetings along a continuum from high intensive to low intensive. The different kinds of meetings that can take place in the library are grouped into six categories:
the library as a public space and a low threshold social meeting place, a place for accidental meetings and conversations, for making appointments to do something else a meeting place between meeting places, an arena where you can find information about and be directed to other meeting places in the community a public sphere in its own right where political and cultural ideas are presented and discussed (e.g. participation in meetings with authors or politicians, search information on community issues) an arena where you can acquire the information and knowledge you need to be an active, involved and participating citizen an arena where you live out professional or private involvements together with colleagues and friends (joint activities) an arena for virtual meetings on the web.
The library is heavily used as a meeting place and the type of meeting with the highest score is that of encounters with people belonging to a different culture, where one has observed and experienced things about these cultures. The library thus appears to be a place where, in a safe environment and in an unobtrusive way, people are exposed to the complexity of the digital and multicultural society and learn something about multiculturalism.
Monetary value of the public library
Also in Norway, Svanhild Aabø 21 conducted research in which the value of public libraries was estimated from the population’s perspective, in monetary terms. Aabø uses the contingent valuation (CV) method, which is based on the individual’s own assessment of the good to be valued and implies “that respondents are asked to state their values of a change in the provision of a nonmarket good, in the form of willingness to pay (WTP) for an improvement or willingness to accept (WTA) a change to the worse”(Aabø, 2005b:178). In this case, respondents were asked to make a trade-off between closing the library to save money for investment in other community fields, such as care for the elderly and education. The overall conclusion from the empirical study is that, on average, Norwegian households value the benefits from public libraries clearly higher than the costs of providing the library services, demonstrating a cost-benefit ratio of approximately 1:4 at the national level. This means that for each NOK of taxes that is used on public libraries, the population gets four times back in benefits from them.
Outcomes in every day life (Finland)
In Finland a quantitative study was performed to explore the benefits in everyday life that adult citizens derive from using public libraries. 22 For measuring the outcomes of public libraries the researchers asked a sample of 1000 respondents from 15–79 years to rate how frequently they have benefited from public library services in the following 22 segments of human daily life: education (finding educational opportunities; completing formal education; work related educational development; self-education during leisure time); work and business (finding jobs; executing specific work tasks; developing job skills); everyday activities (household; child care and schooling; housing; consumer issues; health; travel and holidays; social relations); leisure time (reading fiction / non-fiction; cultural activities; creative activities; outdoor activities; exercise sports; interest in nature; interest in history or society; participating in and following public discussions). After a factor analysis, these benefits were grouped into three categories: everyday activities, benefits in cultural interests, and career benefits. Results show that public libraries seem to be used most commonly for recreational, cultural or educational literary purposes in leisure time. Libraries were least successful in core everyday activities relating to family and household, but did considerably well in other aspects like health and travel.
Dividends: the value of public libraries in Canada 23
In 1996, the Library Action Committee of the Book and Periodical Council undertook a research project to review the importance of public libraries to library users, suppliers, publishers, retailers and other businesses, as well as to Canadian culture. An inventory of several Canadian sources, as presented in a discussion paper that was written by Leslie Fitch and Jody Warner, outlines the many areas in which public libraries play a vital role in Canada. The researchers conclude that public libraries: are cost-effective information providers; support the local economy; contribute to the economic wellbeing of local businesses; improve the market worth of a community; invest in the economy; benefit local businesses; provide high-skill and high-tech jobs; support the cultural industry sector and Canadian culture, support a democratic society, support children and students and lifelong learning; help people to make informed personal decisions and ensure the information highway is accessible to all Canadians.
Overall outcome framework
Based on these studies discussed above and quite a few additional sources in related areas, we derived five categories, or main domains (areas), of (possible) impact: educational, cultural, social, economical and affective. In these domains, the library is assumed and/or proven to be influential, through its function as a warm, welcoming and neutral meeting place, as a supplier (disseminator) of information, as a multimedia learning environment, organizer of or host for cultural activities and debates, etc.
Overview of domains of library impact based on literature review.
The particular concepts that the outcome domains contain are shown in Figure 2. They concern both individual and community outcome dimensions. As is by definition the case with models, this one presents a simplified representation of reality. Relationships between the dimensions covered by the main outcome domains are not shown in the model, whereas in reality these specific concepts (and, by consequence, the dimensions) are narrowly intertwined, both within and between main dimensions; e.g. career management skills or opportunities (economical domain) are obviously related to cognitive skills (educational domain). And preservation of and access to cultural heritage (cultural domain) will be linked to the common identity of society (social domain). Also, the model does not describe the mechanisms through which impact comes about. Finally, because affective outcomes flow directly from library use (i.e. fun in using a library service) as well as indirectly through outcomes that are experienced in the other domains (i.e. self confidence because of reaching educational goals), as in the model above it is situated on a different level than the other four domains.
Specified overview of the domains and dimensions of library impact.
Phase 1, stage 2: verifying/expanding framework: qualitative librarians and stakeholders study – results
In order to examine and further enrich the outcome domains and dimensions identified in the literature study, specifically for Dutch society, in the second stage a qualitative study was conducted in November / December 2011. 24 The aim of this study was to explore the concept of public library outcome in the Netherlands, in all its breadth. In the first research stages, in-depth interviews were held with several stakeholders: seven library directors (hereby covering big city as well as countryside libraries of different sizes and geographically dispersed across the country), experts in specific relevant fields of interest (liveability on the countryside and reading promotion), and partners in the cultural field. These interviews appeared not to bring about much new or unexpected information, thus supporting the resulting model from the literature review (Figure 2). Also, library outcomes were formulated in relatively abstract terms by the stakeholders. In the remainder of the study, the focus was therefore shifted from directors and stakeholders/partners to the user of the library.
The main findings of the study were the following. The dimensions of the literature review (Figure 2) were replicated in the qualitative research and remained intact. The affective dimension was indeed found to be supportive of the other four and can only partly be viewed as an independent one. The research indicated further that a distinction between intrinsic and extrinsic value of the public library’s functioning is of importance. Patrons are not always aware of many outcomes, because some outcomes reveal themselves only after quite a while. Furthermore, since they report using the library for the intrinsic (affective) reward of reading, they may not acknowledge the additional consequences of that activity in their lives.
Another finding is the complicated nature of disentangling outcomes from various sources. It is difficult to establish with any certainty that any particular outcome was the direct result of a specific intervention. The library is but one of many organizations which are all striving for a societal ‘return’ for their investments, quite often in the same domains as the public library. This finding serves as a warning sign for the remainder of the research program, which has (at the time of writing, in May 2012) still to be carried out. We will be turning to it now, by way of conclusion.
Looking ahead: What’s next?
Phase 1, stage 3: verifying/expanding the framework: quantitative (non) users study
Now that the reference framework for the societal outcomes has been developed and qualitatively tested and enriched, it is necessary to try and verify it by way of a quantitative test with its main stakeholders: the users and non-users. The main aim of this last stage in the first phase of the program is to have another check. The outcome dimensions in the literature and the stakeholders’ views are very likely to be coloured by societal discourses about the public library as ‘storehouses of knowledge’, ‘cornerstones of democracy and information freedom’ etc. Noble and perhaps also apt as these labels may be, it could very well be that in the views of the users, their societal importance is of a less elevated nature. In any case, the user interviews in the qualitative study pointed in that direction. Users worded the benefits they derive from the public library’s services more in intrinsic than extrinsic terms.
In August/September 2012, we will have a short quantitative survey research carried out among the Dutch population. The questionnaire to be administered will contain items measuring the four main dimensions – educational, economic, social and cultural benefits, plus affective benefits. As far as possible, the questions will copy those of earlier surveys 25 for cross-national comparison purposes. Other questions will be developed and pre-tested in order to have all dimensions of the framework covered, both in terms of intrinsic and extrinsic benefits people can derive from the public library.
Phase 2: developing and testing new measurement instrument
Once the first phase will have been completed (end of 2012), we will have an empirically validated theoretical framework for measuring societal outcomes of public library services. The next and crucial step will then be to further develop and pre-test measurement instruments, in order to arrive at a set of instruments with which the societal impact of public libraries could be tested in a valid and reliable manner.
As has become clear from the literature review and the qualitative study, it is far from certain that ‘the’ outcomes of the public library system are the same from various perspectives, in particular institutional versus individual ones. To give an extreme example: an institution like a primary school might find that it benefits considerably from the library’s collections and its support in reading promotion activities. The school’s teachers may feel freed from the burden of having to incorporate reading promotion in their curricula without having learned the necessary skills themselves. From the point of view of the individual children, this institutional outcome may not be desirable at all, for now it is left to them to find the way to the library and its reading promotion activities.
So in this second phase of our research program, we will have to decide how to deal with these divergent perspectives on outcomes. Will it be feasible to incorporate all stakeholders’ viewpoints in one measurement instrument, or will we have to resort to multiple instruments, each geared at another group of stakeholders? Furthermore, will it be possible to cover all dimensions – educational, economic, social, cultural – in all of these cases? A decision will probably have to be made to exclude some of the stakeholders and/or some of the dimensions in order to limit the budgetary requirements. As should be clear, the instrument(s) is (are) not developed for the sake of this methodological program itself, but in order to be put into practice in the real world. Practical considerations therefore will have to be leading in this phase.
After the practical boundaries have been set, the project will enter a phase of developing, testing, improving, re-testing and re-improving the measurement instruments. This phase will consequently be of a methodological nature, possibly using multitrait-multimethod designs to judge the validity and reliability of the instruments. 26 In 2014, we hope to be able to present the validated instrument(s) to the library community.
Footnotes
Paper presented at the World Library and Information Congress: 78th IFLA General Conference and Assembly, 11–78 August 2012, Helsinki, Finland, in session 76 — Crisis? What crisis? The use of statistics and data for libraries at a turning point — Statistics and Evaluation.