Cyber society and digital policies: Challenges to evaluation? 1

Abstract

In this article, the authors explore the relationship between (impact) evaluations and cyber society, in particular digital policies. There appears to be a gap between the pace at which internet and digital policies are penetrating society and the attention professional evaluators are paying to these policies. First, we present an analytical framework for studying these policies, while examples of policies are described and categorized. Next, efforts evaluators have made in this area are described based on three different sources. We begin by looking at three public sector institutions and their evaluation activities; then study what peer reviewed evaluation journals have published about digital policies. Finally, we approach a specific field of interventions: digital piracy. A systematic review of 14 studies evaluating the impact of (hypothetical) interventions and mechanisms is presented. Although several studies use designs that are questionable, there is evidence that some mechanisms are able to influence behavioural intentions regarding illegal downloading.

Keywords

digital piracy digital policies evaluation journals internet

Introduction and questions

In her article on the internet and public policy, Margetts (2009: 3) presented data showing that ‘for many people across the world, large chunks of their social, economic and political life have moved online’. This is even more true in 2011, as there are now a little over 2.1 billion internet users worldwide. Compared with 2000, this is an increase of almost 500 percent. Although worldwide internet penetration is around 30 percent, in Europe it is almost 60 percent and in the USA almost 80 percent (http://www.internetworldstats.com/stats4.htm). This development has many consequences:

As well as doing things that they used to do offline, people are doing new things, particularly with the growth of so-called Web 2.0 applications, where users can easily produce as well as consume content themselves. Examples include social networking sites, used by around a third of Internet users; photo and video-sharing sites . . . and social media; and peer-produced information goods such as the online user-generated encyclopaedia Wikipedia, the English language version of which has over 3 million articles and 11 million registered users. (Margetts, 2009: 3)

Similar developments have taken place in the economic world, which has also moved online, ‘particularly in terms of reduced transaction costs, cross-border money flows and spiralling complexity. New “peer-to-peer” markets have developed, including Internet auction “houses” such as e-Bay’ (Margetts, 2009: 3).

The ‘digitization’ of society is not restricted to economic or social life. There are also consequences for politics and policy making, including interest group mobilization through the internet, e-government and the birth of digital policy strategies, programmes and interventions.² In the EC (European Commission) Digital Agenda there are examples of strategies to increase social inclusion through the internet and to stimulate e-health (interventions). Similar developments are taking place in other policy fields, such as foreign affairs and developmental aid. In the world of crime and justice, there are policy strategies and interventions focusing on preventing and reducing cybercrime and cyber insecurity in diverse forms, such as online child pornography, theft of intellectual property and infringement of copyrights (also known as digital piracy), larceny, hacking, skimming, digital warfare and cyber terrorism. In relation to the latter, the OECD (Sommer and Brown, 2011: 6) recently discussed ‘cyberweaponry [like] viruses, worms, Trojans . . . distributed denial of service using botnets [and] root-kits’ and argued that ‘it is a safe prediction that the use of cyberweaponry will shortly become ubiquitous’.

The internet, its practices, applications and policies are nowadays widespread. That means that in large parts of the world people are no longer ‘visiting the web’, but are almost ‘living in the web’, as Van’t Hof et al. (2011) put it. Deibert et al. (2010: 9) frame it a little differently but the message is the same:

Today, with always-on portable devices that are fully connected to the Internet, and much of society’s transaction mediated through ICT, cyberspace is not so much a distinct realm as it is the very environment we inhabit. Our lives have been digitally disassembled, disaggregated, and dispersed into multiple digital domains.

In line with the concept of the ‘digital state’ (Borins et al., 2007), this article focuses on digital policies (i.e. strategies, programmes and interventions), their impact and the role of evaluations. Based on conceptual frameworks developed by the OECD (2009a) and Hood and Margetts (2007), these policies are characterized and categorized. We also discuss why it is important to evaluate them and in particular their impact. To better understand what is occurring with regard to the evaluation of digital policies, we use three different sources. The first source is websites of several public sector institutions (the European Commission’s Information Society and Media Directorate-General, the OECD and the central government of the Netherlands). We investigated whether these institutions are active in commissioning and/or undertaking evaluations of (the impact of) digital policies. The second source shifts the perspective from public sector organizations and ‘their’ evaluations to peer-reviewed international evaluation journals. To what extent are these journals publishing papers on digital policies and interventions? As journals are independent of (governmental) principals and sponsors of evaluations, apply a peer-review system and use (methodological) standards during the review process, it is important to find out to what extent this ‘side’ of the professional evaluation world is addressing digital strategies, programmes and interventions. Unfortunately, we found that not much is occurring. This being the case, we shifted the focus to a third source, a specific type of digital behaviour. What can be learned from a systematic review of studies on the impact of (hypothetical) interventions trying to reduce or prevent digital piracy? As we will see, most of the studies can be labelled as ‘mechanism evaluations’ (Ludwig et al., 2011: 20).

This article is not a full ‘evidence-based’ inventory and analysis of the state of the art in the evaluation of digital policies, nor is it intended to be. Our goal is to shed some light on what is happening in this field by raising a number of issues, and suggest and exemplify needed approaches on the relationship between evaluation and the internet. The study is therefore largely exploratory in nature.

Public policy and the internet: An analytical framework and examples of digital policy strategies, programmes and interventions

Digital policies have their origins in (central) governments, supranational institutions and public-private organizations, including (global) networks and stakeholder groups³ (Mueller, 2010). What are examples of digital policies that have been implemented over recent years? Although the OECD (2009a) model of the information society (Figure 1) is much more broad, we restricted our search activities to two of their categories: ‘information and digital content’ and to ‘ICT in a wider context’. We did not address policies regarding the technical infrastructure like broadband affordability or wireless connectivity and supply and demand aspects. We also left out digital policies that try to have an impact on the world of finance, e-businesses, procurement and related issues.⁴

Figure 1.

OECD conceptual model of the information society (OECD, 2009)

By using Google and the Web of Science and by searching websites of the EU, OECD and the Netherlands central government, we located a number of digital policies. We distinguished between those that were strategies, programmes and interventions as follows.

Strategies

An example of a policy strategy is the EC Digital Agenda which describes not only goals but also more than 100 ‘actions’; it is one of the flagship initiatives of Europe 2020. The 2006 Riga declaration is another example and has as its core theme ‘ICT for an inclusive society’; it formulated 10 priorities and the commitment to act on these. Treaties on cybercrime, cyber (in)security and dealing with net neutrality can also be seen as policy strategies. Strategies are usually of a more overarching nature than programmes or interventions and actions.

Programmes

Programmes are usually combinations of different interventions. An example is the EU sponsored SHAREIT, which aims to promote cohesion through making available ICT tools, courses and insights into technology to socially and economically vulnerable groups (http://www.peopleshareit.eu/). The programme strives to contribute to reinforcing cohesion and social welfare.

Interventions

Interventions are of a more ‘direct’ nature. Examples are digital duties of care that internet service providers have to comply with in the fight against cyber crime, DPIs (deep packet inspections) to trace the content of digital communication, e-health interventions, social software to stimulate cohesion and e-services to improve the quality of delivery by governments.

Hood and Margetts (2007) developed a framework consisting of types of tools that are available to policy makers, which helps to structure this complex picture:

When making policy, governments are trying to influence social behaviour and shape the world outside . . . to undertake this task, governments have four basic types of tool in their toolkit. First, nodality denotes the property of being ‘nodal’ to information and social networks and having the capacity to disseminate and collect information. Second, authority denotes the possession of legal or official power to demand, forbid, guarantee or adjudicate. Third, treasure denotes the possession of money or that which can be freely exchanged. Fourth, organizational capacity denotes the possession of a stock of people and skills, land, buildings, materials, computers and equipment, somehow arranged. (Margetts 2009: 6)

In line with Margetts’ categorization, Table 1 presents examples. Apart from the interesting diversity of policy activities that Table 1 shows, three other findings are worth mentioning. The first is that nodality is present in almost all interventions and programmes; given the crucial role information plays in the world of the internet, this is not surprising. The second finding is that the financial dimension of digital policies (treasure) is not dominant compared to the emphasis ‘traditional’ policies attach to grants, incentives, loans, guarantees, credits and other examples of ‘treasure’. Third, policies focused on reducing or preventing ‘collective bads’ (like cyber crime) are characterized more by using authority (sometimes as part of criminal law) than policies focused on producing ‘collective goods’.

Table 1.

Examples of digital strategies, programs and interventions

Description	Tool(s) used	Source
The EC Digital Agenda is the European commission’s strategy in the digital area; it includes over 100 ‘actions’.	All 4	http://ec.europa.eu/information_society/digital-agenda/index_en.htm
The policy strategy on guaranteeing net neutrality aims at guaranteeing interoperability and non-discrimination of the internet.	Nodality Authority	http://immi.is/Icelandic_Modern_Media_Initiative
Treaties on cyber-security and legislation. The Cybercrime Convention of the EU is an example as are laws aiming at protecting against unauthorized access to computer material.	All 4	Sommer and Brown (2011); Wall and Williams (2007)
Deep packet inspection is a function embedded into an ICT-network. It enables the network owner to analyse internet traffic in real time and in terms of its content.	Authority Nodality Organizational capacity	Mueller (2010: 176).
Notice-and-Take-Down Code of Conduct. The goal of the NTD is to give clarity to intermediaries, such as Internet Service Providers about how they should take action when they receive a request to remove information from the Internet. This can involve criminal phishing sites or child pornography, but also instances of misuse of logos or discrimination.	Nodality Authority Organizational capacity	http://www.ecp-epn.nl/werkgroep-notice-and-takedown; Eyck et al (2010).
The Hadopi-intervention (‘three strikes and you’re out’) tries to prevent and reduce digital piracy. Initially, individuals are warned through an email. ‘The second time the presence of the pirate is detected on a P2P network a certified letter is sent, at the third infringement the ISP is supposed to suspend the internet connection of the pirate’.	Nodality Authority	Dejean et al. (2011:5); http://newteevee.com/2010/08/04/hadopi-gone-wild-france-plans-spyware-for-three-strikes/; http://www.scribd.com/doc/35088715/docprojet-SFH
E-health, e-learning, e-planning, e-participation, e-government, e-inclusion and ‘lookalikes’ as well as strategies to stimulate the use of them. It concerns hundreds of digital interventions and programs ranging from tele-monitoring of sickness behavior and e-therapies to using Virtual Reality to stimulate participation in environmental planning. An example of a strategy is the EU e-Government Action plan 2011–15 that outlines the way forward to implement the Malmö Declaration.	Nodality Treasure	Van Os (2009); Heldal (2007); Postel et al. (2008); Siegen Universität et al. (2010).
Information campaigns are interventions that use mass media, the internet and other forms of communication to stimulate a safe[r] use of internet. In Europe known as the Safer Internet Program.	Nodality	Leeuw and Bogaerts (2009); http://ec.europa.eu/information_society/activities/sip/index_en.htm
Circumvention policy. This policy is to support (financially) the development of new digital tools that enable citizens to exercise their rights of free expression by circumventing politically motivated censorship and help them work around government-imposed firewalls. An example is the ‘panic button’, an app that will both wipe out the phone’s address book and emit emergency alerts to other activists.	Treasure Nodality	http://www.state.gov/secretary/rm/2010/01/135519.htm); http://www.reuters.com/article/2011/03/25/us-rights-usa-technology-idUSTRE72O6DH20110325
WARP (Warning, Advance and Reporting Point). This intervention has been developed to support defense against (digital) attacks. The purpose is to provide a specific community with the capability to share security related information – both problems and solutions – and thereby to develop more secure and responsive environments.	Nodality	http://www.warp.gov.uk/background.html

Evaluating digital policy strategies, programmes and interventions

How important is it to evaluate (the impact of) these and other digital policy strategies, programmes and interventions? For those who – in the earlier years of the internet – had a techno-utopian dream and saw it as the ultimate ungovernable and free space (Barlow, 1996; Hofmann, 2010), this question was largely irrelevant. Who would be interested in evaluations when living in such a ‘beautiful’ world? For those who have or had a more dystopian belief about what the internet would bring society, this question would also probably be irrelevant. The spread of the internet and its ‘dark side’ would probably be ‘unstoppable’, so why think about doing evaluations?⁵ A few decades later things have changed, as Van Dijk (2010: 6) argues: ‘twenty five years of experience affords to strike a better informed balance, both empirically and theoretically than the utopian and dystopian speculations of the 1980s and 1990s’. Evaluation is part of that balance.⁶

There are various reasons why evaluation of digital policies should be undertaken:

⁫ When digital policies do not work or produce serious and negative side effects, there may be consequences for millions of people and organizations worldwide;

⁫ Digital policies incur financial costs that can be higher than the benefits;

⁫ The policies may be based on (behavioural) assumptions that are invalid.

This leads us to investigate current evaluation practice, using three sources of information.

The first source: Evaluating digital policies in three public sector institutions

The breadth of scope of two supranational organizations (OECD and EC [in particular, its Information Society and Media Directorate-General]) and the history of evaluation in the Netherlands (Bemelmans-Videc, 2002; Leeuw, 2009) are the prime reasons to select these institutions. As our goal is not to cover public sector institutions in general, but primarily to present examples of what is occurring with regards to evaluation and digital policies, we selected these institutions.

From scanning the OECD website (http://www.oecd.org/home), it is apparent that in several ways the OECD does pay attention to evaluating digital policies. Considering various documents available on the site: these include the Guide to Measuring the Information Society, 2009 (OECD, 2009b), and the Digital Economy Papers Series. Also, country case studies on this topic are published as well as Policy Briefs. The Guide presents numerous performance indicators while a large-scale case study on the Spanish Plan Avanza and its digital policies was published in 2009. The OECD also studies the way in which governments monitor and evaluate e-government (OECD, 2006).

Regarding the impact of digital policies on society, it appears that ‘in contrast to the strong interest in impacts of ICT use by businesses, there has been little work done on impacts of use by households and individuals’ (OECD, 2009b: 54). A similar statement was published about online citizen engagement: ‘Currently there is a clear lack of an accepted framework on how to evaluate and measure the impact of e-engagement systems’ (OECD, 2003a: 89). In several of the OECD Digital Economy Papers attention is paid to the impact question; already in 1995 a study was published that aimed to determine which IT diffusion policies for SMEs were successful (OECD, 1995). Several years later the importance of databases to help assess the ‘information society’ from a social science perspective was the subject matter of another OECD publication (OECD, 2000). More recently the OECD (2003) tried to determine the benefits of ICT in a digital economy.

Scanning the website of the EC Information Society & Media Directorate-General (http://ec.europa.eu/dgs/information_society/index_en.htm ) also reveals a serious interest in the evaluation of digital strategies, programmes and interventions. Central is the Digital Agenda for Europe which is accompanied by a programme for monitoring and evaluation (M & E), published more or less annually and developed in line with the EC guidelines on impact studies and ex ante evaluations. This M & E programme can be found on the website; the studies are published, and the comments of the EC Impact Assessment Board (http://ec.europa.eu/dgs/information_society/evaluation/impact_assessment/index_en.htm) and the working papers of the EC staff are available on the website. The Board comments on the (methodology of) evaluations. Topics covered by the M & E activities of this Directorate of the EC range from privacy protection policies and the role of ICT to facilitate the transition to an energy-efficient, low-carbon economy in Europe to the interim evaluation of the Safer Internet Programme, the reduction of digital piracy and the efficiency and effectiveness of a European Electronic Toll Service.

However, many of the studies commissioned are not primarily focused on the impact on society but on legal, commercial or technical aspects. However, in 2010 two studies were published on the social and the economic impact of digitization in Europe.⁷ These covered the impact of the digital agenda and its actions on e-participation in policy-making, education and lifelong learning, health, competitiveness and economic growth, community development, work, consumption and globalization (Siegen Universität et al., 2010; Van Reenen et al., 2010). More EC-sponsored impact studies are available, in which sometimes a discussion on the robustness of the methodology can be found (Steyaert, 2010).

The Netherlands Government has organized monitoring and evaluating of its ‘digital’ activities and policies in several ways (http://www.rijksoverheid.nl/onderwerpen/ict/digitale-agenda-in-beeld). With regard to the Digital agenda of the Netherlands, no central evaluation regime is developed or used, as many different policy fields are covered. The Ministry of Finance has issued regulations regarding the quality of evaluations, their independence and use and these also apply to the digital world. Various organizations are undertaking (impact) evaluations: independent governmental research organizations like the SCP (Netherlands Institute for Social research), CPB (Netherlands Bureau for Economic Policy Analysis), WODC (Justice Research and Documentation Centre) and WRR (Scientific Council for Government Policy) as well as other institutes (partly or largely) funded by the central government.⁸ Depending on their field of expertise they either undertake evaluations themselves or commission them. They publish their (evaluation and research) programmes, as well as the studies. Topics range from evaluating the impact of e-government and privacy enhancement approaches, interventions trying to fight cyber-crime and security, behavioural e-health therapies, online education, digital cultural policy making and many others.

Oversight bodies like the Netherlands Health Inspectorate and the Education Inspectorate also publish evaluations and inspections of the impact of ICT projects in schools, of electronic pharmacy and other topics. The Oversight Authority OPTA (http://www.opta.nl/en) reviews and inspects all electronic communications, while in the field of cyber security the Cyber Security and Incident Response Team (http://www.govcert.nl/organisatie) is also involved in doing work of an evaluative nature, for example when security incidents have taken place. Recently, a cyber security board has been set up that will develop a research agenda for this field, including evaluations.

The three institutions selected are active in monitoring and evaluating digital policies and other ICT topics; they have the infrastructure and impact studies are undertaken. The reports can usually be found on the institutions’ websites. Probably most of them belong to the ‘grey literature’. Although search engines allow grey literature to be located much more quickly than previously, it can be assumed that most of the studies will not have been assessed by means of an anonymous review, as occurs in (peer-reviewed) evaluation journals. Therefore, we now turn to the second source of this article: the coverage by evaluation journals of digital policies.

The second source: Evaluating digital policies in 10 international and peer-reviewed evaluation journals

To determine to what extent evaluation journals publish about digital strategies, programmes or interventions, we carried out a word count of 10 journals. We used the journal’s search possibilities to count the number of times the words ‘internet’, ‘digital’, ‘web’, ‘cyber’ or ‘digital policy’ can be found in full text and as keywords (see Table 2).

Table 2.

Appearance of relevant ‘digital (policy)’ words in evaluation journals¹

Journal (data were collected in January/February 2011)	Words internet or web or digital or cyber found in full text	Words internet or cyber or digital or web found in keywords	Words digital and policy found in full text/keywords
Evaluation (1995-)	77 times	—	5/0
Evaluation and Program Planning (1981-)	228 times	*	43/0*
Assessment and Evaluation in Higher Education (1981-)	174 times **	3 times	25/**
Evaluation and Research in Education (1981)	11 times **	—	8/0
Educational Evaluation and Policy Analysis (1981)	67 times	1 time	3/0
Evaluation Review (1981-)	76 times	3 times	9/0
Evaluation and the Health Professions (1987-)	103 times	4 times	2/0
New Directions for Evaluation (1981-)	127 times	*	6/*
American Journal of Evaluation (1981-)	219 times	4 times	18/0
The Journal of Multidisciplinary Evaluation (2004- )	8 times	*	1/0

Notes:

Number of times the words ‘internet’, ‘digital’, ‘web’ or ‘cyber’ and ‘digital and policy’ appeared in 10 peer-reviewed evaluation journals (English) (since 1981 or since the first time the Journal was published – the year in parenthesis is either the starting year of the journal or the starting point of the search.).

——: not found

) this journal does not make it possible through an electronic way to differentiate between keywords and full text, or this service was not available during the preparation of this paper.

) Only three (combined) search words were possible; ‘cyber’ was excluded.

Full text: to check terms or phrases that appear anywhere in the full-text article;

Keywords: to check terms in ‘keywords’, as applied to articles by authors.

Data were collected in January/February 2011 and in October 2011 by the second author and checked by the first author.

Although the findings of the word count only have exploratory relevance, it is clear that articles in the selected evaluation journals very infrequently use the words ‘internet’, ‘web’, ‘digital’, and ‘cyber’. As an illustration: with regard to the American Journal of Evaluation (AJE), we found that of the approximately 11,000 pages that were published since 1981, only 0.00005 percent of the words were ‘internet, digital, web or cyber’. Acknowledging that in AJE issues published before the mid-1990s, the internet could not be a serious topic because it was not yet on the (public) radar, we checked what happened if we left out the first 18 years of the journal. Then the percentage hardly changed and remained invisible to the naked eye.

For the same journals we did a second search for the combination of ‘digital (and) policy’. Again, this combination could be found in the full text and keywords an extremely small number of times (see Table 2).

To test if these findings would change (substantially) when we would count more specific terms, for six of the journals another word count was done, using the combination of the following words: e-health, tele-medicine, tele-care, digital piracy, illegal downloading, e-government, e-business, e-learning, wiki and ICT. For the American Journal of Evaluation we found 11 references over the same period, for Educational Evaluation and Policy Analysis 6, for Evaluation 20, for Evaluation Review 4, for Evaluation & the Health Professions 6 and for Assessment & Evaluation in Higher Education 41.⁹

The conclusion is that over the years peer-reviewed evaluation journals have hardly published any papers on evaluating digital policies (and internet developments more broadly).

The third source: A systematic review of 14 studies evaluating (hypothetical) interventions on digital piracy

The third source is a systematic review of 14 studies focusing on interventions on digital piracy.¹⁰ Digital piracy is the act of downloading and/or uploading digital goods by individual end-users (such as software, documents, movies and music) from an illegal source without permission from the copyright holder. While the copying of copyrighted materials without permission of or compensation to the copyright holder is not a new phenomenon, the scale of and ease with which this currently can be done is significantly different to before the internet. Until recently, individuals would have to copy audiocassettes, VHS tapes or CD-ROMs directly from the original and one at a time. One of the many downsides of this low-tech copying was that the quality of the copy would usually be inferior to the original. Further copies would result in a decrease in quality, and the process itself could also be rather time consuming. However, with the advent of the digital age, copying has boomed. As Yar (2007) notes, quality degradation no longer affects the copies. Furthermore, other characteristics of the digital era, such as the ease of downloading, the standardization of the media, the near infinite-storage capabilities of modern computers and the widespread availability of high-speed internet has had a large impact on the pervasiveness of this type of behavior.

There is some data available to give us an indication as to the scope and impact of digital piracy. In the case of software, it has been estimated that 42 percent of all software globally in use has been pirated while, in 2010, the value of the pirated software was almost US$ 60 billion (BSA/IDC, 2011). This is clearly an immense figure. However, this particular set of data originates from an industry watchdog. Some have criticized the method with which the industry estimates the impact of digital piracy (Moores, 2008). Nevertheless, non-industry researchers like Levin et al. (2004) also report challenging findings; for example, that 63 percent of US undergraduate students download pirated music. More recently, it has been estimated that 130,000 movies are downloaded illegally, per day (Hennig-Thurau et al., 2007). It also has been estimated that the movie ‘Avatar’ is the most downloaded single movie, with about 21 million downloads.¹¹ The precise impact on the digital industry is still rather unclear, as different studies produce different data.¹²

So, while the jury is still out on the exact impact of digital piracy, both governments and (public-)private organizations are attempting to prevent or limit this type of behaviour. A variety of interventions can be used (and some are used). These include, among others, information campaigns, technical countermeasures, three-strikes legislation, civil litigation and criminal prosecutions. The question that should be asked is whether evaluators are looking into the impact these interventions have on (preventing or reducing) digital piracy?

To answer this question, a systematic review has been performed.¹³ Using the University of Maastricht’s Metasearch option, 46 databases were searched (these include Sage, Web of Science, and others) using the following words: ‘digital piracy’ or ‘illegal downloading’. After applying a number of inclusion and exclusion criteria, this resulted in 118 articles dealing with digital piracy studied from a behavioural perspective, some of which also deal with policy.¹⁴ Only 14 addressed the impact of an anti-piracy intervention. However, most of these studies do not deal with interventions that have already been implemented by governments or (public-private) organizations. Instead, they evaluate proposals for policy interventions or hypothetical interventions in order to judge whether these are likely to be effective. The studies aimed to analyse if the mechanisms underlying future interventions can be made to operate effectively. In line with Ludwig et al.’s (2011: 20) concept of mechanism experiments, we label these 14 studies as ‘mechanism evaluations’. A mechanism experiment ‘does not test a policy: it directly tests the causal mechanism that underlies the policy . . . if we believe we know something about the mechanisms through which the policy might operate, why limit ourselves to using this information only after a policy evaluation has been designed and carried out’ (Ludwig et al., 2011: 22).

Table 3 describes several characteristics of the 14 studies included in the systematic review, including designs and some results. Looking at the studies summarized in Table 3, we draw several conclusions. For one, none of the studies were published in an evaluation journal. In fact, most evaluations originated from journals dealing with either the field of (business) economics or ICT. This suggests that evaluation journals have not (yet) ‘discovered’ the field of digital piracy as a relevant topic.

Table 3.

A systematic review of 14 studies evaluating (hypothetical) interventions/mechanisms designed to reduce/prevent digital piracy

Authors	Intervention/mechanism(s)	Design/Method	Results
Al-Rafee and Rouibah (2010)	Three different types of interventions were tested: a religious intervention, an awareness intervention and the law/regulation intervention.	†/‡/•	The awareness and the religious intervention led to a reduced intention to commit piracy, the legal intervention did not.
Wingrove et al. (2010)	Five compliance mechanisms were identified and tested.	◊/§/•	Deterrence, social norms, personal morality and the obligation to obey the law were related weakly to different degrees of the intention to pirate.
Jeong and Lee (2010)	This intervention tested concerns an increase of the transaction costs of illegal file sharing.	◊/§/•	The study claims that increasing the transaction costs of illegal file sharing is an effective way to inhibit this behaviour.
Li and Nergadze (2009)	Perceived certainty of punishment, perceived stigma of label, awareness of the copyright laws and the potential legal sanctions and consensus with the laws were studied .	◊/§/•	The extra-legal mechanisms are found to be relevant, awareness less so.
Sinha and Mandel (2008)	The question is asked whether legal actions of RIAA (Recording Industry Association of America) had an impact on the number of individuals who are engaged in digital piracy.	‡/•/◊	The results indicate that the RIAA’s efforts to use fear and shame to stop digital piracy may be effective.
Wolfe et al. (2008)	The study explored the utility of computer viruses as a deterrent to music piracy.	◊/§/•	The findings indicate that deterrence, guilt and fear of viruses are important in reducing instances of digital piracy.
Chiu et al. (2008)	Four strategies were evaluated in this study. These include a.o. lower-pricing of legal software, legal communication and product strategies.	◊/§/•	The results indicate that lower-pricing, legal communication, and product strategies all enhance customer intentions toward purchasing legal software.
Grolleau et al. (2008)	This study explored whether donation (by the copyright holders) to charitable causes decreases piracy and whether a higher level of donation results in a lower level of piracy.	¶/†/§.	The level of piracy is lower when a donation mechanism is present.
Levin et al. (2007)	The study examined the effectiveness of various disincentives (threat appeals, attributions of harm, subjective norms).	¶/(†)/§	The use of severe threats seems to be an effective way to diminish the likelihood of downloading, while attributing financial harm to either the company or artist was not found to be particularly effective.
Bhattacharjee et al. (2006)	This study deals with the legislative actions by the RIAA.	•/◊/(‡)	The analysis indicates a mixed success for RIAA’s strategy.
D’Astous et al. (2005)	Three experimental conditions were investigated (prosecution, socially unacceptable behaviour and negative consequences for the artist). A fourth condition served as a control.	†/¶/§	The results claim to have demonstrated the ineffectiveness of these anti-piracy mechanisms, and also claim that strategies based on these arguments will most likely have little impact on the intention to engage in digital piracy.
Rochelandet and Le Guel (2005)	The two main risks perceived associated with copying are studied (legal sanction and computer risk).	†/§/•	Copying behaviour is negatively correlated with the willingness to pay when a copy is available.
Maffioletti, and Ramello (2004)	This study dealt with the issue whether subjects thought they were likely to be caught downloading/burning music files.	◊/§/•	The study suggests that lawsuits can effectively lower the rate of illegal copying.
Gopal and Sanders (1997)	The effect of deterrence on digital piracy is tested .	†/¶/§	The results suggest that preventive controls do not increase publisher profits, while deterrent controls can have a positive effect on profitability.

† = the use of both exposure and control groups.

• = no random assignment to groups

‡ = measured over multiple periods of time

◊ = no control group

§ = single measurement in time

¶ = random assignment to groups

() = indicates that a method approximates one of the above

A second conclusion is that no randomized controlled experiments and quasi-experiments were undertaken, although some tried to come close.¹⁵ Although the relevance of these designs for impact evaluations is broadly acknowledged (Leeuw and Vaessen, 2009), when focusing on mechanism (evaluations) this in particular is the case (see Ludwig et al., 2011). Other studies use designs with serious limitations, such as a lack of control versus exposure groups and/or measurements of only single points in time. The value of the results is hampered by this. This also tells us that with regard to the methodology of impact studies, there is room for improvement.¹⁶

What can be said about the effectiveness of the mechanisms that were tested in these studies? In line with the Maryland Scientific Methods Scale, well-known in criminology (Sherman et al., 1998), to answer this question, we restricted our selection to impact studies that at least make comparisons between control and intervention groups and/or different times of measurement. The following conclusions can then be drawn.

Al-Rafee and Rouibah (2010), who use a methodology akin to a quasi-experiment, demonstrate that raising the awareness of the dangers of digital piracy reduces the intention to engage in this behaviour. Sinha and Mandel (2008) show that mechanisms related to fear and shame (invoked through a law suit) can be effective, though only for certain segments of the population. Though Grolleau et al. (2008) only measure at a single point in time, they do use a control group. Their finding is that prompting the copyright holders to make donations to charitable causes results in a lower level of piracy. Gopal and Sanders (1997) did not measure at multiple time points, but did create a control group and used randomization. Their results indicate that deterrent controls can have an impact on digital piracy. Moving further down the scale, the study of Levin et al. (2007) lacks a proper control group (though one of the six experimental conditions approximates one) and the measurement occurs once. This study seems to indicate that using a threat appeal is a mechanism that can be operational. Finally, Bhattacharjee et al.’s (2006) study lacks a control group, but it does measure the population over a period of time, during which four relevant events occurred (legal actions). They show that legal threats can reduce the number of files shared until a certain level (‘threat level’).

Summarizing the findings from these studies, it appears that although a couple of other mechanisms may be relevant, deterrence-related mechanisms, linked to the ‘authority’ tool distinguished by Hood and Margetts (2007), may indeed be able to influence digital piracy.

Conclusions and discussion

Although our findings are exploratory and not based on a survey or another inventory, they suggest a gap between the ‘evaluation world’ and the ‘digital world’. One indicator is that although the three public sector institutions we described are seriously involved in M&E digital policies, the fruits of this work cannot yet be found in the evaluation journals we studied. A word count of these journals further showed that the attention paid to digital policies (and developments) is very limited.

The third part of our analysis looked into studies evaluating (hypothetical) interventions and mechanisms to reduce digital piracy; it not only confirmed the gap, but also showed that with regard to the methodology of impact studies there is room for improvement. This is also the case elsewhere. Steyaert (2010: 158) noticed that with regard to e-inclusion programmes of the EC ‘it is strange to observe that most of [them] have not been evaluated beyond a project description to satisfy the funder’s information needs . . . There is an urgent need to start evaluating the myriad of initiatives and learn about what works when and for whom’. Also, in the EC-sponsored study on the social impact of ICT (Siegen Universität et al., 2010), critical remarks were made. Examples are the following: ‘[when] evaluations of impacts are conducted at all, they have often been quite limited in terms of scope and methods. It also seems that the evaluations that have been conducted have often been carried out by those directly involved and thus may lack the necessary objectivity’ (p. 253). And: ‘the evaluation methodologies used . . . have varied considerably in their robustness’ (p. 232). For the field of e-education, Gareis and Stubbe (2010: 342) mention the use of cross-sectional data where longitudinal are needed and there are misunderstandings about (quasi-)experimental designs.

Needless to say our ‘educated guess’ could have been different if we had looked into other fields such as telemedicine, e-government or e-business. However, there is no a priori evidence for such an assumption.¹⁷ Therefore, we call on evaluators to get more involved in evaluating the impact of digital policies and pay attention to the methodological quality of the studies. Here, one can benefit from experiences with impact evaluations in the (‘off-line’) world.

Digital policies are here to stay and will probably become more important over the coming years. They affect many individuals, both in positive and negative ways. As such, evaluating these policies is crucial. Given our exploration of the field, it is time for professional evaluators and those who commission evaluations to move on with increased speed.

Footnotes

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

1.

The article is partly based on the keynote of the first author given at the Prague Conference of the European Evaluation Society in October 2010.We thank three anonymous reviewers for their important and challenging comments and suggestions.

Notes

References

Al-Rafee

Rouibah

(2010) The fight against digital piracy: an experiment. Telematics and Informatics 27(3): 283–92.

Barlow

(1996) A Declaration of the Independence of Cyberspace. URL: http://w2.eff.org/Censorship/Internet_censorship_bills/barlow_0296.declaration

Bemelmans-Videc

(2002) Evaluation in the Netherlands 1990–2000: consolidation and expansion. In: Furubo

Rist

Sandahl

(eds) International Atlas of Evaluation. New Brunswick, NJ: Transaction.

Bhattacharjee

Gopal

Lertwachara

Marsden

(2006) Impact of legal threats on online music sharing activity: an analysis of music industry legal actions. Journal of Law and Economics 49(1): 91–114.

Black

Car

Pagliari

Anandan

Cresswell

Bokun

McKinstry

Procter

Majeed

Sheikh

(2011) The impact of ehealth on the quality and safety of health care: a systematic overview. PLoS Med 8(1): 1–16.

Borins

Kernaghan

Brown

Bontis

, 6 P, Thompson

(2007) Digital State at the Leading Edge. Toronto, ON: University of Toronto Press.

Chaudry

Wang

Maglione

Mojica

Roth

Morton

Shekelle

(2006) Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine 144(10): 742–52.

Chiu

Hsieh

Wang

(2008) How to encourage customers to use legal software. Journal of Business Ethics 80(3): 583–95.

D’Astous

Colbert

Monpetit

(2005) Music piracy on the web: how effective Are anti-piracy arguments? Evidence from the theory of planned behavior. Journal of Consumer Policy 28(3): 289–310.

10.

Deibert

Palfrey

Rohozinski

Zittrain

(eds) (2010) Access Controlled: The Shaping of Power, Rights, and Rule in Cyberspace. Cambridge, MA: The MIT Press.

11.

Dejean

Penard

Suire

(2011) The French ‘Three Strikes Law’ against digital piracy and the change in usages of pirates. Paper, M@rsouin, CREM, Université de Rennes, 2011. URL: http://www.marsouin.org

12.

Gareis

Stubbe

(2010) Community and family. In: Siegen

Universität

., Study on the Social Impact of ICT, Final Version, 289–374.

13.

Gopal

Sanders

(1997) Preventive and deterrent controls for software piracy. Journal of Management Information Systems 13(4): 29–47.

14.

Grolleau

Mzoughi

Sutan

(2008) Please do not pirate it, you will rob the poor! An experimental investigation on the effect of charitable donations on piracy. Journal of Socio-Economics 37: 2417–26.

15.

Heldal

(2007) Supporting participation in planning new roads by using virtual reality systems. Virtual Reality 11(2): 145–59.

16.

Hennig-Thurau

Henning

Sattler

(2007) Consumer file sharing of motion pictures. Journal of Marketing 71(4): 1–18.

17.

Hood

Margetts

(2007) The Tools of Government in the Digital Age. London: Palgrave Macmillan.

18.

Hofmann

(2010) Et in arcadio ego: from techno-utopia to cybercrime. In: Margetts

Hood

(eds) Paradoxes of Modernization: Unintended Consequences of Public Policy Reform. Oxford: Oxford University Press.

19.

Jeong

Lee

(2010) Estimating consumer preferences for online music services. Applied Economics 40(30): 1–9.

20.

Leeuw

Bogaerts

(2009) High-Tech Crime en voorlichting. Een inventarisatie van voorlichtingsinitiatieven in Nederland en een analyse van kansen en mogelijkheden. Leuven: Linc.

21.

Leeuw

(2009) Evaluation policy in the Netherlands. New Directions for Evaluation 123: 87–102.

22.

Leeuw

Vaessen

(2009) Impact Evaluation and Development. Washington, DC: Nonie/Worldbank.

23.

Levin

Dato-on

Chin

(2004) Money for nothing and hits for free: the ethics of downloading music from peer-to-peer web sites. Journal of Marketing Theory and Practice 12(1): 48–60.

24.

Levin

Dato-on

Manolis

(2007) Deterring illegal downloading: the effects of threat appeals, past behavior, subjective norms, and attributions of harm. Journal of Consumer Behavior 6(2–3): 111–22.

25.

Nergadze

(2009) Deterrent effect of four legal and extralegal factors on online copyright infringement. Journal of Computer-Mediated Communication 14(2): 307–27.

26.

Ludwig

Kling

Mullainathan

(2011) Mechanism experiments and policy evaluations. Journal of Economic Perspectives 25(3): 17–38.

27.

Maffioletti

Ramello

(2004) Should we put them in jail? Copyright infringements, penalties and consumer behaviour: insights from experimental data. Review of Economic Research on Copyright Issues 1(2): 81–95.

28.

Margetts

(2009) The internet and public policy. Policy & Internet 1(1): 1–22.

29.

McKenzie

(2009) Illegal music downloading and its impact on legitimate sales: Australian empirical evidence. Australian Economic Papers 48(4): 296–307.

30.

Moores

(2008) An analysis of the impact of economic wealth and national culture on the rise and fall of software piracy rates. Journal of Business Ethics 81(1): 39–51.

31.

Mueller

(2010) Networks And States: The Global Politics of Internet Governance. Cambridge: MIT Press.

32.

OECD (1995) IT Diffusion Policies for SMEs, OECD Digital Economy Papers, No. 9. Paris: OECD.

33.

OECD (2003) Seizing the Benefits of ICT in a Digital Economy. OECD Digital Economy Papers, No. 72. Paris: OECD.

34.

OECD (2003a) Promise and Problems of E-Democracy: Challenges of Online Citizen Engagement, Paris: OECD.

35.

OECD (2006) E-Government Studies Chapter 8, Monitoring and Evaluation OECD, 145–53.

36.

OECD (2009a) Good Governance for Digital Policies: How to Get the Most Out of ICT: The Case of Spain’s Plan Avanza. Paris: OECD.

37.

OECD (2009b) Guide to Measuring the Information Society, 2009. Paris: OECD. URL: http://www.oecd.org/dataoecd/25/52/43281062.pdf

38.

Postel

De Haan

De Jong

(2008) E-therapy for mental health problems: a systematic review. Telemedicine and E-Health 14(7): 707–14.

39.

Rochelandet

Le Guel

(2005) P2P music sharing networks: why the legal fight against copiers may be inefficient. Review of Economic Research on Copyright Issues 2(2): 69–82.

40.

Sherman

Gottfredson

MacKenzie

Eck

Reuter

Bushway

(1998) Preventing crime: what works, what doesn’t, what’s promising. Research in Brief July: 1–19.

41.

Siegen

Universität

. (eds) (2010) Study on the Social Impact of ICT, Final Version.

42.

Sinha

Mandel

(2008) Preventing digital music piracy: the carrot or the stick? Journal of Marketing 72(1): 1–15.

43.

Sommer

Brown

(2011) Reducing Systemic Cybersecurity Risk. Paris: OECD.

44.

Steyaert

(2010) Where the worlds of e-inclusion and evidence based practice meet. In: Engelen

. (eds) Proceedings of the Conference on Social Media for Social Inclusion of Youth at Risk, Leuven University, 151–161.

45.

Van der Knaap

Leeuw

Bogaerts

Nijssen

(2008) Combining Campbell standards and the realist evaluation approach: the best of two worlds? American Journal of Evaluation 29(1): 48–57.

46.

Van Dijk

JAGM

(2010) Conceptual framework. In: Siegen

Universität

. (eds) Study on the social impact of ICT, Final Version, 2–29.

47.

Van Os

(2009) Coordination of national e-government policy coordination of information exchange between administrative levels in Denmark, Austria, and the Netherlands. Paper, EGPA conference, Malta, 2–5 September.

48.

Van Reenen

(ed.) (2010) The Economic Impact of ICT. SMART N. 2007/0020.

49.

Van ’t Hof

Van Est

Daemen

(eds) (2011) Check In/Check Out. The Public Space as an Internet of Things. Rotterdam: NAi Publishers.

50.

Waldfogel

(2010) Music file sharing and sales displacement in the iTunes era. Information Economics and Policy 22(4): 306–14.

51.

Wall

Williams

(2007) Policing diversity in the digital age: maintaining order in virtual communities. Criminology and Criminal Justice 7(4): 391–415.

52.

Wingrove

Korpas

Weisz

(2010) Why were millions of people not obeying the law? Motivational influences on non-compliance with the law in the case of music piracy. Psychology, Crime & Law 17(3): 261–76.

53.

Wolfe

Higgins

Marcum

(2008) Deterrence and digital piracy: a preliminary examination of the role of viruses. Social Science Computer Review 26(3): 317–33.

54.

Yar

(2007) Teenage kicks or virtual villainy? Internet piracy, moral entrepreneurship, and the social construction of a crime problem. In: Jewkes

(ed.) Crime Online. Portland, OR: Willan Publishing.

55.

Zentner

(2006) Measuring the effect of file-sharing on music purchases. Journal of Law and Economics 49(1): 63–90.