Toward a citizen science framework for public policy evaluation: Lessons from a field experiment on Freedom of Information laws in Belgium

Abstract

This study pioneers the use of citizen science in evaluating Freedom of Information laws, with a focus on Belgium, where since its 1994 enactment, Freedom of Information’s effectiveness has remained largely unexamined. Utilizing participatory methods, it engages citizens in assessing transparency policies, significantly contributing to public policy evaluation methodology. The research identifies regional differences in Freedom of Information implementation across Belgian municipalities, highlighting that larger municipalities handle requests more effectively, while administrations generally show reluctance to respond to requests from perceived knowledgeable individuals. This phenomenon reflects a broader European caution toward well-informed requesters. By integrating citizen science, this study not only advances our understanding of Freedom of Information law effectiveness in Belgium but also advocates for a more inclusive, collaborative approach to policy evaluation. It addresses the gap in researchers’ experience with citizen science, showcasing its vast potential to enhance participatory governance and policy evaluation.

Keywords

citizen science field experiment freedom of information laws policy evaluation transparency

Introduction

Collaborative governance has long been recognized as a transformative approach in public policy and administration (Emerson et al., 2012). Departing from traditional top-down policy paradigms, it strongly emphasizes partnerships between government agencies, research institutions, non-governmental organizations, communities, and citizens (Ansell and Gash, 2008). It recognizes the intricate nature of contemporary policy challenges, promoting shared decision-making, collaborative problem-solving, and mutual accountability (Siddiki et al., 2017).

Collaborative approaches to public policy extend throughout the entire policy cycle, from agenda setting to evaluation (Ansell and Gash, 2008). A notable advantage lies in their potential to enhance policy and program evaluations, as opposed to the conventional approaches (Bryson et al., 2006). Traditional evaluations, typically carried out by external experts or governmental bodies, often adopt a top-down approach that may impose disciplinary and power dynamics on local communities and citizens, stemming from a lack of meaningful participation from those impacted (Esposito et al., 2024). In contrast, collaborative governance adopts a more inclusive evaluation process (Taylor and de Loë, 2012). This inclusivity can result in more comprehensive and contextually relevant evaluations. This approach underscores the importance of co-creating knowledge and facilitates mutual learning among stakeholders. The concept of co-creating knowledge enables involved parties to negotiate shared interpretations of reality, leading to evaluations that, while subjective, represent a collective subjectivity that benefits all stakeholders (Tassie et al., 1996). In complex policy landscapes, this approach leverages the expertise of diverse stakeholders, shedding light on intricate issues (Gerlak and Heikkila, 2011). Furthermore, it has the potential to yield comprehensive and contextually appropriate approaches that acknowledge the impact of policies and programs on various stakeholder groups, fostering shared comprehension of public policy problems and potential solutions (Beierle and Cayford, 2002; Maggioni et al., 2012).

In parallel, there are established participatory methods that facilitate collaborative approaches to policy evaluation. They prioritize stakeholder engagement, including citizens, affected communities, and experts, within a structured and inclusive framework. For example, citizen panels involve the random selection of a representative group of citizens who convene regularly to discuss and evaluate policies (Renn et al., 1993). They are often instrumental in gathering input on specific policy issues and gauging public opinion. Focus groups assemble a small, diverse set of participants to engage in guided discussions about policy-related topics (Kahan, 2001), enabling exploration of in-depth perspectives and experiences. Deliberative polling combines surveys with small-group discussions (Hoekman and Rojas-Romagosa, 2022), providing insights into shifts in public opinion and understanding. Community-based participatory research serves as another collaborative approach in which community members and researchers collaborate to design and conduct research, including policy evaluation, addressing community needs and concerns (Garcia et al., 2013).

Among these collaborative endeavors, one promising methodology that has gained widespread support is citizen science (Council of the EU, 2021; Sanz et al., 2020 [2014]). Citizen science involves the active involvement of the public, often called citizen scientists, in scientific research, spanning from data collection and generation to data analysis and interpretation (Bonney et al., 2009). It is traditionally linked with research in environmental monitoring, biodiversity studies, sustainable agriculture, and technological innovation (e.g. Ebitu et al., 2021; Eitzel et al., 2017). However, this methodology has begun expanding its scope to encompass psychology, humanities, and social science research (Tauginienė et al., 2020; Vohland et al., 2021). Citizen science offers several advantages for policy evaluation. It fosters, for example, a sense of ownership and participation among citizens, empowering them to contribute to the assessment of policies that affect them (Wiggins and Crowston, 2011). It also leverages the collective intelligence and distributed efforts of diverse groups, enabling the collection of large datasets and enhancing the robustness of evaluations (Bonney et al., 2009). However, outside the biology and environmental studies field, the experience of researchers with citizen science remains limited, both on an individual level—with the majority having never or only occasionally engaged with it—and collectively, as we are just starting to uncover its vast potential (Li et al., 2022). Therefore, it is crucial to further develop the support structures needed to effectively organize future citizen science studies (Ebitu et al., 2021) as well as to experiment with citizen science in policy evaluation across various domains. Identifying best practices for its seamless integration into the policy cycle is essential (Hager et al., 2021; Hecker et al., 2019; Wehn et al., 2021).

In response to the growing need for exploratory research in policy evaluation, this study investigates the potential of citizen science for assessing the effectiveness of transparency policies, with a specific focus on Freedom of Information (FOI) laws that empower citizens with the right to access information held by the government. Our focus on FOI law implementation for citizen science policy evaluation serves both scientific and pedagogical purposes. As explained in section “Field-experiment evaluations of FOI laws in the existing literature,” there is a rising trend in scientific research on FOI laws, emphasizing field experiments that involve real-world assessments, including FOI request submissions and responses. Citizen scientists can support professional researchers and be involved in this approach by actively collecting data related to FOI implementation (e.g. access to government records). Moreover, citizen science empowers ordinary citizens to become active co-researchers, bridging the gap between scientific inquiry and practical application. Participants gain a deeper understanding of how FOI laws operate and how to effectively utilize them. This enhances citizens’ knowledge and skills, enabling them to navigate and employ these laws more effectively.

While citizen science holds promising potential as a valuable approach for conducting field experiments to test the effectiveness of FOI law implementation, its application in such experiments remains unexplored. As detailed in section “Field-experiment evaluations of FOI laws in the existing literature,” existing research on FOI law, conducted in various countries worldwide including the United States, Brazil, England, the Netherlands, Slovakia, Mexico, and Italy, has primarily utilized field experiments carried out exclusively by professional scientists, without involving citizens. Against this backdrop, this article contributes in two significant ways. First, it pioneers the integration of citizen science in conducting a field experiment to assess the effectiveness of FOI law implementation. Second, it delves into the Belgian FOI case study. Despite Belgium’s adoption of FOI legislation in 1994, there has yet to be a comprehensive evaluation of its implementation effectiveness, whether through field experiments or other methodologies. Thus, this article, on one hand, offers a novel and unprecedented methodological contribution by promoting collaborative approaches to policy evaluation, especially in the domain of FOI law implementation. On the other hand, it aims to enrich our understanding of FOI implementation in Belgium.

The article is structured as follows. Section “Field-experiment evaluations of FOI laws in the existing literature” provides a comprehensive review of the literature about field experiments in FOI evaluation. Section “Research design: Involving citizen scientists in a field-experiment evaluation of FOI implementation in Belgium” delves into the detailed design of the proposed methodology to employ citizen science in the conduct of a field experiment to evaluate FOI law implementation in Belgium. In section “Findings,” we present the results. In section “Discussion and conclusion,” we discuss the contributions of our study and offer concluding remarks.

Field-experiment evaluations of FOI laws in the existing literature

Field experiments on FOI are on the rise but remain relatively nascent. They commonly entail submitting requests to public administrations, observing their responses, and occasionally modifying content to gauge their impact.

Cuillier (2010) conducted two field experiments in Arizona with the assistance of student journalists from the University of Arizona. In the first experiment, they requested use-of-force reports from police agencies, employing either stern (“vinegar”) or friendly (“honey”) letters. The results revealed that stern letters generated higher response rates and potentially quicker responses, while friendly letters encouraged more cooperative behavior from the agencies. In the second experiment, public school districts were targeted, employing three types of letters: stern, friendly, and neutral. Stern letters again led to better response rates and quicker replies, while friendly letters prompted agencies to go beyond legal requirements.

Michener and Rodrigues (2015) conducted two field experiments in Brazil. In the first experiment, they submitted information requests to eight Brazilian jurisdictions using various identities. Discrimination in favor of institutional identities was observed among females, with the female PhD student identity receiving higher response rate compared to the non-institutional identity. Among males, both institutional and non-institutional identities received similar response rates. In the second experiment, requests were sent to 29 Brazilian Ministérios Públicos by two male citizens, one with a non-institutional identity and the other with an institutional identity related to public officials’ salaries and hiring policies. Surprisingly, only 50 percent of the requests received responses, with no significant variation based on the requesters’ profiles, indicating similar response rates regardless of identity. Michener et al. (2020) also investigated whether large Brazilian municipalities perform identity-questing and discriminate toward institutional requesters. The results confirm both hypotheses, as non-institutional requesters are less likely to receive a compliant response compared to their institutional counterparts.

Worthy et al. (2017) observed low response rates (15%) among English parish councils. However, requests explicitly mentioning FOI norms received more replies. Interestingly, neither preexisting transparency levels nor council size significantly impacted responses. This experiment was replicated in the Netherlands (Grimmelikhuijsen et al., 2018), resulting in a substantially higher response rate (approximately 77%). Like in England, this article finds that public administrators are more inclined to respond to requests that explicitly mention FOI.

In their study on Slovakian local governments, Spáč et al. (2018) employed three types of requests to gather information about local elections. The first request was purely for research purposes, the second included a “moral appeal” emphasizing the value of cooperation, and the third explicitly cited the FOI law. Response rates were approximately 30 percent, with higher rates in large municipalities. Requests framed within the context of FOI yielded higher responses, particularly in smaller municipalities, while moral appeals had no discernible impact.

In the Mexican study by Lagunes and Pocasangre (2019), FOI requests consisting of 13 questions were sent to 197 national government entities. Response rates ranged from 71 to 81 percent, with no discrimination between responses to regular citizens and influential individuals. However, responses improved over time, although the speed and quality of responses decelerated.

Wagner (2021) conducted a study involving 1002 FOI requests sent across 9 US states and 334 jurisdictions, seeking various types of information. Positive outcomes correlated with the incidence of White population, Republican votes, and state legislature representation. Requests in central and Southern US regions exhibited lower success rates and longer processing times.

Cicatiello et al. (2022b, 2024) investigated the implementation of FOI regulations in Italian municipalities which a prior study identified as being characterized by a significant level of heterogeneity regarding the publicity granted to FOI regulations (Cicatiello et al., 2022a). They submitted requests for identity card issuance data signed either by common citizens or a lawyer, in some cases mentioning FOI and in others not. Treatment assignment was stratified by macro areas (North, Center, South) using stratified randomization. Response rates varied across macro areas and were higher among larger cities. There were notable differences in responses between Northern and Southern municipalities to lawyer-presenting requesters. In the North, common citizens received fewer complete responses, whereas in the South, lawyers mentioning the FOI law often faced silence. The study also revealed that bureaucrats in corruption-prone areas reacted negatively to lawyer requests, and in regions with low regulatory quality, they responded unfavorably to FOI-aware requesters.

In summary, the existing field experiments have provided valuable insights into FOI law implementation across various countries and regions. However, none of these studies have specifically focused on Belgium or utilized citizen science methodologies. Therefore, the following sections will present how we used citizen science methods to conduct a field-experiment evaluation of FOI law implementation in Belgium.

Research design: Involving citizen scientists in a field-experiment evaluation of FOI implementation in Belgium

Empirical setting: FOI laws in the institutional context of Belgium

Belgium operates under a complex federal system. At the national level, there is the Federal government. Below it, there are three language-based communities (the Dutch-speaking Flemish Community, the French-speaking Walloon Community, and the German-speaking Community) and three regions (Flanders, Wallonia, and the Brussels-Capital Region). In addition, there are 10 provinces and 581 municipalities spread across these regions.

In 1994, Belgium implemented an FOI law that applied to the federal administration. As in other countries, it took some years for Belgian legislators to find an agreement on the final text of the law, and some additional years to refine the authorities covered by the law, the nature of information concerned, and the regime of exemptions. It is important to note that FOI is constantly evolving, depending on political willingness to extend or restrict the scope of the law, pressure from advocacy organizations, jurisprudence, and adoption of provisions related to FOI in international law. This national law partly influenced the decrees adopted in the Belgian regions.

Since Belgium is a federal state, there is autonomy for the regions to draft their own laws according to the division of power between the federal state and the federated entities. The first jurisdiction to pass an FOI law was Flanders in 1991 (Keunen and Van Garsse, 2019). This process is not uncommon in federal states, due to regional autonomy. For example, in Switzerland, the Canton of Bern adopted an FOI law in 1995, 11 years before the national law entered into force; in Germany, four Länder already had an FOI law before the national Parliament passed an FOI legislation in 2005.

Each region adopted their own legislation, based on its own decrees governing municipalities. FOI allows individuals to submit written requests (including via email) for documents held by administrative authorities. Requesters generally do not need to demonstrate a specific interest, except for personal documents. Authorities must respond to FOI requests within 30 days, which can be extended to 45 days if necessary, and refusals must be substantiated.

However, regional variations exist. In the Brussels-Capital Region, requesters must attach a scanned copy or photograph/photocopy of their ID to their demand. Wallonia has a 30-day response time, extendable by up to 15 days, with reasons for rejection or postponement communicated. The Brussels-Capital Region’s response times range from 20 to 40 days based on information volume and complexity. In Flanders, the processing time is 20 calendar days, extendable to 40 days with justification. In Flanders, requests that are overly general may need revision and completion before being addressed by authorities.

Recruitment of citizen scientists and setting up of the research team

The research project presented in this article was initiated and led by the four academic researchers who author this article. They worked closely with a third-sector organization and a group of 36 citizen scientists.

Scholars (Göbel et al., 2021) highlight the involvement of various third-sector organizations in citizen science, particularly those addressing political or social issues. Anticor, the organization that collaborated in our citizen science experiment focused on FOI evaluation in Belgium, falls within this category. Operating from Brussels, Anticor’s mission centers on combating corruption and promoting transparency to reinforce citizens’ oversight of politicians and public managers. The organization’s commitment was secured during the study’s preparatory phase, and it willingly cooperated with the researchers and citizen scientists.

The literature suggests that third-sector organizations can play multiple roles in citizen science, including technical roles in scientific knowledge production, governance roles in organizing research activities, and advocacy roles in disseminating knowledge beyond research outputs (Göbel et al., 2021). Anticor possesses not only technical expertise in Belgian FOI legislation but also considerable experience in using FOI to gather information. This expertise played a pivotal role in shaping the study’s design and in training the citizen scientists involved.

Citizen scientists were recruited from a pool of 45 graduate students enrolled in the 2022/2023 cohort of the “Innovations in Policy Evaluation” course within the Master of Science in “Public Administration” at the Université Libre de Bruxelles (Belgium). The course spanned 24 hours and provided participants with knowledge encompassing the policy cycle and its key phases, the role of evaluation in policymaking, traditional techniques to assess public policy (i.e. social experiments, difference-in-differences, before-and-after methods), innovative (i.e. field experiments) and collaborative approaches (i.e. participatory evaluation and citizen science) to policy evaluation, and basic knowledge of collection and analysis of qualitative (i.e. thematic coding of texts collected through participant observation and interviews) and quantitative data (i.e. descriptive statistics). Building on this foundational knowledge, students delved into practical cases of government transparency policies, with a primary focus on FOI in various countries during the course. They were offered the opportunity to become citizen scientists in a research project evaluating FOI implementation in Belgium. The process emphasized the voluntary nature of participation and the fact that involvement would not impact the course grades. A total of 36 students willingly chose to engage in the project, formalizing their commitment through signing a non-disclosure agreement to ensure the confidentiality of all information used during the research.

The evaluation questions

Our citizen science evaluation aims to assess Belgian municipalities’ compliance with FOI laws. This involves two evaluation questions: (EQ1) Do Belgian municipalities respond to FOI requests and provide the requested information? (EQ2) Do they discriminate among requesters? Answering to these questions implies submitting FOI requests, quantifying and analyzing responses, assessing response quality, and exploring discriminatory practices in FOI request processing.

In line with Senabre Hidalgo et al. (2021), we involve citizen scientists as co-researchers, recognizing their valuable insights and firsthand experiences related to transparency issues and FOI implementation at the municipal level in Belgium. These individuals have observed and interacted with municipal administrations in their daily lives and studies, offering unique perspectives often overlooked by professional researchers. Their diverse backgrounds enrich the research, providing broader insights and more comprehensive results.

Our citizen scientists are actively involved in data collection, generation, analysis, and interpretation. They also contribute participant observations to describe and reflect on their experiences, offering insights into organizing workflow in citizen science projects.

Data collection and analysis

Building upon citizen science and the scholarly contributions summarized in section “Field-experiment evaluations of FOI laws in the existing literature,” our research team orchestrated a field experiment aimed at addressing the evaluation questions delineated in section “The evaluation questions.” The experiment is based on sending access to information requests to all Belgian municipalities. The requests pertain to the same subject matter but are formulated slightly differently, so as to be attributable to individuals with different profiles (a professor, an ordinary citizen, an advocacy organization active in the field of transparency) and demonstrate varying levels of familiarity with FOI legislation (mentioning it or not). These requests and municipalities were randomly paired.

To set up and carry out the experiment, the research team undertook the following tasks: (1) selecting the relevant document for FOI requests from municipalities, (2) creating and sending standardized email templates for the submission of these document requests to the municipalities, (3) meticulously coding and analyzing the interactions that occurred between the requesters (citizen scientists) and the municipalities in response to requests, and (4) organizing the workflow of the citizen scientists to ensure the efficient and effective execution of these tasks.

Document selection for FOI requests

The team opted for exploiting FOI to request the explanatory note for the last municipal council meeting, along with its annexes. The choice of this document followed several discussions with Anticor.

This note, within the context of municipal council meetings, offers essential background information, context, and explanations about the topics, agenda items, or decisions set for discussion or decision-making during the forthcoming meeting. Its primary purpose is to inform both council members and the public about the issues at hand, aiding their understanding of the agenda items’ purpose and significance. Usually, this note is accompanied by annexes containing supplementary information and details related to the explanatory note. Their content varies depending on the meeting’s subject and local government practices. Common items found in annexes include comprehensive reports, studies, or analyses relevant to the agenda items; copies of pertinent laws, regulations, or legal opinions; financial information, such as budgets, financial statements, or cost estimates associated with agenda items; and copies of contracts, agreements, or proposals integral to the agenda items.

Anticor highlighted the document’s importance for citizens in enhancing democratic processes. By making it available, it empowers citizens’ participation in governance and holds the council accountable, enabling public oversight. In addition, it serves as a tool for civic education and inclusivity by ensuring equal access to information.

Email text drafting and sending

The researchers, Anticor, and citizen scientists drafted an email during a class session to submit an official request to Belgian municipalities for the notes and its annexes. To examine whether public administrations discriminate in request processing based on the requester’s profile and FOI mention, two elements of the text—the signature and a sentence quoting the regional FOI law as the legal basis—were manipulated. This resulted in five alternative text versions:¹

(a) Request by a non-Googleable² citizen not mentioning FOI law (“common citizen request”).

(b) Request by a non-Googleable citizen mentioning FOI law (“common citizen + FOI request”).

(d) Request by a University Professor (signed as Professor) mentioning FOI law (“Professor + FOI request”).

(e) Request by a non-governmental organization (“Advocacy organization request”).

The non-Googleable citizen signing requests (a) and (b) was a Belgian citizen who collaborated but was not part of the citizen scientists’ team. The University Professor signing requests (c) and (d) is one of the study’s co-authors. Requests (a), (b), (c), and (d) administered by the citizen scientists were divided into four groups, whereas Anticor administered the (e) request.

Each of the 581 Belgian municipalities was assigned to one treatment, a process conducted through randomization stratified by region and municipalities’ population size (<12k, between 12k and 50k, and >50k inhabitants). Table 1 provides a summary of the composition of treatment groups, whereas the balance tests of the groups are provided in the Supplemental Appendix. The list of municipalities assigned to each treatment is available upon request.

Table 1.

Number of municipalities assigned to each treatment by region.

Treatment	Brussels-capital region	Flanders	Wallonia	Total
(a) Common citizen	4	60	53	117
(b) Common citizen + FOI	4	61	52	117
(c) Professor	4	60	53	117
(d) Professor + FOI	4	60	53	117
(e) Advocacy organization	3	59	51	113
Total	19	300	262	581

The procedure for sending requests was standardized as follows. Citizen scientists logged into the mailbox designated for their group, opened a new email, and copied and pasted the subject and text specific to their group, ensuring language accuracy. They attached the requester’s ID card and copied the municipality’s address from the list into the “To” field before sending the email. Requests were sent individually. With roughly 117 municipalities per group, the task was divided among group members. To maintain consistency and uniformity, it was crucial that all requests were sent promptly. This process was carried out from 13 to 15 March 2023. The data collection phase was stopped on 10 May, beyond the response time set by the laws.

Coding, analyzing, and interpreting interactions between municipalities and citizen scientists

Once the four groups of citizen scientists received responses, they had to translate the content into variables through a coding process, converting qualitative information into numerical data. Within the initial days of the experiment, the entire research team collectively agreed upon the coding scheme as follows:

No reply: The municipality failed to provide a reply which was not an automatic message.

Denial: The municipality replied but did not disclose data, or by plainly denying the request or by inquiring additional information not in line with FOI provisions.

Unsatisfactory reply: The municipality provided some kind of information, but not the data requested (e.g. they disclosed an outline of the meeting instead of the explanatory note).

Partial reply: The municipality provided the explanatory note but not the annexes.

Complete reply: The municipality provided both the explanatory note and the annexes.

This scheme was incorporated into a comprehensive codebook that served as a guiding framework for the citizen scientists, facilitating their navigation of the response process (when required). Anticor used the same codebook, ensuring consistent and systematic coding. Furthermore, the research team set weekly meetings, facilitating continuous communication between the researchers and the citizen scientists’ spokespersons. These meetings helped provide regular updates on the experiment’s progress.

For measuring compliance, the analysis focused on measuring the overall response rate and incidence of disclosure (i.e. when municipalities provided at least partial information). Conversely, to address discrimination toward different requesters, the analysis compared the response rate and the incidence of disclosure for requests (a) and (b) (common citizen) with those for requests (c) and (d) (University professor). Then, the analysis compared the response rate and the incidence between requests (a) (Common citizen) and requests (b) (Common citizen mentioning FOI), and between requests (c) (Professor) and requests (d) (Professor mentioning FOI). This offered valuable insights into whether referencing FOI (and who is referencing FOI) in the request influences outcomes. Finally, to assess the extent to which submitting the request through a pressure group influences the outcome, requests (a) (Common citizen) were compared with requests (e), signed by Anticor.

In line with our citizen science approach, we relied on the contextual knowledge of our Belgian citizen scientists to interpret the results. Consequently, we requested them to present their findings in a report. In the reports, citizen scientists identified significant geographical variations in response rates among municipalities. These variations were observed across municipalities of different sizes and administrative regions. They explained these differences based on two main factors: first, the availability of technological, organizational, and financial resources, which they found to be inadequate in some cases; and second, insufficient legal and administrative provisions. Furthermore, we also asked the citizen scientists to dedicate a section of their report to describe how they interpreted and implemented our guidelines in terms of the workflow organization of the research project. In addition, we encouraged them to share reflections on their positive and negative experiences as citizen scientists evaluating FOI implementation. Table 2 summarizes our thematic analysis of the report passages where citizen scientists provide their interpretations of geographical variations in municipalities’ response rates to citizen requests; reflections on their experiences of involvement in a citizen science project to evaluate FOI implementation; and organization of workload within groups.

Table 2.

Thematic analysis of citizens’ scientists reports (n = number of coded passages).

Higher level themes	Lower level topics	Sample quotes
Interpretation of geographical variations in municipalities response rates to citizen request	Availability of technological, organizational, and financial resources (n = 10)	“The Walloon region offers the possibility for municipalities to publish the explanatory notes and the annexes through the platform Deliberation.be. However, there are geographical differences: municipalities using the platforms are mainly big urban areas which corroborates our interpretation”
		“Compared to Flemish municipalities, Walloon ones provided more answers with technical problems or administrative red tape including problems opening WeTransfer’s and broken links”
		“The municipality cannot send us the annexes because the selection and examination of each of the documents annexed to the deliberations is very demanding logistically, particularly in terms of time and verification with regard to the GDPR”
		“The request is complex and consists of 25 points + 11 additional points. There is not a single document with the full report, we only have documents per point separately. The decisions alone therefore consist of 36 documents. According to the tariff rules for administrative services, this is equivalent to 36 A4 scans x 3.48 euros = 121.8 euros. Then there are all the annexes, that is to say at least 36 other documents, possibly many more”
	Inadequate or insufficient legal and administrative provisions (n = 12)	“Based on articles L3231-1 and -2 of the CDLD, some municipalities say that our request is too broad and that they cannot respond to it”
		“Given that our municipality is part of the German-speaking Community, our municipality is no longer subject to the Code of local democracy and decentralization, but to the municipal decree of the German-speaking Community of 23 April 2018”
Reflection on experiencing involvement in citizen science project to evaluate FOI implementation	Positive civic experience enabling to learn FOI and contribute to policy evaluation (n = 17)	“I decided to take care of encoding the responses because I was curious to see whether the municipalities would respond positively”
		“The citizen-science approach had a major added-value as it allowed us to learn how research works concretely, though it involves biases and is greatly challenging in terms of internal organization”
	Negative experience due to longtime commitment and burdening workload (n = 4)	“Some of us have a busier schedule then others during the semester with the thesis work and other pedagogical activities. All these activities are not easy to organize daily”
		“It’s worth noting that this was not an easy task to accomplish, requiring concentration, accuracy and attention to ensure that not a single step was missed”
Workload organization within groups	Model 1—Centralized collaboration (n = 26)	“Recognizing the importance of regular communication, the group established a weekly meeting schedule, holding sessions every Thursday. These meetings served multiple purposes, including the review of tasks, debriefing the coordination meetings, and discussing any points of clarification or debate. The general manager took the lead in providing updates during these meetings, ensuring everyone was informed about the progress and objectives”
		“To streamline the workflow and maximize efficiency, the group implemented a system of middle management roles for sub-tasks. However, while each member had their designated responsibilities, the intention was not to restrict individuals to specific tasks but rather encourage collaboration and shared involvement in all aspects of the project”
		“Leadership played a crucial role in the group’s organization and operational efficiency. The group embraced the concept of shared leadership, distributing mid-level management roles to ensure practicality and effective coordination”
		“The general manager provided all group members with a comprehensive view of the ongoing processes and offer guidance or input whenever necessary. By staying informed and engaged, the general manager facilitated effective coordination and alignment in the group”
	Model 2—Decentralized collaboration (n = 4)	“Organizing weekly meetings is a real headache since almost none in the group has the same schedule during the week”
	Model 2—Decentralized collaboration (n = 4)	“We decided to operate with a shared document where each of us clearly states the issues encountered during the work. Even if this is not the best way to work because the problems are not always clearly explained by individual inputs, it allows us to keep collective track of problems and possible solutions”
	From Model 1 to Model 2 (n = 21)	“During the initial phase of the work, we all met face-to-face to review the information we needed to know to implement this citizen project”
	From Model 1 to Model 2 (n = 21)	“We organized initial small-group meetings before realizing that, given our different schedules, it would be easier to do this work mainly remotely”
	From Model 2 to Model 1 (n = 16)	“it is important to point out that the organizational style within the group did not remain the same during the entire process of carrying out the work. In fact, during the initial phase of the project, information exchange among teammates and task management were ensured through a group on the ‘messenger’ platform”
	From Model 2 to Model 1 (n = 16)	“Then, during the coding phase of our work, around 10 April, we decided to change the organizational structure within the group in order to facilitate collaboration and progress on the project. To do this, we decided together to set up a weekly meeting on Teams during which we could more easily discuss the progress of our individual work and ask the group questions about parts of the work that we didn’t understand.”

Organizing the workflow of citizen scientists

The organization of the citizen scientists into four groups, each corresponding to one of the treatment categories (a to d), was a pivotal aspect of our research design. Citizen scientists were randomly assigned to groups and were asked to assign internally the following roles to ensure the smooth functioning of the group:

Email Box and Coding Management: Responsible for handling emails directed to municipalities and coding the responses according to the prescribed codebook.

Database Management: Tasked with maintaining the integrity of the database, conducting regular accuracy checks, and ultimately delivering the finalized database.

Data Analysis Management: Responsible for analyzing the collected data and providing descriptive statistics.

Report Communication Management: Responsible for interpreting and communicating the analyzed data through a report detailing descriptive statistics, subsequently presented orally to Anticor representatives and the four professional researchers.

General Management: Responsible for overseeing and coordinating the overall group operations, ensuring alignment among the aforementioned tasks.

At the end of this process, citizen scientists chose to form groups of 8 to 10 members (Table 3). Each group operated under the careful supervision of a researcher, ensuring strict adherence to scientific standards. Citizen scientists with a management role acted as a spokesperson and maintained regular communication with Anticor and the researchers, who constituted the coordination committee. This weekly communication provided guidance to the citizen scientists and guaranteed scientific rigor throughout the experiment (Figure 1).

Table 3.

Groups of citizen scientists.

	Groups	Citizen scientists
(a) Common citizen	Group 1 (G1)	10
(b) Common citizen + FOI	Group 3 (G3)	9
(c) Professor	Group 4 (G4)	9
(d) Professor + FOI	Group 2 (G2)	8

Figure 1.

Governance of the citizen science project and field experiment treatments.

Within their respective groups, citizen scientists were given the autonomy to self-organize and were asked to conduct a thorough analysis of their group dynamics. This analysis served to meticulously document the organizational structures employed, offering empirically grounded insights into their practices of organizing the workflow. These insights were invaluable for enhancing the understanding of effective patterns in managing citizen science projects.

The establishment of this governance and organizational structures was indispensable because of the diversity among citizen scientists (in terms of background, motivation, skills, etc.). This heterogeneity can significantly complicate the orchestration of field experiments, especially when several groups are concurrently maneuvering through different phases of data collection. The implementation of these structures played a crucial role in minimizing the potential introduction of bias into the results.

However, we noticed significant differences in how various groups of citizen scientists responded to emails. These disparities were considered during the group assessment, when professional researchers assigned grades³ based on citizen scientists’ quality of coding and commitment to the assigned tasks (Table 4). Therefore, we explored whether the varying levels of commitment influenced the experiment’s outcomes, and whether high levels of commitment, reflected in high grades, resulted in more favorable and comprehensive responses from the municipalities.

Table 4.

Groups’ grades and qualitative descriptions of grades.

Group	Grade	Qualitative evaluation description
G1	14	Very good engagement, competent in data collection and coding with a good grasp of theoretical concepts. However, more detailed explanations are needed in methodology, and clearer articulation in analysis is required.
G2	11	Met the basic standards in coding correctness and showed good engagement with feedback, but the report and data analysis suffered from poor academic practices and a superficial interpretation of results through theory.
G3	16	Showed continuous improvement and effective collaboration over time with a solid engagement in coding, though the report lacked detailed context and precision in references.
G4	19	Demonstrated impeccable teamwork and engagement in data collection and coding, with an effective analysis and interpretation of data, despite slight inconsistencies in report structure and theoretical application to data interpretation.

The following section provides the results of the experiment. We start with essential statistics and then delve into the findings, investigating the link between commitment levels among citizen scientists and the municipalities’ response types. In addition, we illuminate the organizational structures adopted by the citizen scientists to streamline individual contributions, fostering team alignment, and mitigating issues related to insufficient individual commitment.

Findings

Results of the citizen science evaluation of FOI law implementation in Belgium

The responsiveness of Belgian municipalities to FOI requests of citizen scientists

Figure 2(a) shows that approximately 36 percent of municipalities, regardless of their treatment assignment, did not respond. This suggests a significant portion of municipalities are not compliant with FOI laws or have responsiveness issues. The figure also shows that around 30 percent of municipalities did offer an explanatory note, indicating that a substantial proportion of municipalities acknowledge requests but do not always provide complete information.

Figure 2.

Results of the citizen science evaluation of FOI law implementation.

Figure 2(b) shows that just under half of small municipalities did not respond at all, whereas medium-sized municipalities exhibited a 28 percent no-response rate, and large municipalities only 17 percent. Over 50 percent of large municipalities provided at least partial disclosure, while medium-sized municipalities had a 36 percent disclosure rate, and small municipalities disclosed information in 21 percent of cases. Therefore, municipality size significantly impacts response rates, with larger municipalities being more responsive and likely to provide at least partial disclosure compared to smaller municipalities.

Figure 2(c) displays regional disparities, with Brussels’ municipalities leading at a compliance rate of approximately 58 percent, followed by Flemish municipalities at 46 percent, and Walloon municipalities at 21 percent. These differences suggest variations in the interpretation and adherence to FOI laws, or in the municipalities’ willingness to respond to the requests.

Figure 2(d) shows that treatment (c) Professor achieved the highest response rate at 75 percent, but a notable portion of these responses (28%) were considered unsatisfactory. Treatments (d) Professor + FOI and (e) Anticor struggled to elicit responses from municipalities, with no-response rates of 49 and 39 percent, respectively, although they did result in some form of disclosure (21% and 15%, respectively). Treatment (b) Common citizen + FOI achieved the highest rate of at least partial compliance at 42 percent, followed by treatment (c) Professor at 38 percent, and treatment (a) Common citizen at 27 percent.

According to the citizen scientists (Table 2), geographical variations in FOI implementation across Belgian regions and municipalities result from a combination of administrative, cultural, and technological factors. Different administrative cultures, characterized by local practices and norms, affected the interpretation and application of FOI, with different political and legislative frameworks across the regions further impacting its implementation. This variation was compounded by disparities in resource allocation and technological infrastructure. Notably, municipality size emerged as a critical factor: large municipalities, benefiting from more substantial resources and advanced infrastructure, generally exhibited a higher rate of responsiveness and transparency compared to small municipalities.

The link between municipalities’ responsiveness and citizen scientists’ commitment

Given the substantial variability in outcomes across treatments, we conducted three regression analyses using the final grade as the independent variable. While all participants mentioned the positive civic experience of engaging in a citizen science project and learning about FOI functioning, some also expressed concerns about time commitments and heavy workloads. These varied levels of commitment among participants had different effects on the outcomes of the FOI requests sent to municipalities.

The results are reported in Figure 2(e). Column (1) demonstrates that a higher grade is associated with a higher outcome value. Since outcome is an ordered variable, this result suggests that a higher level of commitment correlates with more favorable responses from the municipalities. Column (2) reinforces this finding, indicating a positive association between grade and disclosure. Column (3) suggests that this outcome likely occurred because higher commitment levels led to increased interactions between the groups and the municipalities.

Organizational patterns within the groups of citizen scientists

The analysis of reports revealed that citizen scientists employed two organizational models to structure work within their groups and streamline collaboration among members (Figure 3). While some groups consistently adhered to one of these models, others alternated between them, exhibiting hybrid and varying organizational patterns (Figure 4).

Figure 3.

Models to organize collaboration among citizen scientists.

Figure 4.

Organizational structures of citizen scientists’ groups.

Organizational models

Model 1 features a centralized structure with a clear central authority responsible for coordination and decision-making, typically a strong leader overseeing General Management. This leader ensures team alignment with project goals, with members assigned specific roles and responsibilities. Essential to this model are regular weekly meetings that ensure structured communication. Decision-making balances leader oversight and collective problem-solving, enabling effective navigation of complex project challenges. The leader promotes participation and idea exchange, and harnesses collective intelligence, maintaining project coherence. Model 1’s hallmark is its centralized coordination and decision-making, coupled with an emphasis on collaborative input.

Model 2 adopts a horizontal approach, emphasizing less supervised, more autonomous work. Leadership focuses on coordinating member inputs and relaying instructions from a higher authority, like a coordination committee, rather than overseeing tasks directly. Team members work independently, with leadership compiling and communicating these contributions. Communication is less structured, posing challenges in sustaining effective dialogue. A key limitation is the lack of strong decision-making routines, with the group relying on the coordination committee for direction. Workflow involves members contributing independently to a collective document or platform, with minimal live collaboration. The leader’s role centers on integrating these contributions according to the coordination committee’s guidelines.

Use of organizational models by citizen scientists’ groups

In examining within-group arrangements throughout the experiment, different patterns of strategic adoption of the two above-mentioned organizational models emerged. Groups identified as G1, G2, G3, and G4 each adopted the two organizational models differently, which significantly influenced their approaches to task management, problem-solving, and overall project execution.

G4 demonstrated a steadfast application of Model 1, maintaining a consistent, centralized leadership approach throughout the experiment. This model fostered a strong commitment among team members, evidenced by regular weekly meetings aimed at debriefing task implementation and identifying areas for improvement. Such structured communication and clear role definitions under a centralized leadership facilitated an environment conducive to effective collaboration and project coherence.

Conversely, G2 adhered to Model 2 (decentralized structure) because of logistical challenges, as divergent personal schedules hindered the possibility of regular in-person meetings. This approach not only underscored the group’s autonomous work ethic but also highlighted the challenges of maintaining effective communication without a strong central leadership.

G1 initially embraced the centralized structure of Model 1 but transitioned to the decentralized approach of Model 2 as aligning members’ schedules proved difficult. This shift to a more flexible organizational model allowed for continued individual contributions to the project despite reduced opportunities for direct collaboration.

G3 began with the decentralized Model 2 but opted for a strategic shift toward the centralized Model 1. This transition was driven by a desire to enhance task efficiency and improve the workflow, leveraging the benefits of structured leadership and regular communication to overcome initial challenges and streamline project execution.

We therefore found four different structures (Figure 4), encompassing a centralized leadership structure (G4), an adaptive hybrid structure (G3), a flexible hybrid structure (G1), and an independent decentralized structure (G2). These structures provide a comprehensive framework for understanding the diverse strategies used. Each structure reflects a unique approach to balancing autonomy with leadership, flexibility with hierarchy, and individual contributions with collective goals.

Group performance, as indicated by grades (Table 4), shows an interesting association with organizational structures. G4, with its centralized leadership, attained a high grade of 19, illustrating the benefits of structured leadership. G3, employing an adaptive hybrid model, achieved a score of 16, highlighting how a mix of leadership and flexibility can enhance collaboration. G1, with a flexible hybrid model, scored 14, indicating that while adaptability supports engagement, it might compromise focus. G2, adopting a decentralized approach, received the lowest grade of 11, pointing out the challenges in maintaining academic standards and effective communication without centralized coordination. These findings suggest that centralized and adaptive structures generally lead to better outcomes, whereas flexible or decentralized models necessitate careful handling of collaboration and oversight. This nuanced association between organizational structures and group performance underscores different models’ impacts on project efficacy. Centralized and adaptive hybrid structures seem to promote superior performance, but the effectiveness of flexible and decentralized models depends on the group’s capacity to manage the limitations of less direct oversight and collaboration.

Discussion and conclusion

This article provides a twofold contribution. First, it evaluates FOI implementation in Belgian municipalities, thereby broadening our international understanding of FOI law effectiveness in local governments. Second, it meets the growing demand for experimental research in policy evaluation by offering empirically based insights into the organization of citizen science projects to assess transparency policies.

The implementation of FOI laws in Belgium: An international comparison

First, our study reveals significant regional variations in the implementation of FOI laws in Belgium. Municipalities in Brussels and Flanders exhibited higher compliance rates than Wallonia, suggesting that different regional governance models and administrative practices influence transparency. Similar regional differences were found in other studies. Wagner’s (2021) research across nine US states illustrated how regional variations could significantly affect FOI law execution, with Central and Southern regions showing diminished responsiveness and extended delays. Cicatiello et al.’s (2024) research on FOI implementation in Italy also highlighted that regional governance systems create territorial divides, with Northern Italy showing more effective responses due to robust governance, ample resources, and a strong culture of transparency, unlike the strained and corrupt Southern regions. These findings are consistent with regional differences observed in Belgium (Esposito et al., 2021). In Brussels and Flanders, which have a more service-oriented economy, higher gross domestic product (GDP) per capita, and better human capital and information and communications technology (ICT) infrastructure, public officials benefit from robust governance systems and ample resources that are likely to make them feel well equipped and supported, leading to more transparent and effective FOI responses. Conversely, in the Walloon region, where the economy is less service-oriented and levels of GDP per capita, human capital, and ICT infrastructure are lower, officials face less robust governance and limited resources, leading to less effective responses.

Second, in line with Spáč et al. (2018) and Cicatiello et al. (2022b), we observed that municipality size impacts Belgian municipalities’ responsiveness to FOI requests. Larger municipalities with better-funded administrative systems and more advanced technological infrastructure handled requests more efficiently. This highlights that increasing administrative resources and investing in technology may lead to more effective FOI implementation. Our study also indicates that large municipalities may have specialized staff with expertise in handling FOI requests, suggesting that targeted training and expertise development can significantly enhance transparency outcomes.

Third, our study reveals that Belgian municipalities are more reluctant to respond to FOI requests from individuals perceived as knowledgeable about transparency policy and FOI functioning, such as university professors and advocacy organizations like Anticor. This finding aligns with Cicatiello et al. (2024), who observed similar reluctance in Italian municipalities, particularly in the South, toward requests from legally knowledgeable individuals, such as lawyers. Cicatiello et al. (2024) adopt a transaction cost economics perspective, arguing that legal expertise signals a higher likelihood of initiating disputes, leading to elevated transaction costs. Consequently, administrations may refrain from responding to avoid these costs. Similarly, professors or advocacy members with significant knowledge of FOI laws can pose comparable threats due to their capacity to challenge non-disclosure effectively. This suggests the presence of behavioral biases, such as risk aversion and defensive attribution biases, influencing public officials’ behavior. Officials may fear increased scrutiny or legal challenges, leading to defensive behavior and hesitance to engage openly with knowledgeable requesters.

Fourth, our study examines the role of citizen commitment in submitting FOI requests, building on previous research (Cicatiello et al., 2024; Cuillier, 2010; Grimmelikhuijsen et al., 2018; Lagunes and Pocasangre, 2019; Michener et al., 2020; Michener and Rodrigues, 2015; Spáč et al., 2018; Wagner, 2021; Worthy et al., 2017). We found that more committed individuals tend to receive more comprehensive responses, highlighting the vital role of citizen engagement and dedication in driving transparency within public administrations. However, we are aware that commitment alone is not enough; citizens must also strategize to enhance their access to public information. For example, Cuillier (2010) demonstrated that assertive tones compelled quick responses due to perceived urgency. Similarly, Worthy et al. (2017), Spáč et al. (2018), and Grimmelikhuijsen et al. (2018) showed that formal requests citing specific FOI norms signal an awareness of rights, often leading to more diligent responses from public officials. In line with Heimstädt and Dobusch (2018), these dynamics suggest that FOI implementation is a negotiation process between municipal authorities’ officials (information providers) and requesters (information seekers), including individual citizens, academics, lawyer, or advocacy organizations. Our research indeed identifies variations in responses based on the perceived knowledge and authority of the requesters, indicating that public officials balance the demands for information with the perceived risks and costs of disclosure. Citizens, in turn, must commit to this process by deploying strategies to increase their chances of securing the requested information.

Toward a citizen science framework for public policy evaluation

Citizen science is a transformative methodology for public policy evaluation, democratizing scientific research by involving the public directly in scientific endeavors (Vohland et al., 2021). This approach bridges the gap between academic research and community engagement, fostering inclusivity and leveraging diverse perspectives and local knowledge (Hecker et al., 2019).

A study by Tauginienė et al. (2020) found that citizens in social sciences and humanities primarily contribute through data collection. They are rarely involved in the initial stages of research, such as designing the study, or later stages like data analysis and interpretation. However, our study shows that citizen scientists can be effectively engaged in various stages of the research process, from research design to data analysis and interpretation, provided they receive appropriate training and organizational support from professional researchers.

Our findings indicate that the success of citizen science in public policy evaluation depends on the commitment and engagement of the citizen scientists. Varying levels of commitment can influence the outcomes of FOI requests. Structured organization and clear role definitions within citizen scientist groups help manage workload and maintain commitment, aligning their contributions with the overall research project’s goals. These findings are consistent with studies in the natural sciences, which show a significant correlation between the research outputs of citizen scientists and professional researchers, with volunteers performing better when they receive appropriate training and organizational support (Dickinson et al., 2010; Mason and Arathi, 2019).

Our study contributes to understanding the necessary organizational structures for conducting citizen science projects effectively (Ebitu et al., 2021). A well-defined governance framework at the project level is crucial for providing oversight and coordination. At the group level, a more flexible approach allows citizen scientists to adopt organizational practices that better suit their needs. The importance of organizational frameworks to structure the work of citizen scientists can therefore influence the future development of citizen science in policy studies.

However, one should bear in mind that, originating in natural science, citizen science often functioned as self-taught amateur naturalists, driven by their own motivations and timelines, sometimes making unanticipated discoveries without adhering to standardized protocols. Therefore, it is important that in applying citizen science to social sciences and humanities, and specifically in developing a citizen science approach to public policy evaluation, researchers pay adequate attention to maintaining citizen motivation and creativity, thereby enhancing their added value to the scientific enterprise. Providing recognition of their contributions, along with training and development opportunities, can enhance motivation and creativity. Creating an inclusive environment where citizen scientists feel valued and respected is crucial, as is ensuring their work has a tangible impact on policy and community outcomes.

In summary, the application of citizen science in our study not only enhances the understanding of FOI implementation in Belgium but also contributes to the broader development of citizen science in public policy evaluation. As this methodology expands into new domains, we offer a model for future studies seeking to harness the power of citizen engagement in policy evaluation. Future research should focus on exploring the scalability of this framework and its effectiveness in different policy contexts. The potential of citizen science to transform policy evaluation is immense, and our research contributes a crucial step toward realizing this potential.

Supplemental Material

sj-docx-1-evi-10.1177_13563890241270195 – Supplemental material for Toward a citizen science framework for public policy evaluation: Lessons from a field experiment on Freedom of Information laws in Belgium

Supplemental material, sj-docx-1-evi-10.1177_13563890241270195 for Toward a citizen science framework for public policy evaluation: Lessons from a field experiment on Freedom of Information laws in Belgium by Giovanni Esposito, Lorenzo Cicatiello, Vincent Mabillard and Giuseppe Lucio Gaeta in Evaluation

Footnotes

Acknowledgements

We would like to extend our heartfelt thanks to the citizen scientists who participated in the research project that forms the basis of this article: Mehdi Adib, Sekou Amadou Berete, Malak Benhaddou, Antoine Bostyn, Guillaume Briegleb, Davide Castiglioni, Sébastien Ceuppens, Carole Dewael, Quentin Dewinne, Aurélie Duchatelet, Brahim Ebn Aissa, Dominika Godlewska, Ioana-Madalina Gondos, Justine Grandelet, Emine Hasani, Sébastien Heps, Grégoire Hostens, Élodie Jeangout, Sarah Kabedi Kabongo, Yusuf Koyuncu, Clarysse Ledent, Mathis Menestria, Alexis Monier, Yousra Najem, Marie-Christine Ndjali-Ya-Longo, Marion Philippe, Sayed Qaderi, Diego Ramanana, Théo Rolland, Clémentine Schumacker, Stephan de Silva, Aurore Tresnie, Dimitri Van Hassel, Lorenzo Riccobon Vega, and Pranvera Zavalina. We also wish to express our gratitude to Benedetta Scotti and Olivier Van Heuverzwijn for their assistance in implementing the field experiment and to Claude Archer and Patrick Installé from Anticor for their contributions to the experiment with the platform Transparencia.be.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Giovanni Esposito

Supplemental material

Supplemental material for this article is available online.

Notes

Giovanni Esposito is a Professor at the Université Libre de Bruxelles and the academic director of the Centre d’Étude des Politiques et de l’Administration Publique (CEPAP) at the Institute of Sociology. He holds a PhD in Public Management from the University of Liège. His research focuses on public policy and administration, state–citizen interactions, policy process analysis, and critical policy studies.

Lorenzo Cicatiello is an Assistant Professor at the University of Naples L’Orientale. He holds a PhD in Public Economics from the University of Naples L’Orientale. His research focuses on the analysis of public policies, in particular, transparency of governments, administrations, and public budgets.

Vincent Mabillard is an Assistant Professor at the Université libre de Bruxelles. He holds a PhD in Public Administration from the University of Lausanne. His work focuses mostly on administrative transparency.

Giuseppe Lucio Gaeta is an Associate Professor at the University of Naples L’Orientale. He holds a PhD in Public Economics from the Italian Institute of Human Sciences. His research primarily focuses on public policy evaluation and public administration.

References

Ansell

Gash

(2008) Collaborative governance in theory and practice. Journal of Public Administration Research and Theory 18(4): 543–71.

Beierle

Cayford

(2002) Democracy in Practice: Public Participation in Environmental Decisions. Washington, DC: Resources for the Future Press.

Bonney

Ballard

Jordan

, et al. (2009) Public participation in scientific research: Defining the field and assessing its potential for informal science education. A CAISE Inquiry Group Report. Washington, DC: Center for Advancement of Informal Science Education (CAISE).

Bryson

Barbara

Crosby Stone

(2006) The design and implementation of cross-sector collaborations: Propositions from the literature. Public Administration Review 66: 44–55.

Cicatiello

De Simone

Di Mascio

, et al. (2022a) Exploring FOI publicity patterns: The case of Italian municipalities. Government Information Quarterly 39: 101678.

Cicatiello

De Simone

Di Mascio

, et al. (2022b) Exploring patterns of implementation of the Freedom of Information Act (FOIA) in local government: The case of Italy. Etica Pubblica 2: 63–79.

Cicatiello

De Simone

Di Mascio

, et al. (2024) Response, awareness and requester identity in FOI law: Evidence from a field experiment. Journal of Economic Behavior & Organization 220: 12–30.

Council of the EU (2021) Council recommendation on a pact for research and innovation in Europe. Document 14136/21, 19 November. Brussels: Council of the EU.

Cuillier

(2010) Honey v. vinegar: Testing compliance-gaining theories in the context of freedom of information laws. Communication Law and Policy 15(3): 203–29.

10.

Dickinson

Zuckerberg

Bonter

(2010) Citizen science as an ecological research tool: Challenges and citizen. Annual Review of Ecology, Evolution, and Systematics 41: 149–72.

11.

Ebitu

Avery

Mourad

, et al. (2021) Citizen science for sustainable agriculture—A systematic literature review. Land Use Policy 103: 105326.

12.

Eitzel

Cappadonna

Santos-Lang

, et al. (2017) Citizen science terminology matters: Exploring key terms. Citizen Science: Theory and Practice 2: 1–20.

13.

Emerson

Nabatchi

Balogh

(2012) An integrative framework for collaborative governance. Journal of Public Administration Research and Theory 22(1): 1–29.

14.

Esposito

Clement

Mora

, et al. (2021) One size does not fit all: Framing smart city policy narratives within regional socio-economic contexts in Brussels and Wallonia. Cities 118: 103329.

15.

Esposito

Terlizzi

Pichault

(2024) The Panopticon reloaded: A critical analysis of performance management systems in the trans-European transport network policy. Environment and Planning C: Politics and Space 42(3): 476–98.

16.

Garcia

Wallerstein

Hricko

, et al. (2013) The (trade, health, environment) impact project: A community-based participatory research environmental justice case study. Environmental Justice 6(1): 17–26.

17.

Gerlak

Heikkila

(2011) Building a theory of learning in collaboratives: Evidence from the everglades restoration program. Journal of Public Administration Research and Theory 21(4): 619–44.

18.

Göbel

Ottolini

Schulze

(2021) Science as a lever: The roles and power of civil society organisations in citizen science. In: Vohland

Land-Zandstra

Ceccaroni

, et al. (eds) The Science of Citizen Science. Cham: Springer, 331–49.

19.

Grimmelikhuijsen

John

Meijer

, et al. (2018) Do freedom of information laws increase transparency of government? A replication of a field experiment. Journal of Behavioral Public Administration 1(2): 1–10.

20.

Hager

Gold

Wehn

, et al. (2021) Onto new horizons: Insights from the WeObserve project to strengthen the awareness, acceptability and sustainability of citizen observatories in Europe. JCOM: Journal of Science Communication 20(6): A01.

21.

Hecker

Wicke

Haklay

, et al. (2019) How does policy conceptualise citizen science? A qualitative content analysis of international policy documents. Citizen Science: Theory and Practice 432(1): 1–16.

22.

Heimstädt

Dobusch

(2018) Politics of disclosure: Organizational transparency as multi-actor negotiation. Public Administration Review 78(5): 727–38.

23.

Hoekman

Rojas-Romagosa

(2022) EU trade sustainability impact assessments: Revisiting the consultation process. Journal of International Economic Law 25(1): 45–60.

24.

Lagunes

Pocasangre

(2019) Dynamic transparency: An audit of Mexico’s freedom of information act. Public Administration 97(1): 162–176.

25.

Germine

Mehr

, et al. (2022) Developmental psychologists should adopt citizen science to improve generalization and reproducibility. Infant and Child Development 33: 1.

26.

Kahan

(2001) Focus groups as a tool for policy analysis. Analyses of Social Issues and Public Policy 1(1): 129–46.

27.

Keunen

Van Garsse

(2019) Access to information in Belgium. In: Dragos

Kovač

Marseille

(eds) The Laws of Transparency in Action: A European Perspective. Cham: Palgrave Macmillan, 119–62.

28.

Maggioni

Nelson

Mazmanian

(2012) Stakeholder participation and state climate change policies. Policy Studies Journal 40(2): 234–55.

29.

Mason

Arathi

(2019) Assessing the efficacy of citizen scientists monitoring native bees in urban areas. Global Ecology and Conservation 17: 00561.

30.

Michener

Rodrigues

(2015) Who wants to know? Assessing discrimination in transparency and freedom of information regimes. In: Fourth Global Conference on Transparency Research, Lugano, Switzerland, 4–6 June.

31.

Michener

Velasco

Contreras

, et al. (2020) Googling the requester: Identity-questing and discrimination in public service provision. Governance 33: 249–67.

32.

Renn

Webler

Rakel

, et al. (1993) Public participation in decision making: A three-step procedure. Policy Sciences 26: 189–214.

33.

Sanz

Holocher-Ertl

Kieslinger

, et al. (2020 [2014]) White Paper on Citizen Science for Europe. Brussels: European Commission.

34.

Senabre Hidalgo

Perelló

Becker

, et al. (2021) Participation and co-creation in citizen science. In: Vohland

Land-Zandstra

Ceccaroni

, et al. (eds) The Science of Citizen Science. Cham: Springer, 199–218.

35.

Siddiki

Kim

Leach

(2017) Diversity, trust, and social learning in collaborative governance. Public Administration Review 77(6): 863–74.

36.

Spáč

Voda

Zagrapan

(2018) Does the freedom of information law increase transparency at the local level? Evidence from a field experiment. Government Information Quarterly 35(3): 408–17.

37.

Tassie

Murray

Cutt

, et al. (1996) Rationality and politics: What really goes on when funders evaluate the performance of fundees? Nonprofit and Voluntary Sector Quarterly 25: 347–63.

38.

Tauginienė

Butkevičienė

Vohland

, et al. (2020) Citizen science in the social sciences and humanities: The power of interdisciplinarity. Palgrave Communications 6: 89.

39.

Taylor

de Loë

(2012) Conceptualizations of local knowledge in collaborative environmental governance. Geoforum 43(6): 1207–17.

40.

Vohland

Land-Zandstra

Ceccaroni

, et al. (2021) The Science of Citizen Science. Cham: Springer.

41.

Wagner

(2021) Piercing the veil: Examining demographic and political variables in state FOI Law administration. Government Information Quarterly 38(1): 101541.

42.

Wehn

Ajates

Fraisl

, et al. (2021) Capturing and communicating impact of citizen science for policy: A storytelling approach. Journal of Environmental Management 295: 113082.

43.

Wiggins

Crowston

(2011) From conservation to crowdsourcing: A typology of citizen science. In: 2011 44th Hawaii international conference on system sciences, Kauai, HI, 4–7 January, 1–10. New York: IEEE.

44.

Worthy

John

Vannoni

(2017) Transparency at the parish pump: A field experiment to measure the effectiveness of Freedom of Information requests in England. Journal of Public Administration Research and Theory 27(3): 485–500.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB