Occupational health and safety regulatory interventions to improve the work environment: An evidence and gap map of effectiveness studies

Mapping the evidence on occupational health and safety regulatory interventions reveals that more research is needed

Evidence for the effects of occupational health and safety regulatory interventions is unevenly distributed across intervention domains, pointing to both evidence clusters suited for further systematic review activity and evidence gaps where more primary research is needed.

What studies are included?

The EGM includes six systematic reviews, 28 effect studies, and three on-going studies. Included studies had to be either randomised controlled trials, non-randomised studies with a comparison of two or more groups of participants, or systematic reviews of effects.

Furthermore, the population of interest was restricted to workers above the age of 15 employed at workplaces within the OECD.

What are the main findings of this EGM?

Evidence for the effects of health and safety regulatory interventions is unevenly distributed across intervention domains. The largest cluster of studies is found in the intervention domain covering inspection activity (21 studies).

There are a number of studies investigating information, guidance and consulting activities (12 studies), incentives for compliance (seven studies), and regulatory enforcement in the form of sanctions (five studies).

There are very few studies investigating the effects of training initiatives (two studies) and formulating regulatory standards (one study).

Included systematic reviews were critically appraised. This appraisal led to the conclusion that there is a lack of rigorously-performed systematic reviews exploring the effects of occupational health and safety regulatory interventions.

In terms of the geographical spread of studies across the OECD area, North America (USA and Canada) is the most densely populated, followed by Europe (represented by countries located in both the Northern and Southern parts of the continent), and finally Eastern Asia (represented by South Korea).

What do the findings of this EGM mean?

This EGM provides an opportunity for stakeholders from the research, policy and practice communities to gain an overview of the available evidence on the effects of occupational health and safety regulatory interventions.

By pointing to both evidence clusters and gaps in the current evidence base, the EGM can inspire future research projects and intervention activities.

How up-to-date is this EGM?

The authors carried out literature searches up to January 2023.

Introduction

EGM: Definition and purpose

This EGM provides a visual overview of the available evidence from primary quantitative effect studies and systematic reviews on the effects of a set of occupational health and safety regulatory interventions. In this sense, the EGM identifies what quantitative empirical knowledge is available about the effects of these interventions by mapping available systematic reviews and primary studies of effectiveness and placing them in a graphical matrix displaying areas characterised by strong, weak, or nonexistent evidence. It is important to note here that an EGM shows the availability of data, but does not synthesise the data. As expressed by Saran (2018) in their paper on approaches to EGMs: ‘Evidence maps summarize what evidence there is, not what the evidence says’ (Saran, 2018, p. 11).

The present EGM consists of two primary dimensions: rows listing interventions, and columns listing outcomes, meaning that each cell shows studies containing evidence on that particular combination of intervention and outcome. The map also contains relevant filters (study design including AMSTAR-2, OECD country, and publication status) making it possible to focus on a subset of studies meeting certain criteria. The map has been populated based on:

Criteria for inclusion and exclusion of studies.

Framework development and scope

We have followed the standard EGM framework as a matrix, with rows containing intervention domains and columns containing outcome categories. Our framework of specific interventions and outcomes, as described in the following, has been developed in cooperation with stakeholders from The Danish Working Environment Authority. It has been our goal to build a framework of relevance, not only to other researchers, but also to decision makers and practitioners who are tasked with developing regulatory standards, performing inspections and enforcing compliance, and taking other measures to improve working environment conditions.

Stakeholder engagement

Engaging primary stakeholders in an EGM can play a central role in defining the scope of the investigation, developing a coherent framework, and making sure that the questions asked are the most relevant and pertinent to the research subject. As noted by Keown (2008), a systematic review (or in this case, an EGM) provides a number of engagement opportunities in various phases of the review process, from cooperating with stakeholders on finding a topic of mutual relevance, to arranging on-going feedback meetings, and involving stakeholders in the dissemination of the review or EGM to reach, for example, practitioner groups. In the current study, we have drawn on the support of a group of working environment experts within The Danish Working Environment Authority, as they are uniquely positioned to judge what the most pertinent topics are when it comes to enforcing occupational safety and health regulations. In line with Keown (2008), it has been our aim to engage our stakeholders in all parts of the EGM process, starting by consulting with them in the initial process of setting the scope of the EGM and defining relevant outcomes and interventions. This process was initiated in the Fall of 2019 when an open meeting was held between the research team from VIVE and working environment experts from The Danish Ministry of Employment and The Danish Working Environment Authority who expressed their needs for a better overview of existing occupational health and safety research. Having guided us in determining the scope of the EGM, the stakeholders further provided feedback on the title registration and subsequent protocol drafts. This cooperative process has been instrumental in increasing the relevance and precision of the EGM framework. Going forward with the EGM, we have kept our stakeholders updated on the mapping process and the ensuing results, asking for their assessments of, for example, the clarity of presentation and user-friendliness of the final map. Following publication of the EGM, we will also enlist the support of our stakeholders in determining how best to disseminate the results of the investigation to key interest groups.

In addition to the VIVE research team, three international researchers in working environment issues have contributed as co-authors of the EGM:

Professor Peter Hasle, University of Southern Denmark.

It is important to state that the involvement of The Danish Ministry of Employment and The Danish Working Environment Authority in this EGM has amounted only to consultative support and has not entailed ownership or financial support. Any research decisions made have been the responsibility of the review authors (i.e., the research team from VIVE and the three international researchers) and the conclusions of this EGM are the sole responsibility of the authors.

Conceptual framework

Figure 1 shows the logic model for the interventions and how they link to the outcomes. This does not provide a detailed theory of change of how the interventions may work, but provides a conceptual framework of how we imagine that the inputs in focus may lead to the chosen outcomes. As noted previously, we focus on six types of occupational health and safety regulatory initiatives: (1) formulation of regulatory standards, (2) incentives for compliance, (3) inspection by regulatory agencies, (4) enforcement by regulatory agencies (sanctions), (5) information, guidance, and consulting, and finally (6) training initiatives. These six types of initiatives may have an effect on outcomes through processes of general deterrence, which would be the case if, for example, inspections led to an increased general awareness among businesses of the risk of inspection and potential punishment, resulting in increased compliance with regulatory standards. Interventions may also have specific deterrence effects, as would be the case if actual inspections and consequent punishments of non-compliant businesses lead to decreases in violations among those punished (Mischke, 2013). As the focus of this EGM is on initiatives by working environment regulatory authorities or regulatory agencies and other organisations assigned as regulators, we have excluded interventions started by individual businesses or employers at their own initiative.

OPEN IN VIEWER

Figure 1

Logic model of interventions and outcomes.

In terms of outcomes, regulatory enforcement is thought to influence health and safety at workplaces both on an organisational (workplace) level and on an individual (worker) level. Workplace compliance with regulatory standards and work-related exposures are positioned as intermediate outcomes, since the ultimate goal of regulatory enforcement is to reduce negative final outcomes (which are work-related injuries, health, and sickness absence in this EGM); compliance and reductions in exposure to hazardous materials are essentially means to that end (Tompa, 2016).

Dimensions

As stated previously, this EGM has two primary dimensions: interventions (rows) and outcomes (columns). Additional dimensions in the form of filters are: 1.

Study design, including AMSTAR-rating for systematic reviews.

In the following, we specify the criteria that have been used to determine eligibility of references based on study design, interventions, population, and outcomes.

Types of study design

This EGM focuses on the effectiveness of occupational health and safety regulatory interventions, including measures to ensure compliance and/or deter non-compliance with regulatory standards. To provide an overview of what is known about the effects of such interventions, at the individual and workplace level, we have included all study designs that use a well-defined control group. The study designs eligible for inclusion in this EGM have therefore been: 1.

Randomised controlled trials (RCTs).

We have defined a systematic review and a meta-analysis in line with the definitions used by PRISMA and The Cochrane Collaboration, in which it is stated that: ‘(…) a systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review (…) Meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies’

Studies using single group pre–post comparisons have been excluded. A further requirement posed on all types of studies (randomised as well as non-randomised) has concerned their ability to identify an intervention effect. Studies where, for example, the treatment was given to one single organisation/company/workplace only and the comparison group was another organisation/company/workplace (or more organisations/companies/workplaces for that matter) would not be able to separate the treatment effect from the organisation/company/workplace effect. Therefore, it has been a requirement that included studies must have at least two organisations/companies/workplaces in both the treatment and comparison group.

Furthermore, NRS using an instrumental variable approach have been excluded—see Supporting Information: Appendix 1 (Justification of exclusion of studies using an instrumental variable (IV) approach) for our rationale for excluding studies with these designs.

We have also taken special caution concerning studies using regression discontinuity designs (RDDs) to estimate the treatment effect. In sharp RDDs, a threshold of a (non-manipulable) forcing/running variable determines which workplaces receive a treatment and which do not, that is, the design is similar to an RCT in the sense that the random sequence determining treatment assignment can be seen as a running variable (Lee, 2008). In contrast, in ‘fuzzy’ RDDs, being on one side of a threshold only makes it more likely that a workplace ends up in the treatment or control group, and the threshold is used as an instrument to estimate local average treatment effects (LATE) (Angrist, 2009; Imbens, 2008). That is, fuzzy RDD is a form of IV analysis, which we have excluded. Sharp RDDs have been included.

Finally, we have not included qualitative research.

Please note that in implementing the above restrictions on types of study designs, we acknowledge that we have limited our focus to a selected field of the occupational health and safety literature on regulatory interventions. We are fully aware of the methodological diversity of this research field and of the value generated by using different types of methodologies to gain insight into the complexities of such interventions. An example of this is the contributions made within the sociology of law, represented by, for example, Keith Hawkins,

Our approach in this EGM has specifically targeted quantitative effect studies. In conducting the EGM and reporting on its’ results, we wish to make it clear that this study does not aim to give a full and comprehensive overview of the regulatory intervention literature per se; it covers a select subset of studies characterised by the use of distinct, quantitative methods of effect estimation.

Types of intervention/problem

In this EGM, our focus has been exclusively on interventions initiated by working environment regulatory authorities or regulatory agencies and other organisations authorised as regulators, thus excluding interventions started by individual businesses or employers at their own initiative. In Table 1, we present the included intervention categories.

Types of population

The population of relevance to this EGM has been workers above the age of 15 and their workplaces. Conceivably, significant differences exist between the occupational health issues faced by developing nations as opposed to those seen in high-income countries, which makes it difficult to create a coherent and comparable framework to cover occupational health interventions on a global scale. Therefore, we have chosen to limit our scope to workplaces located in nations within the OECD. Note here that it is the workplaces that had to be located in OECD countries, whereas workers in these workplaces may be citizens of all countries.

Types of outcome measures

Contraventions of occupational health and safety regulatory standards and poor working conditions may result in a number of adverse outcomes of which we have included two intermediate and three final outcomes (see also Figure 1):

Workplace compliance with regulatory standards (intermediate outcome): Workplace compliance with regional, national, or international working environment regulatory standards concerning, for example, the company's health and safety performance, ergonomics, use of technical machinery, and dangerous chemicals, as monitored by regulatory agencies.

Other eligibility criteria

As previously noted, we have included only workplaces located in OECD nations.

Search methods and sources

Relevant studies were identified through searches in electronic databases, hand search in journals, grey literature searches, searches in repositories and web resources, citation tracking, and contact to international experts, with the aim of retrieving both published, on-going, and unpublished studies.

Example of a search string

The search string below was used to search SocINDEX through the EBSCO search interface and exemplifies the search facets:

OPEN IN VIEWER

#	Query
S15	S4 AND S10 AND S14
S14	S11 OR S12 OR S13
S13	DE (‘effect size’ OR ‘Control Groups’ OR ‘Experimental Groups’ OR ‘Experiments’ OR ‘Matched Groups’ OR ‘Quasiexperimental Design’ OR ‘Randomized Controlled Trials’ OR ‘Comparative Testing’)
S12	AB (((control* OR difference* OR matched* OR random* OR reference* OR compare* OR longitudinal OR cohort*) N3 (group* OR trial* OR test* OR study OR studies OR analy*))) OR AB ((intervent* OR experiment* OR impact* OR ‘systematic review’ OR ‘meta analy*’ OR metaanaly* OR meta-analy* OR ‘gap map’ OR ‘follow-up stud*’ OR ‘follow up stud*’ OR ‘followup stud*’))
S11	TI (control* OR difference* OR matched* OR random* OR reference* OR compare* OR group* OR trial* OR test* OR intervent* OR experiment* OR impact* OR ‘systematic review’ OR ‘meta analy*’ OR metaanaly* OR meta-analy* OR ‘gap map’ OR study OR studies OR analy* OR longitudinal OR ‘follow-up stud*’ OR ‘follow up stud*’ OR ‘followup stud*’ OR cohort*)
S10	S5 OR S6 OR S7 OR S8 OR S9
S9	SU ((occupational N3 (health OR safety))) OR SU ((work* N3 (health OR safety)))
S8	TI ((occupation* N3 (health OR safety))) OR TI ((work* N3 (health OR safety)))
S7	DE ‘INDUSTRIAL hygiene’
S6	AB ((incentive* OR fund* OR subsid* OR recogni* OR award* OR inspect* OR audit* OR consult* OR sanction* OR penalt* OR fine* OR prosecution* OR ‘enforceable undertaking*’ OR ‘order to comply’ OR citation* OR notification* OR violation* OR breach* OR enforce* OR regulation)) OR AB (((information* OR awareness OR training) N3 (campaign* OR initiative* OR program*)))
S5	TI (incentive* OR fund* OR subsid* OR recogni* OR award* OR inspect* OR audit* OR consult* OR sanction* OR penalt* OR fine* OR prosecution* OR citation* OR notification* OR violation* OR breach* OR ‘enforceable undertaking*’ OR ‘order to comply’ OR information* OR awareness OR training OR enforce* OR regulation)
S4	S1 OR S2 OR S3
S3	(((DE ‘WORK environment’) OR (DE ‘WORK environment—Psychological aspects’ OR DE ‘WORK environment—Research’)) OR (DE ‘ERGONOMICS’)) OR (DE ‘WORK environment—Social aspects’)
S2	AB ((work* OR company OR companies* OR firm OR firms OR organization* OR organisation* OR business* OR institut* OR employe* OR worker* OR staff* OR job) N5 (environ* OR occupational))
S1	TI (work* OR company OR companies* OR firm OR firms OR organization* OR organisation* OR business* OR institut* OR employe* OR worker* OR staff* OR job)) AND TI ((environ* OR occupation*))

A complete overview of the search strings used and the resulting references found for each database can be found in Supporting Information: Appendix 2 and Figure 2.

OPEN IN VIEWER

Figure 2

Search in Social Citation Index Expanded (1900–2022) and Social Sciences Citation Index (1956–2022). Search date: 04.01.2022 through the Web of Science platform. Search modes—Boolean/Phrase.

Analysis and presentation

Dependency

In cases where we have located multiple references reporting the same study results, we use the latest or most complete version in the map. That is, we include the version that reports most fully on the outcomes, and in case of no differences between reports, the latest edition. The following included studies: Kemmlert (1996), Rautiainen (2008), van der Molen (2018), Johnson (2019), and McLeod (2019) have one or more secondary references which are listed under their primary references in the list of included studies (see Included studies).

Furthermore, the systematic reviews included in the map contain numerous of the primary studies shown on the map. Our approach has been to include all relevant primary studies whether they have been included in a systematic review or not, and we have performed ‘un-zipping’ of the systematic reviews, meaning that we have ensured that all primary studies included in the systematic reviews are also included individually on the map.

Data collection and analysis

Data extraction and management

Data extraction was performed in EPPI-Reviewer (EPPI-Reviewer) using a codeset designed to accommodate the creation of the EGM in EPPI-Mapper (EPPI-Mapper). The codeset included clickable options with free-text fields supplied to capture all relevant details surrounding the publication itself, the study design, intervention and outcome domains, and population details, as shown in Supporting Information: Appendix 7. The extraction was performed by a research assistant or review author and independently checked by another member of the review team.

Tools for assessing risk of bias/study quality of included reviews

Since systematic reviews are often used for decision making, we performed critical appraisals of the included systematic reviews using AMSTAR-2 (Shea, 2017), which covers 16 domains, seven of which are critical (these critical domains are shown in Figure 3). The 16 domains take the assessor through the integral parts of the review process, from protocol to literature search and over to risk of bias assessment, meta-analytic procedures, and exploration of potential publication biases, ending in an overall rating of high, moderate, low, and critically low confidence. The overall rating is based on the existence of critical or non-critical weaknesses in the 16 domains. We used the full AMSTAR-2 tool, but added three additions that we found to be important for the critical appraisal: (1) assessment of whether the authors present a plan for handling dependent effect sizes, (2) assessment of whether risk of bias assessments of included studies are performed in duplicate, that is, performed independently by two reviewers after which a consensus coding is agreed upon, and (3) assessment of the authors’ investigation of causes for heterogeneity under the domain covering statistical combination of results in the meta-analysis. The overall rating for the systematic reviews is included in the study design filter of the interactive map. It should be noted that we performed quality appraisal of the included systematic reviews in duplicate, that is, two reviewers independently assessed each review and reached a consensus, with the involvement of a third reviewer in cases of uncertainty or disagreement. We did not assess the quality of included primary effect studies, since the purpose of the EGM was to identify existing studies, not to make decisions based on single trials (Welch, 2019).

OPEN IN VIEWER

Figure 3

Assessment of critical domains of the AMSTAR-2 tool.

Description of studies

Results of the search

We summarise the search results in a flow chart in Figure 4. The total number of potential relevant studies was 102.314 after excluding duplicates (electronic databases: 82.336, grey literature: 19.573, hand searches: 15.223, and citation-tracking: 2.641). We screened all studies based on title and abstract; 101.782 references were excluded for not fulfilling the screening criteria. Four studies were written in a language no one in the review team mastered (they are listed in Table 2). Three studies were unobtainable despite efforts to locate them through libraries and searches on the Internet (these studies are listed in Table 5). Accordingly, 525 studies were ordered, retrieved, and screened in full text. Of these, 480 studies did not fulfil the screening criteria and were excluded. As a result, we located a total of 37 studies, of which six were systematic reviews (and in addition, there were 4 secondary systematic reviews, that is, if a systematic review was updated, we characterised the latest publication as the primary review and the rest as secondary), 28 were primary effect studies (in addition, there were 4 secondary effect studies, that is, if the same intervention using identical data was reported in several papers, we characterised the latest publication as the primary study and the rest as secondary), and three were on-going studies. References to the 37 included studies (and their secondary references) are listed in Included studies and Ongoing studies.

OPEN IN VIEWER

Figure 4

Flowchart.

OPEN IN VIEWER

Table 5

Studies not available.

Study	Notes
ERG (2004)	Found on the United States Department of Labor's CLEAR (Clearinghouse for Labor Evaluation and Research) website: https://clear.dol.gov/study/evaluation-osha%E2%80%99s-impact-workplace-injuries-and-illnesses-manufacturing-using-establishment
Mendeloff (1976)	Report prepared for the Office of the Assistant Secretary for Policy, Evaluation and Research (ASPER), U.S. Department of Labor, Washington DC, 1976.
Macpherson (2021)	Final report submitted to the Saskatchewan Workers’ Compensation Board. Partnership for Work, Health and Safety. School of Population and Public Health, University of British Columbia, Vancouver, BC.

The six systematic reviews all had broader inclusion criteria for including studies than what is the case for this EGM; that is, they aimed to locate both studies relevant to this EGM (i.e., studies investigating the effects of regulatory enforcement using designs with a well-defined control/comparison condition) as well as studies irrelevant to this EGM (i.e., studies not investigating regulatory interventions or not using a design with a control/comparison condition). All six systematic reviews were therefore included in this report based on their inclusion criteria. However, the searches performed in the six systematic reviews did not necessarily result in the location of studies of relevance to this EGM. In some cases, only studies not eligible for this EGM were located (i.e., studies that either did not investigate the effects of regulatory enforcement or did not use an eligible study design). Studies that did not end up including eligible studies are described and listed in this report, but not shown on the interactive map. The following four systematic reviews included studies relevant to this EGM and are accordingly shown on the interactive map: Andersen (2019); Dyreborg (2022); Mischke (2013); van der Molen (2018). The remaining two systematic reviews which were Rautiainen (2008) and Cashman (2009) did not include any studies relevant to this EGM and are therefore not shown on the map.

The primary effect studies predominantly included NRS; 24 of the effect studies were NRS and three were RCTs. The following studies used non-randomised designs: Agnesi (2016); Alwall (2020); Björkdahl (2008); Chen (2008); Dahl (2013); Dahl (2022); Foley (2012); Johnson (2019); Kim (2021); Lewchuk (1996); López-Ruiz (2013); Macpherson (2022); Mancini (2005); McCaffrey (1983); McLeod (2018); McLeod (2019); Moran (1985); Nelson (1997); Rubio-Romero (2015); Ruser (1991); Smith (1979); Stocks (2015); Weil (1996); Wickizer (2004). The following studies used randomised controlled designs: Hogg-Johnson (2012); Kemmlert (1996); Soriano-Serrano (2020). Finally, Peto (2016) was both an RCT and a NRS-study.

The three on-going studies included an RCT (Abildgaard, 2021), a CRCT (Indregard, 2019), and a systematic review (Sundstrup, 2022). The ongoing systematic review is not shown on the interactive map, since no clear intervention/outcome categories could be coded based on the trial information available at the time of writing this report.

For a visualisation of the division of included studies by study design, see Figure 5.

OPEN IN VIEWER

Figure 5

Number of studies by study design. Please note that one study used both an randomised controlled trial and a non-randomised studies design and is therefore counted in both categories.

Synthesis of included studies

The interactive EGM contains 34 studies (since two systematic reviews did not include studies relevant to this map, and one ongoing systematic review did not contain sufficient information to allow a clear intervention/outcome coding, as described in ‘Results of the search’). A link for the interactive map is provided in Supporting Information: Supporting Information: Appendix 8 and additionally, Table 6 shows the number of primary studies, systematic reviews, and ongoing studies for each combination of intervention and outcome (with systematic reviews and ongoing studies reported in brackets). When looking at the map, it is immediately clear that the intervention category covering inspection activity is the most densely populated, with a total of 21 studies, five of which are RCTs (one is an ongoing study), 14 are NRSs, and three are systematic reviews (note that one study is both an RCT and a NRS and therefore counted in both categories). The studies exploring the effects of inspections cover all outcome categories, but with most studies located in the injuries domain (15 studies, where one study is counted as both an NRS and an RCT), followed by compliance (11 studies, where one study is counted as both an NRS and an RCT), health (three studies, where one study is counted as both an NRS and an RCT), work-related exposures (three studies), and finally sickness absence (which is covered by only one ongoing study).

The intervention category covered by the second most studies is that of ‘Information, guidance, and consulting activity’, where 12 studies are located, three of which are RCTs (two are ongoing studies), six are NRSs, and three are systematic reviews. In terms of outcomes, the injuries domain is again the most densely populated with eight studies (one of which is ongoing), followed by compliance (five studies, two of which are ongoing), health (three studies, one of which is ongoing), work-related exposures (two studies), and sickness absence (only one ongoing study).

There are seven studies investigating the effects of incentives for compliance, including one RCT, six NRSs, and one systematic review (note that one study is both an RCT and a NRS and therefore counted in both categories). All seven studies in this intervention category include injury incidence as an outcome, with one study additionally looking at health outcomes and compliance (counted as both and RCT and a NRS and therefore illustrated with two bubbles). There are no studies investigating the effects of incentives for compliance on work-related exposures or sickness absence.

Five studies are focused on the effects of enforcement activities: one RCT, one NRS, and three systematic reviews. Four of these studies include injury incidence as an outcome, four studies include a compliance outcome, two studies include work-related exposures, and one study includes health outcomes. As was the case for the incentives category, there are no studies investigating the effects of enforcement activity on sickness absence.

The final two intervention categories, training initiatives and formulation of regulatory standards, are only scarcely populated, with two NRS studies for the training domain and one NRS study for the formulation of regulatory standards domain. The NRS study investigating the effects of setting regulatory standards includes injury incidence as an outcome, and as for the two studies exploring training initiatives, one study includes work-related exposures and one includes compliance as an outcome.

Reflected in the above summation of the map is the fact that we find both relatively well-populated intervention and outcome domains and domains which are empty or only very scarcely covered, the latter of which points to the existence of gaps in the evidence concerning the effects of occupational health and safety regulatory interventions. As for the well-populated domains, there is an evidence base available to those seeking information about the effects of inspection activity, especially related to injury incidence and compliance outcomes. The same is the case for the effects of information, guidance, and consulting activities and incentives for compliance, again centred on incidence of work-related injuries. There is furthermore a smaller evidence base covering the effects of regulatory enforcement in the form of sanctions, scattered with relatively few studies for each outcome domain (excluding sickness absence which has no studies in this intervention domain). When it comes to the evidence gaps identified by this map, two main areas stand out. Firstly, the sickness absence outcome domain is almost empty (populated by only one ongoing study located under two intervention domains). While it is possible that sickness absence is in some cases integrated into or covered by injury incidence or health outcomes, the lack of studies for this domain is still conspicuous and points to a relevant venue for new research. Similarly, two intervention categories, training initiatives and formulation of regulatory standards, are very scarcely populated, meaning that at present point, there is little guidance in the literature as to how regulatory agencies can most effectfully enact new regulatory standards or implement training initiatives.

In terms of the geographical spread of studies across the OECD, the country filter of the interactive map and Figure 6 illustrates that most studies were conducted in The United States (15 studies), followed by Canada (eight studies), Spain and Sweden (five studies each), Italy (four studies), and Norway (three studies). Finally, South Korea, The United Kingdom, and Denmark are represented by one study each. It should be noted that the systematic reviews were coded according to which countries were represented in their included studies, meaning that a systematic review could have more than one country code. As is evident from Figure 6, North America (USA and Canada) is the most densely populated area in terms of number of studies, followed by Europe (represented by countries located in both the Northern and Southern parts of the continent). Finally, Eastern Asia is represented by one study from South Korea.

OPEN IN VIEWER

Figure 6

Number of studies by country. Please note that included systematic reviews may include studies from several countries.

We had originally considered implementing a filter on the map for work sectors/industrial groupings, but refrained from doing so since detailed information about work sectors was not available or unclear in a number of studies, and there was no consistent sector division or typology common to the included studies. However, it was evident that the manufacturing and construction sectors were well represented in the included studies.

Risk of bias in included reviews

We located six systematic reviews in our searches and critically appraised their quality using AMSTAR-2 (with three additions, see ‘Tools for assessing risk of bias/study quality of included reviews’). The full rating with justification for each domain is documented in Supporting Information: Appendix 9 and Figure 3 gives a visual illustration of our assessment of critical domains of the AMSTAR-2 tool for the six included systematic reviews. As a result of the appraisal, three systematic reviews received a ‘critically low’ rating, one received a ‘low’ rating, and two did not receive a full rating due to them not containing studies eligible for this EGM.

The four systematic reviews that received a full AMSTAR-rating all included both studies which were eligible for this EGM (relevant intervention, outcome and study design) and studies which were ineligible (e.g., interrupted time series designs without a control group). When performing the AMSTAR-assessments, we focused only on the authors’ treatment of studies eligible for this EGM. As for the three systematic reviews with ‘critically low’ confidence, main limitations included the absence of appropriate methods for statistical combination of results in the meta-analysis and inadequate investigation of the potential impact of publication bias. Furthermore, heterogeneity was not sufficiently assessed and discussed. According to the AMSTAR-guideline, ‘critically low’ confidence entails that the review has more than one critical flaw (with or without non-critical weaknesses) and should not be relied on to provide an accurate and comprehensive summary of the included studies. As for the review with low confidence, the only flaw was a failure to adequately investigate publication bias; otherwise, this review was well-performed and followed standard systematic review procedures.

The two systematic reviews without eligible studies for this EGM were those of Rautiainen (2008) and Cashman (2009). Rautiainen (2008) aimed to include randomised controlled trials, cluster-randomised controlled trials, prospective cohort studies with a concurrent control group, and interrupted time series studies. However, the literature search only located studies irrelevant to this EGM (five RCTs/CRCTs that did not investigate regulatory enforcement and three ITS studies with no control group). Therefore, the AMSTAR-assessment of this review was only performed in relevant domains (e.g., risk of bias and meta-analysis domains were not relevant when no eligible studies were involved) and no overall AMSTAR-rating was given. Cashman (2009) aimed to include RCTs, cluster-randomised trials, controlled clinical trials, controlled before-and-after studies (more than three time points to be measured before and after the study), and interrupted time-series (ITS) studies. However, as a result of the searches, authors located just two interrupted time-series studies, none of which were relevant to this EGM, and the AMSTAR-assessment was performed accordingly, with no overall rating given.

Overall, our assessment of the included reviews suggests that there is a lack of high quality systematic reviews and meta-analyses exploring the effects of occupational health and safety regulatory interventions. This points to the relevance of commissioning and performing more rigorous systematic reviews in areas sufficiently covered by primary effectiveness studies. One such area which stands out in the current EGM is that of inspection activity which is populated by 17 completed primary studies and one ongoing cluster-randomised controlled trial.

Summary of main results

After performing comprehensive searches for both published and unpublished literature, we ended up with a total of 37 studies, six of which were systematic reviews (with four related secondary systematic reviews), 28 of which were primary effect studies (with an additional four related secondary effect studies), and three of which were on-going studies. The 28 primary effect studies predominantly used non-randomised designs. It should be noted that since two of the systematic reviews did not contain intervention and outcome domains and/or study designs eligible for this EGM and one ongoing systematic review did not contain sufficient information to allow a clear intervention and outcome coding, the final map includes 34 studies. Furthermore, as a result of the AMSTAR-rating, three systematic reviews received a ‘critically low’ rating, one received a ‘low’ rating, and two did not receive a full AMSTAR-rating due to them not containing studies eligible for this map.

As noted in Results, this EGM uncovers both relatively well-populated and empty or nearly empty intervention and outcome domains. To sum up, there is an evidence base available concerning the effects of inspection activity on injury incidence and compliance outcomes as well as for the effects of information, guidance, and consulting activities and incentives for compliance on incidence of work-related injuries. Additionally, there is some evidence covering the effects of regulatory enforcement in the form of sanctions.

Areas of major gaps in the evidence

The map identifies two main evidence gaps: (1) there is a lack of evidence for the sickness absence outcome domain, which is populated by only one ongoing study, and (2) there is very limited evidence available concerning the effects of regulatory agencies formulating regulatory standards and carrying out training initiatives. Furthermore, our assessment of the included reviews points to a lack of rigorously performed systematic reviews and meta-analyses investigating the effects of occupational health and safety regulatory interventions. It is therefore of relevance to researchers and research commissioners to consider initiating more systematic reviews in areas sufficiently covered by primary effectiveness studies, such as that of inspection activity which is well populated on this EGM.

Potential biases in the mapping process

For this EGM, we performed extensive searches for both published and unpublished literature and screened the records found in duplicate. Furthermore, coding of included studies and assessment of systematic reviews with the AMSTAR-tool was also performed in duplicate and thoroughly checked. There was a small number of potentially relevant studies which were excluded due to inavailability or language proficiency restrictions within the review group. Otherwise, we believe that we have taken all possible precautions to eliminate or diminish the potential for biases in the mapping process.

Limitations of the EGM

As is the case for all types of evidence maps or syntheses, it has been necessary to implement restrictions on types of interventions and outcomes, types of population (only OECD countries), and types of study designs to create a coherent map and secure a manageable mapping process in terms of number of references (note that we have processed more than 100.000 references for this map). This means that, for example, other types of work environment interventions initiated by workplaces or other non-regulatory institutions are not covered by the current EGM, just as research drawing on other methods, be it quantitative studies without control/comparison conditions, qualitative studies, or narrative reviews, is not included. Also, this EGM only covers the OECD and not, for example, developing nations. It is important for us to stress that the above restrictions are implemented from a wish to ensure coherence and adequate detail in the final map and to create a manageable workload, not from a perspective of, for example, judging some methods or study designs as better than others or to suggest that work conditions outside the OECD are of less relevance. We acknowledge that we have limited our focus to a select field of effectiveness research within a limited set of nations and are fully aware of the diversity and value generated by using different types of methodologies and exploring work conditions across a wide array of geographic locations, just as we are aware of the inherent complexity of regulatory processes and the challenges in performing controlled trials in this field. Since the effects of regulatory measures are complex and related to a number of mechanisms and processes in the overall regulatory system, there is a need for more research drawing not only on quantitative effect measures, but also on qualitative and mixed methods approaches exploring the depths and complexities of occupational safety and health regulatory enforcement in different institutional structures.

Stakeholder engagement throughout the EGM process

We are very grateful for the engagement of our stakeholders within The Danish Working Environment Authority throughout the process of planning, designing, and completing this map. Their engagement has played a central role in defining the scope of the map and developing a coherent framework of interventions and outcomes. Furthermore, we have received constructive feedback on both the title registration, protocol, and the final report. Following publication of the EGM, we will cooperate with our stakeholders disseminating our findings to key interest groups.

Implications for research, practice and/or policy

This EGM of systematic reviews and primary effect studies uncovers both well-populated and empty or nearly empty intervention and outcome domains in the area of occupational health and safety regulatory intervention research. Most evidence is available concerning the effects of inspection activity, but we have also located a number of studies investigating information, guidance, and consulting activities, incentives for compliance, and, to a lesser extent, regulatory enforcement in the form of sanctions. On the other hand, the map points to a lack of studies including sickness absence as an outcome and a very limited evidence base concerning the effects of formulating regulatory standards and carrying out training activities (initiated by regulatory agencies). More research using rigorous effectiveness designs is therefore needed to better cover these intervention and outcome domains such that work environment authorities, lawmakers, administrators, practitioners, and other key stakeholders may find guidance in their endeavours to improve work conditions and prevent adverse outcomes for workers. This call for more research is however not limited to quantitative effectiveness research, but extends to other types of research as well, including mixed methods designs and in-depth qualitative inquiries which are needed to shed light on the complex mechanisms behind how different types of occupational health and safety regulatory enforcement function and interact with other processes taking place within a given institutional system.

Furthermore, based on our assessment of the included systematic reviews, we find that there is a need for more high quality reviews investigating the effects of all types of occupational health and safety regulatory interventions, as well as for rigorous systematic reviews of other types of research literature exploring central processual or institutional structures within different regulatory systems. One suggestion based on the evidence located with this map would be to carry out a focused systematic review and meta-analysis of the effects of inspection activity since this domain is relatively well-populated by primary effect studies. It should be noted that since we did not perform risk of bias assessments of the included primary studies, no judgments on the quality of primary effectiveness studies within this research field can be made on the basis of this EGM.

Finally, this EGM only includes effect studies investigating workplaces located in OECD member countries. It is of utmost relevance to perform similar mapping procedures for studies performed, for example, in developing nations and more generally, to explore the particular conditions surrounding work environment regulations in different regions of the world with an array of different research methods.