Use of Modified Delphi to Plan Knowledge Translation for Decision Makers: An Application in the Field of Advanced Practice Nursing

Abstract

Disseminating research to decision makers is difficult. Interaction between researchers and decision makers can identify key messages and processes for dissemination. To gain agreement on the key findings from a synthesis on the integration of advanced practice nurses, we used a modified Delphi process. Nursing decision makers contributed ideas via e-mail, discussed and clarified ideas face to face, and then prioritized statements. Sixteen (89%) participated and 14 (77%) completed the final phase. Priority key messages were around access to care and outcomes. The majority identified “NPs increase access to care” and “NPs and CNSs improve patient and system outcomes” as priority messaging statements. Participants agreed policy makers and the public were target audiences for messages. Consulting with policy makers provided the necessary context to develop tailored policy messages and is a helpful approach for research dissemination.

Keywords

advanced practice nursing decision making clinical nurse specialist nurse practitioner knowledge translation and exchange research dissemination

Much has been written about the difficulty of disseminating research to decision makers and the need to develop key messages from research results. Although consensus that research messaging should be created purposively to meet the needs of target audiences exists (Lavis, Robertson, Woodside, McLeod, & Abelson, 2003), researchers and decision makers live in “two worlds” (Lomas, 2000, p. 140), and the successful development of messages by researchers for decision makers is difficult to accomplish in isolation. Perhaps researchers should do more than consider decision makers when crafting messages and in fact enlist decision-maker involvement in the task. The purpose of this article is to share efforts to engage decision makers to assist with the dissemination of the results of a synthesis of research on advanced practice nursing (APN). First, the context of APN in Canada is presented, including specifics about a commissioned decision support synthesis on the integration of clinical nurse specialists (CNSs) and nurse practitioners (NPs). A knowledge transfer framework to guide the dissemination of research findings to organizations is presented. Next, the methodology and findings from a study utilizing modified Delphi methodology to prioritize messaging about the synthesis are presented. The discussion includes issues related to the findings as well as the implications for other researchers wanting to enlist the help of decision makers in efforts to get their research findings out.

Background

In Canada, decision makers at both the policy and organizational levels are the target audiences for nursing research, including APN research. Advanced practice nursing is an umbrella term used to describe “an advanced level of clinical nursing practice that maximizes the use of graduate educational preparation, in-depth nursing knowledge and expertise in meeting the health needs of individuals, families, groups and communities” (Canadian Nurses Association [CNA], 2009, para. 1). Two types of APNs exist in Canada: CNSs and NPs. CNSs have expertise in a clinical nursing specialty and perform a role that includes practice, consultation, collaboration, education, research, and leadership. They address complex health care issues, contribute to the development of nursing knowledge and evidence-based practice, and facilitate system change (CNA, 2009).

Nursing licensing bodies estimate approximately 2,400 CNSs are working in Canada, and a large proportion work in urban areas (Kilpatrick et al., 2013). NPs work in both primary health care (PHC) and acute care settings. NPs possess and demonstrate the competencies to autonomously diagnose, order, and interpret diagnostic tests, prescribe pharmaceuticals, and perform specific procedures within their legislated scope of practice (CNA, 2009). More than 3,000 NPs work in Canada, and the majority are primary care NPs who work in community settings, such as family health teams, community health teams, aboriginal centers, and NP-led clinics (CNA, 2013).

Despite being the second largest country in the world, Canada has a relatively small population (approximately 35 million), and more than 80% of its people live in and around urban centers in the southernmost region of the country. The country is divided into 10 provinces and three territories, and dispersed across the vast geography are smaller urban areas and communities of people living in rural and remote parts, some in the far north. Canada has a publicly funded health care system, the organization of which is determined by the Canadian constitution, which divides the responsibilities among the federal, provincial, and territorial governments (Health Canada, 2013). The provinces and territories are responsible for the delivery of health care and social services. The federal government takes responsibility for health care for specific groups of people, such as First Nations people and veterans. This structure has resulted in regional differences in the way care is delivered, regulations and policies, and the scope of practice for providers.

A number of issues hinder successful integration of CNSs and NPs into health care settings, often because of a lack of coordination across Canada on health human resources issues, education, and legislation and regulatory frameworks (DiCenso et al., 2010). Because of these issues, a decision support synthesis (APN Synthesis) was commissioned by the Canadian Health Services Research Foundation (CHSRF; a national health services research funding foundation) and the Office of Nursing Policy (a federal policy unit at Health Canada; DiCenso & Bryant-Lukosius, 2010). The decision support synthesis process model was developed by CHSRF as a way to summarize research and other evidence for health care managers and policy makers (Lomas, 2007). The intent of the process is to allow decision makers and researchers to develop policy-relevant research questions and then follow a procedure that includes defining the scope of the review, summarizing the research and evidence, extracting implications from the findings, and developing recommendations for management and policy.

The methods used for the APN Synthesis were regular consultation with an advisory board, a scoping review of 468 papers from the published and unpublished literature, 62 key informant interviews, four focus groups, and a multidisciplinary roundtable to formulate recommendations. Throughout the process, regular linkage and exchange between the researchers and decision makers occurred. The final report included an overview of Canadian and international research, integrated with a descriptive analysis of the interview data and a list of recommendations.

Review of the Literature

The difficulties in getting evidence into the hands of decision makers to influence policy have been described elsewhere (Lavis et al., 2002; O’Brien-Pallas & Hayes, 2008). It has been suggested that researchers cannot expect evidence to be used directly to address policy issues; rather, work needs to be done to insert the evidence into policy options or action proposals that support a course of action (Contandriopoulos, Lemire, Denis, & Tremblay, 2010). This implies that researchers need to be immersed in policy discussion to understand how their evidence may translate into action. It also assumes some sort of working relationship or, at the very least, an opportunity to have a dialogue with policy makers; yet, in Canada, the largest proportion of researchers doing health services and policy research work in traditional research settings (i.e., universities) and not in government agencies (Dobrow, Costa, Israr, & Chafe, 2010). Thus, the researcher’s ability to inform policy is dependent on a working relationship with policy makers and an understanding of policy context. A number of variables increase the likelihood that evidence is being used in policies, including high levels of interaction between researchers and decision makers and the timeliness of the research evidence (Lavis, Boyko, Oxman, Lewin, & Fretheim, 2009).

Study Framework

Lavis et al. (2003) developed an overall framework for guiding dissemination of research findings to decision makers in healthcare. The framework uses five questions to shape knowledge translation strategies with decision makers: (a) what should be transferred to decision makers? (the message), (b) to whom should research knowledge be transferred? (the target audience), (c) by whom should the research knowledge be transferred? (the messenger), (d) how should research knowledge be transferred? (the knowledge transfer processes and supporting communication infrastructure), and (e) with what effect should research knowledge be transferred? (evaluation). This framework was used to guide our study and develop our research questions.

Study Objectives

Despite continuous interaction between the research team and an array of decision makers throughout the APN Synthesis process and end-of-project dissemination plans, the enormity of the findings and recommendations warrants a more detailed discussion of how to disseminate results, particularly to nursing leaders and policy decision makers throughout Canada. Recognizing that legitimate knowledge transfer planning must include nursing leaders from provinces, territories, and federal agencies, we designed a consensus-building strategy using the framework as described by Lavis et al. (2003) as a guide. The objective for this study was to determine the best ways to disseminate the findings of the APN Synthesis by engaging nursing decision makers from across Canada and gaining agreement on priorities for knowledge transfer activities. Specifically, the research questions were as follows: (a) What are the key actionable messages from the APN Synthesis? (b) Who are the target audiences for messages from the APN Synthesis? and (c) What processes should be used to distribute messages from the APN Synthesis?

Methods

Participants in our study were members of the Principal Nursing Advisors (PNA) working group. Many federal, provincial, and territorial government departments in Canada have a nursing workforce planner, principal nursing advisor, or chief nursing officer who provides policy advice on nursing workforce issues as part of health system policy development and health human resource planning. The PNA includes members from all provinces, territories, and federal agencies providing health care to groups and was established to provide a forum for networking and sharing best practices. Six members of the PNA were involved in the APN Synthesis process: two were advisory board members, two were interviewed as stakeholders, and two were members of a CHSRF-sponsored roundtable after the APN Synthesis was completed. The principal investigator participated in a policy practicum at the Office of Nursing Policy at Health Canada as part of a postdoctoral fellowship and was a member of the APN Synthesis research team. After being involved with the PNA, the principal investigator decided that the PNA group would be an important stakeholder in helping to shape knowledge dissemination strategies for the APN Synthesis because members of the group represented all regions of the country and the multiple sectors in which CNSs and NPs work. The prior relationship between the principal investigator and the PNA group likely established entree for the research.

The study was designed using a modified Delphi technique, which enables group problem solving in an iterative process of problem definition, discussion, feedback, and revisions. Delphi methodology has been used previously to assist with priority setting in health care (Burnette, Horrow-Howell, & Chen, 2003; Moscovice, Armstrong, Shortell, & Bennett, 1977) and is particularly appropriate when the face-to-face exchange of ideas is difficult and distance and time limitations inhibit frequent meetings (McBride, Pates, Ramadan, & McGowan, 2003). A flow chart of the data collection process is shown in Figure 1. The timing of the study was designed to coincide with a preplanned in-person meeting of the PNA. All 19 members of the PNA were approached to participate in the study, including representatives from the 13 provinces and territories and six federal agencies (Health Canada [First Nations and Inuit Health], Department of National Defence, Veterans Affairs, Correctional Services, Citizenship and Immigration, and the Public Health Agency of Canada). Participants were included if they were proficient in the English language, which resulted in the exclusion of one PNA member. The McMaster University Research Ethics Board granted approval to conduct the study.

Figure 1.

The data collection process (N = 18).

Phase 1

The study was carried out in spring 2010. Initial contact was made via an endorsement e-mail sent to PNA members at their work settings by a senior policy advisor in Health Canada’s Office of Nursing Policy. PNA members were then e-mailed an information letter with details about the study, a request for consent to participate, and three additional documents from the APN Synthesis: the Key Messages, Executive Summary, and Roundtable Recommendations. Participants were asked to read the three APN Synthesis documents, respond to the following three questions, and return responses via e-mail:

What are the priority actionable messages from the APN Synthesis?

To whom should the messages from the APN Synthesis be disseminated?

How should the actionable messages be disseminated?

Participants were also given the opportunity to make a phone appointment with the principal investigator to provide their responses. Data were analyzed by the principal investigator throughout the study. All returned comments were categorized according to the questions and summarized as concisely as possible to allow for easier review and critique during Phase 2.

Phase 2

The second round of data collection occurred during a 3-hr face-to-face meeting of the PNA in eastern Canada. After a presentation of the highlights of the APN Synthesis by the principal investigator, the summary of responses from Phase 1 was presented to the group in a PowerPoint presentation, and participants were asked to evaluate responses, clarify or add to ideas, comment on feasibility, and brainstorm additional ideas. Although participants were asked the same three study questions, when reflecting on Phase 1, they chose to first discuss issues they were experiencing with the integration of NPs and CNSs in their respective jurisdictions. They then crafted context-specific messages from the APN Synthesis. Responses were recorded on flip charts. At the end of the discussion, these responses were reviewed, and participants were asked to identify the priorities. After the meeting, this information was again organized into three categories, and responses were summarized in short statements.

Phase 3

Two weeks after the PNA meeting, an e-mail was sent to PNA members with a list of statements in three categories: priority APN issues, APN synthesis messages, and target audiences for APN synthesis messages. Participants were asked to choose the top three priority statements in each category by marking an “X” beside the statement and then return the e-mail to the principal investigator. A reminder e-mail message was sent 2 weeks after the initial Phase 3 e-mail. Priority statements from all participants were reviewed together, and responses were tallied.

Results

Of the total study participants (N = 18), six (33%) responded in Phase 1, 12 (67%) responded in Phase 2, and 14 (78%) responded in Phase 3. In the third and final phase, all 12 participants (100%) from the provinces and territories responded. As shown in Table 1, 16 participants completed only one phase (89%), 12 participants (67%) completed two phases, and four participants (22%) completed all three phases of data collection. We examined the data to uncover or identify trends related to region, provincial and territorial population, or CNS/NP per capita estimates, and no trends were found.

Table 1.

Composition of the Sample by Subgroup and Participation in Study Phases (N = 18).

	Responses by subgroup, n (%)			Participation in study phases, n (%)
	Phase 1	Phase 2	Phase 3	1 of 3 Phases	2 of 3 Phases	All 3 phases
Provinces (n = 9)	5 (56)	8 (89)	9 (100)	9 (100)	9 (100)	4 (44)
Territories (n = 3)	0	0	3 (100)	3 (100)	0	0
Federal agencies (n = 6)	1 (16)	4 (44)	2 (33)	4 (67)	3 (50)	0
All jurisdictions (N = 18)	6 (33)	12 (67)	14 (78)	16 (89)	12 (67)	4 (22)

Priority APN Issues

Most of the respondents from the provinces and territories agreed on the priority statements in Phase 3, as shown in Table 2. Two thirds of the respondents from the provinces and territories chose the following three Priority APN Issue statements: “maximizing scope of practice for all nursing roles,” “the inability to fill NP positions in rural and remote areas,” and “development of models for primary care, including nurse-led clinics.” There was less agreement from the federal agencies, and two of these statements were not chosen as a priority by federal agency respondents. Four of the five priority issue statements chosen by federal agency respondents were chosen by fewer than half of the provincial and territorial respondents. All respondents agreed on which issue statements were not a priority.

Table 2.

Items Generated in Phase 2 and Identified as Top Three Priorities in Phase 3.

	Provinces (n = 9)	Territories (n = 3)	Federal agencies (n = 2)	Total (n = 14)
Priority APN issues
• Maximizing scope of practice for all nursing roles	7	2		9
• The inability to fill NP positions in rural and remote areas	6	2	1	9
• Development of models for primary care, including nurse-led clinics	6	2		8
• Funding mechanisms for primary care	4	1	1	6
• Standardization of CNS regulation and education	2		1	3
• Legislative barriers to NP role integration	1		1	2
• Specialist referrals for NPs		1		1
• Lack of jobs for NPs in urban areas			1	1
• Classification systems and pay scales for APNs working in federal jurisdictions
• Role scope issues related to CNS roles
• Pay scale issues for CNSs
• The lack of research on CNS roles in Canada
APN synthesis messages
• NPs increase access to care	6	2	2	10
• NPs and CNSs improve patient and system outcomes	5	1	1	7
• Care from NPs and CNSs is cost effective	3	1	1	5
• CNSs promote quality improvement	3	1	1	5
Target audiences for APN synthesis messages
• Policy and decision makers should receive messages about NPs	6	2	1	9
• The public should receive messages about NPs	5	2	2	9
• Public service announcements about APNs are needed	5	1		5
• Employers and executives should receive messages about CNSs	3			3
• APN success stories would help nurses and the public	2		1	3
• APN research should be published in academic journals	1		1	2
• CNS messages should go to nurses

Note. NP = nurse practitioner; CNS = clinical nurse specialist; APN = advance practice nurse.

APN Synthesis Messages and Target Audiences

During the face-to-face meeting in Phase 2 of data collection, participants stated there was a need to separate out NP and CNS messages. In Phase 3, at least two thirds of respondents from the provinces, territories, and federal agencies agreed on the top priority APN Synthesis messaging statement, “NPs increase access to care,” and at least 50% of respondents from the provinces, territories, and federal government agencies agreed on another APN Synthesis messaging statement: “NPs and CNSs improve patient and system outcomes.” More than two thirds of all respondents picked policy and decision makers as the primary target audience to receive messaging about the NP role. Fifty percent chose the public as the target audience to receive APN Synthesis messaging.

Discussion

In three rounds of data collection, provincial, territorial, and federal nursing advisors prioritized APN issues, general messages from the APN Synthesis, and targets for these messages. The response rate was the highest in the third and final round, and two thirds of participants agreed on both priority issues and priority messages. The three priority issues were “maximizing scope of practice for all nursing roles,” “the inability to fill NP positions in rural and remote areas,” and “development of models for primary care, including nurse-led clinics.” The top priority APN Synthesis message was “NPs increase access to care.” Participants also agreed that messages should be directed to policy and decision makers and to the public.

It should be noted that the majority of priority statements was related to one type of APN role—the PHC NP. It is not surprising the PNA members focused on this NP role. Many provincial and territorial governments have invested significantly in the role of the PHC NP to address access-to-care issues within their health systems, thus the increased interest in this role over the CNS and other acute care NP roles. A review of the literature on the exchange of knowledge at the organizational and policy levels suggests that the relevance, legitimacy, and accessibility of the knowledge are crucial to its uptake (Contandriopoulos et al., 2010). A key finding in this study was that decision makers identified and prioritized evidence related to their particular interests and the regions they represented. In this case, many PNA members’ interests are to prioritize messages of the results of the investments in the PHC NP role made by their governments. It is not that there was a lack of interest in the CNS role generally, but there was a real interest in the NP because the job of these advisors is to bring that evidence to policy discussions. The CNS research, although still important, was not a current policy priority for many of the participants.

Many CNSs and NPs work in acute care settings, and for these roles, hospital chief nursing officers or chief executive officers would be the more interested decision makers needed to identify messages. This was validated in the APN Synthesis, where it was noted that government participants seemed to know very little about the CNS role (Bryant-Lukosius et al., 2010). The participants of this study recommended that CNS and NP messaging be separated out, as opposed to making general statements about APNs. Issues related to APN role clarity have been reported previously, as there is a lack of awareness and understanding of the CNS and NP roles and their respective scopes of practice, and is a key issue facing CNSs and NPs in Canada (Donald et al., 2010).

Having decision makers craft and prioritize the messages in this study provided the necessary context in which the evidence would be viewed. For instance, the participants framed the messages in terms of broader health system issues, such as health human resource and PHC reform, as opposed to focusing on the NP or CNS role. Some of the messages were different from those the research team would have created. One of the priority statements, “maximizing scope of practice for all nursing roles” was not message specific to APN at all but a broader nursing issue in Canada. The content of this priority message was somewhat surprising, perhaps indicative of the narrower view researchers with a concentrated focus bring. The large variability in the mandates of members from the provinces and territories and those of federal agencies must also be acknowledged. Members from provinces and territories are responsible for nursing issues affecting general populations, whereas members of federal agencies, such as Corrections Canada, have a much narrower focus.

This exercise likely saved resources and effort associated with the creation and delivery of ineffectual messages. Decision makers have little time to review larger volumes of research for synthesis and uptake of research, so assistance in making the APN Synthesis research messages competitive in getting noticed was welcomed. The key messages are the tags that make the decision makers want to look at the research, and developing these tags with the PNA assisted with knowledge translation efforts. Messaging from researchers must be timely to take advantage of any policy window opportunities. This was a streamlined approach to obtaining input on messaging. The three phases of this priority setting exercise were completed in fewer than 6 weeks.

The primary aim of this study was to engage nursing decision makers to develop APN Synthesis messaging; however, the approach we took might be helpful to other researchers wanting to engage decision makers about their research findings. Use of the modified Delphi process resulted in multiple exposures of our research to senior nursing decision makers. Data collection through e-mail allowed us to reach out to participants across a wide geographic area. Other strategies for other researchers to consider included identifying a national group of senior nursing decision makers (the PNA group), utilizing prior relationships between the research team and decision makers (the postdoctoral relationship with the Office of Nursing Policy and APN Synthesis participants), and organizing the phases of the Delphi study around a planned meeting of the group and securing an invitation to present findings (Phase 2 of the study).

Strengths and Limitations

This study took place within the context of Canadian policies and health care provision, and readers should consider this when interpreting results. The study has three strengths. The rigorous process allowed for iterative, broad-based participation of senior decision makers from across Canada. The use of the modified Delphi technique provided a timely approach to this inquiry, and data collection was completed relatively quickly. However, the limitation of this speedy approach was that few advisors were able to participate in all three phases of the process. In hindsight, more time should have been allowed for participants to respond to the e-mail in the first phase. Furthermore, the costs associated with traveling long distances precluded some PNA members from attending the face-to-face meeting in Phase 2. These two factors resulted in limited participation from members from the northern territories in Phases 1 and 2. However, the rigorous execution of the study resulted in a 78% participation rate in the final prioritization phase, which included all the provinces and territories.

Implications

This study provides an example of the value of decision-maker input on the dissemination of research. The products of this Delphi exercise, which are messages that reflect the context in which the research will be viewed, are a result of outreach by a researcher to a group of senior decision makers. Although there were established relationships and dialogues between the research team and decision makers throughout the APN Synthesis process, the opportunity to interact with a key decision-maker group provided direction for the next steps and served to focus the messaging. For instance, evidence briefs can be developed based on these key messages and distributed to identified target audiences. The opportunity to present findings in person to the PNA in Phase 2 was, in itself, a type of knowledge transfer activity, both a packaging and push effort (an effort to by our research team to disseminate to knowledge users; Grimshaw, Eccles, Lavis, Hill, & Squires, 2012) and an exchange effort (Ellen, Lavis, Ouimet, Grimshaw, & Bedard, 2011).

Having this group craft messages and prioritize messaging strategies went even further to clarify and focus disseminations efforts, and is an approach that could be used in other researcher/decision-maker partnerships. The availability of synthesized results (key findings, executive summary, and recommendations) was critical to our interaction with decision makers. Once a research project is completed, the researcher should identify key audiences and then work with each separately to identify the key messages, prioritize them, and consider ways to get out the word. The modified Delphi process used was effective in carrying out this research efficiently with busy participants across a vast country, and we purposefully created and organized the study using quick-to-complete e-mail surveys that could be easily returned. However, as noted, efforts must be made to focus participants on the evidence presented.

It can be argued that enlisting the assistance of decision makers to craft messages will lead to more effective research dissemination, but this will need to be confirmed. As one of the less common types of knowledge translation research, evaluating the impact of knowledge transfer activities on policy decision making is difficult (Buykx et al., 2012; Lavis et al., 2003). It would also be enlightening to survey the participants in this study to ask for their reactions to the Delphi process, how they might improve it for future use, and whether this would be something they would be willing to engage in regularly with researchers.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The principal investigator (NC) was supported with a Canadian Health Services Research Foundation Postdoctoral Award for this research.

Author Biographies

Nancy Carter, PhD, RN, is an assistant professor in the School of Nursing at McMaster University in Hamilton, Ontario, Canada, and teaches in the undergraduate, graduate, and nurse practitioner programs. She is a researcher in the Canadian Centre for Advanced Practice Nursing Research, focusing on uptake of evidence on clinical nurse specialists and nurse practitioners by administrators and decision makers.

John N. Lavis, MD, PhD, is the Director of the McMaster Health Forum, the Associate Director of the Centre for Health Economics and Policy Analysis, and a professor (in both the Department of Clinical Epidemiology and Biostatistics and the Department of Political Science) at McMaster University and Department of Global Health and Population, Harvard School of Public Health, Boston, MA, USA. His principal research interests include knowledge transfer and exchange in public policy-making environments and the politics of health systems.

Sandra MacDonald-Rencz, MA (Ed), CHE, RN, was the Executive Director of the Office of Nursing Policy, Health Policy branch at Health Canada, from 2004 to 2012. She then assumed a special assignment within the Strategic Policy branch, Health Canada, and provided policy advice on health systems and policy issues across directorates and the branch.

References

Bryant-Lukosius

Carter

Kilpatrick

Martin-Misener

Donald

Kaasalainen

DiCenso

(2010) The clinical nurse specialist role in Canada [Special issue]. Canadian Journal of Nursing Leadership 23: 140–166. doi:10.12927/cjnl.2010.22273.

Burnette

Horrow-Howell

Chen

(2003) Setting priorities for gerontological social work research: A national Delphi study. The Gerontologist 43: 828–838. doi: 10.1093/geront/43.6.828.

Buykx

Humphreys

Wakerman

Perkins

Lyle

McGrailand

Kinsman

(2012) Making evidence count: A framework to monitor the impact of health services research. The Australian Journal of Rural Health 20: 51–58. doi: 10.1111/j.1440-1584.2012.01256.x.

Canadian Nurses Association. (2009). Advanced nursing practice: A national framework. Retrieved from http://www.nanb.nb.ca/PDF/practice/ANP_National_Framework_e.pdf.

Canadian Nurses Association. (2013). Canadian Nurses Association says ‘It’s about time’. Retrieved from http://www.cna-aiic.ca/en/news-room/news-releases/2013/canadian-nurses-association-says-its-about-time-9.

Contandriopoulos

Lemire

Denis

J. L.

Tremblay

(2010) Knowledge exchange processes in organizations and policy arenas. A narrative systematic review of the literature. The Milbank Quarterly 88: 444–483. doi: 0.1111/j.1468-0009.2010.00608.x.

DiCenso, A., & Bryant-Lukosius, D. (2010). Clinical nurse specialists and nurse practitioners in Canada: A decision support synthesis. Retrieved from http://www.cfhi-fcass.ca/SearchResultsNews/10-06-01/b9cb9576-6140-4954-aa57-2b81c1350936.aspx.

DiCenso

Bryant-Lukosius

Martin-Misener

Donald

Abelson

Bourgeault

Harbman

(2010) Factors enabling advanced practice nursing role integration in Canada [Special issue]. Canadian Journal of Nursing Leadership 23: 211–238. doi:10.12927/cjnl.2010.22274.

Dobrow

M. J.

Costa

Israr

Chafe

(2010) Mapping health services and policy research settings in Canada: Following the money, the publications, and the interest. Healthcare Policy 6(2): 84–98.

10.

Donald

Bryant-Lukosius

Martin-Misener

Kaasalainen

Kilpatrick

Carter

DiCenso

(2010) Clinical nurse specialists and nurse practitioners: Title confusion and lack of role clarity [Special issue]. Canadian Journal of Nursing Leadership 23: 189–210.

11.

Ellen

M. E.

Lavis

J. N.

Ouimet

Grimshaw

J. M.

Bedard

P. O.

(2011) Determining research knowledge infrastructure for healthcare systems: A qualitative study. Implementation Science 6: 60doi: 10.1186/1748-5908-6-60.

12.

Grimshaw

J. M.

Eccles

M. P.

Lavis

J. N.

Hill

S. J.

Squires

J. E.

(2012) Knowledge translation of research findings. Implementation Science 7: 50doi: 10.1186/1748-5908-7-50.

13.

Health Canada. (2013, November). Canada health act. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/medi-assur/cha-lcs/index-eng.php.

14.

Kilpatrick

DiCenso

Bryant-Lukosius

Ritchie

J. A.

Martin-Misener

Carter

(2013) Practice patterns and perceived impact of clinical nurse specialist roles in Canada: Results of a national survey. International Journal of Nursing Studies 50: 1524–1536. doi: 10.1016/j.ijnurstu.2013.03.005.

15.

Lavis

J. N.

Boyko

J. A.

Oxman

A. D.

Lewin

Fretheim

(2009) SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Research Policy & Systems 7(Suppl. 1): S14doi:10.1186/1478-4505-7-S1-S14.

16.

Lavis

J. N.

Robertson

Woodside

McLeod

Abelson

(2003) How can research organizations more effectively transfer research knowledge to decision makers? The Milbank Quarterly 81: 221–248. doi: 10.1111/1468-0009.t01-1-00052.

17.

Lavis

J. N.

Ross

Hurley

Hohenadel

Stoddart

Woodward

Abelson

(2002) Examining the role of health services research in public policymaking. Milbank Quarterly 80: 125–154. doi: 10.1111/1468-0009.00005.

18.

Lomas

(2000) Connecting research and policy. ISUMA Canadian Journal of Policy Research 1: 140–144.

19.

Lomas

(2007) Decision support: A new approach to making the best healthcare management and policy choices. Healthcare Quarterly 10(3): 16–18.

20.

McBride

A. J.

Pates

Ramadan

McGowan

(2003) Delphi survey of experts’ opinions on strategies used by community pharmacists to reduce over-the-counter drug misuse. Addictions 98: 487–497. doi: 10.1046/j.1360-0443.2003.00345.x.

21.

Moscovice

Armstrong

Shortell

Bennett

(1977) Health services research for decision-makers: The use of the Delphi technique to determine health priorities. Journal of Health Politics, Policy and Law 2: 388–410.

22.

O’Brien-Pallas

Hayes

(2008) Challenges in getting workforce research in nursing used for decision-making in policy and practice: A Canadian perspective. Journal of Clinical Nursing 17: 3338–3346. doi: 10.1111/j.1365-2702.2008.02641.x.