An evaluation tool for assessing performance in priority setting and resource allocation: multi-site application to identify strengths and weaknesses

Abstract

Introduction

An evaluation tool should help improve formal priority setting and resource allocation (PSRA) processes in Canada and elsewhere. These are crucial to maximizing value from limited resources.

Methods

On the basis of case studies, balanced scorecard development protocols and use-focused evaluation principles, an evaluation tool was developed based on an existing framework for high PSRA performance and implemented in two health care organizations in British Columbia, Canada.

Results

Implementation of the tool identified areas of strength, improvement and weakness in the pilot organizations’ processes for PSRA including: communication, staff engagement and culture. Refinements were identified and incorporated into the tool for future application.

Conclusion

This is the first documented multi-site application of such an evaluation tool. Broader dissemination should have use both in further refining the basis of the tool and in catalysing improved performance of PSRA practice.

Keywords

evaluation priority setting resource allocation

Introduction

The pressures on publicly financed health care systems created by the growing costs of, and demand for, care have generated a sharpened awareness of the need for effective and explicit priority setting.^1,2

Klein defines priority setting as: ‘decisions about the allocation of resources between the competing claims of different services, different patient groups or different elements of care’.³ While this can occur in times of resource abundance where new funds are invested, the present public sector financial climate in many countries is forcing health care organizations to decrease their budgets and disinvest. There have also been renewed efforts to make resource allocation decisions more explicit to local populations in health care systems worldwide.⁴

In a 2011 survey of senior decision makers in Canadian health care organizations, 50% of respondents reported conducting resource allocation on the basis of historical patterns or political influences (rather than using formal approaches), and a minority (20%) conducted evaluation of their organizations’ priority setting and resource allocation (PSRA) processes.⁵ These organizations had less fair and less satisfactory PSRA processes as reported by survey participants – a connection supported by the broader literature.^1,5,6

While several studies have attempted to evaluate the quality of PSRA,^1,7,8 and there has been no documentation of multi-site implementation of a PSRA evaluation tool. Initial attempts to comprehensively evaluate resource allocation decisions have been undertaken by Sibbald et al.⁷ who applied a conceptual framework including ten criteria to a community hospital in Ontario. They suggested that ‘future research is required to determine the best combination of [evaluative] components’ as well as the best method for evaluation.⁷

In order to address the aims that Sibbald et al.⁹ laid out, we performed case studies of six Canadian health care organizations and combined these findings with existing literature to develop a framework of key elements for high performance in PSRA. This was designed to provide a means for organizations to ‘assess their current practices and determine areas for improvement based on evidence for best practice’.¹⁰ Participants in an earlier survey nominated organizations that they considered to be ‘high performers in priority setting and resource allocation’. These nominees were then vetted by an expert panel, and six organizations were chosen. Qualitative data from these sites were combined with a literature review to produce the high-performance framework. In addition to Sibbald et al.’s work,⁷ key models for priority setting that were drawn upon included Accountability for Reasonableness and Program Budgeting and Marginal Analysis.^11,12

In this paper, we describe the process of converting the high-performance framework into a tool and its implementation in two British Columbia (BC) health care organizations. Three questions guided the research:

Can an evaluation tool for achieving high performance in priority setting and resource allocation be developed from a conceptual framework describing elements of high performance?

Can this tool capture the strengths and weaknesses of a health care organization’s PSRA process?

What refinements are needed to improve the tool for future application?

Methods

Development of the evaluation tool

We followed the processes used in the development of another well-known assessment tool – the balanced scorecard – as a guide.^13,14 This approach was transferrable given: the similar goals of a balanced scorecard (to measure organizational performance) and this research (to create a tool that assesses an organization’s PSRA process), the structural similarities of the scorecard (with four perspectives) and the high-performance framework (with four domains), and the ability of the scorecard visually to translate the priorities of an organization to stakeholders (a desirable feature for the evaluation tool as well).

Performance indicators and possible measures were developed for each element within the high-performance framework. For example, ‘use of criteria to evaluate investment and disinvestment proposals’ would be an indicator for an explicit PSRA process. A measurement instrument (the evaluation tool) was created to collect data related to each element in the tool. A ‘use focused approach’ to evaluation that places the intended users’ perspective at the center of the evaluation (to ensure that action is taken as a result of the evaluation report), formed the framework for implementation.¹⁵ Based on this theoretical approach, investigators determined that qualitative methods would be the most appropriate method of data collection.

Each element and its respective indicators were turned into questions for a semi-structured interview. Across the instrument as a whole, a combination of direct (both positive and negative) and indirect (both positive and negative) questions were used.^16–18 Separate questionnaires were developed: for senior managers and lower level managers.

A draft of the questions were sent to experts in health care decision-making and implementation of priority setting frameworks for review. Revisions to language, simplification of questions and combination of similar concepts were carried out based on feedback.

Implementation of the evaluation tool

Implementation of the tool followed case study methods where a bounded system is examined and a detailed description emerges as the researcher analyses themes or issues and makes assertions about the case.^19,20 Here, the bounded system was the PSRA process of the selected organizations. Purposive sampling was used to select organizations that were familiar with the researchers and open to participating in an evaluation.²¹

To obtain a sample within each organization that reflected the diversity of the organization, interviewees were selected across geographical locations, departments, and (in organization #1) levels of success with submitting past proposals for investment and disinvestment as part of a formal PSRA initiative. In order to obtain this distribution of participants, a purposive-criterion sampling matrix was used (Table 1). The reader should note that the table below is meant to be illustrative and contains fictitious names and titles.

Table 1.

Sampling matrix.

	Submitted investment proposal			Submitted disinvestment proposal
	Name	Title	Location	Name	Title	Location
Approved	John S	Director	Location A	James D	CIO	Location G
	Alice C	Nurse Manager	Location C	Jane E	IT Manager	Location B
	Bill S	VP of Finance	Location E	Steph B	CEO	Location E
Not approved	Jason R	Clinician Manager	Location F	Dave D	VP Operations	Location E
	Jill V	Director	Location I	Doug L	Director	Location A
	Kent F	VP Planning	Location E	Mark J	Service Admin	Location I
No Submission	Kendra C	Chief Medical Health	Location B
	Michael K	Manager Home Health	Location G
	Ron D	Surgical Lead	Location F

Interviews were conducted by two of the authors, with each primarily responsible for one organization; however, several early interviews at each site were conducted jointly in order to ensure rigour of data collection. The semi-structured interviews lasted approximately 60 min with 45 min for the evaluation and 15 min for participant feedback on the tool. Interviews were audio-taped with permission.

In organization #1: Twenty-nine members of the organization were invited to participate and 27 were interviewed using the evaluation tool. Two members were unable to participate due to scheduling conflicts. Participants included 3 clinical leaders, 20 managers and 4 senior managers. There was representation from across geographical regions of the organization; clinical and non-clinical departments were represented. Participants included members who had submitted investment and/or disinvestment proposals (both accepted and rejected) in their previous PSRA cycle, as well as individuals who had not submitted proposals. The interviews were carried out with participants in person (n = 12), through videoconference (n = 13), and by telephone (n = 2).

In organization #2: Twenty-five members of the organization were invited to participate; one declined. Interviewees included 11 members of the senior management team, and 13 managers. The latter were equally divided between the organization’s two main service areas, with wide representation in terms of programme responsibilities. Twenty-two interviews were conducted in person, and two by telephone.

Data analysis began with template analysis in which the high-performance framework served as the ‘template,’ and responses from the interview transcripts were sorted into the elements of the framework.^9,22 Once categorized, data were examined in three ways using content analysis to determine whether each element was a ‘strength,’ ‘area of improvement,’ or ‘weakness’ of the organization’s PSRA process.²³

First, quotes in each element were coded as ‘positive’ or ‘negative’ based on language used (e.g. ‘excellent,’ ‘very good,’ ‘done very well’ vs. ‘not good,’ ‘poor,’ ‘totally lost on this’). Second, examples provided in quotes served as downstream indicators for certain elements (e.g. knowledge of resource allocation criteria by middle managers would be an example/downstream indicator of a successful training programme). Finally, descriptions of elements from the high-performance framework were used as reference points for determining the strength of elements.

Elements described by the majority of participants in a ‘positive’ way which agreed with descriptions in the high-performance framework and that had supporting examples, were categorized as ‘strengths.’ Negatively worded quotes related to a particular element that conflicted with the description in the high-performance framework and that were associated with negative examples were categorized as ‘weaknesses’. Elements were identified as ‘areas for improvement’ when their sub-elements formed a mix of strengths and weaknesses, or the negative language in the quotes was softer (e.g. ‘needs some improvement,’ ‘something we need to work on,’ ‘not where we want to be on this’).

To ensure accuracy of analysis and inclusion of a broader perspective, several strength and weakness determinations were initially carried out by two research team members independently and, subsequently, compared to test for agreement. All final determinations were discussed among the research team.

Refinement of evaluation tool

Refinement of the tool was informed in three ways. The first was by using investigator field notes collected during interviews including the ability of participants to comprehend questions and potential logistical issues with timing or running of interviews. In addition, 15-min debriefs were carried out at the end of each interview. These discussions provided an opportunity to question each participant directly about the content and delivery of the evaluation tool. Finally, observations based on the analysis of interview data were used to inform refinements (for instance, where the data obtained from the original evaluation questions appeared to be too ‘thin’ to support qualitative analyses).

Results

Tool development

All 19 elements of the high-performance framework were operationalized as described in the methods section to create the evaluation tool. Elements generally contained two to three questions. The evaluation tool in both the senior and middle manager forms is located in Appendix 1 and 2 (available online).

Tool implementation

Data were collected and analysed from each organization and a report was generated describing each organization’s current processes including: a dashboard of the framework for high performance representing the strengths, weaknesses and areas for improvement in the organization; in-depth descriptions of strengths and weaknesses; and recommendations for improvement. The dashboards for the organizations are shown in Figures 1 and 2 – strength = white, area for improvement = grey, weakness = dark grey.

Figure 1.

Site 1 Dashboard.

Figure 2.

Site 2 Dashboard.

Given the extensive amount of data produced in each case, this paper reports results from three elements of the tool in each implementation. These examples were selected from different domains to illustrate the scope of the concept of ‘high performance’ and to reflect the ability of the evaluation tool to capture performance at different levels, with the aim of showing the effectiveness of the tool.

Process: communication

In both organizations, this element was seen as a weakness, due to the preponderance of criticisms reported in the interviews. While there were some positive perspectives, most senior and middle managers believed that substantial improvements could be realized in this area.

Organization 1

In Organization 1, communication issues were raised in several aspects of the organization’s formal PSRA model including: initial messaging; process transparency; and feedback.

Initial messaging. When the pilot organization’s PSRA process began it was perceived as being ‘introduced actually as a disinvestment process, not an investment process.’ As a result, it did not provide the ‘confidence that was needed for [members of the organization] to buy into the process’ (Organization 1, Middle Manager).

Process transparency. Several middle managers, who were new to the organization’s PSRA process, reported similar experiences when submitting their proposals for investment or disinvestment without any idea when the proposal would be returned or what the general process was – ‘you send this thing off into the big black hole and they say basically… don’t call us, we’ll call you’ (Organization 1, Middle Manager). Participants also reported that some members experienced feelings of alienation and distrust.

“I’m not clear because I’m not - I’m not privy to the intricacies of how they’re determining or which criteria is used to determine when a successful proposal is or not. And be -- because I don’t know -- know that, then it makes me wonder why some are approved over others” (Organization 1, Middle Manager).

Feedback. There was virtually unanimous agreement among members of the organization that one of the weakest aspects of communication was the rationale for resource allocation decisions. Middle managers who created proposals reportedly did not receive ‘any communications back from the committee’ or have any way ‘to go back to the committee and say, why didn’t you fund [my proposal].’ (Organization 1, Middle Manager)

Organization 2

Communication was also an issue for respondents in Organization 2.

“Where I think there is a disconnect, is with the communication of how they arrive at the decisions, sometimes the people at the grassroots don’t quite understand all that”. (Organization 2, Middle Manager)

“The communication about what the decisions are – really good, but the communication about why those decisions were made – less so”. (Organization 2, Middle Manager)

From the senior management perspective, one challenge was the lack of formalized PSRA activity itself:

“Communication can be enhanced by having clear, transparent, formal processes, because that makes it easy to say what is going on … because we don’t really exactly have that, it is harder at times to communicate some of those things”. (Organization 2, Senior Manager)

Participants also suggested that the organization could make more use of different mediums for communication. Several expressed the desire for more face-to-face encounters. Some also noted that meetings were ‘forever cancelled or postponed’ (Organization 2, Middle Manager), which might be read, even if unintended, as a statement about the importance which senior managers assigned to such meetings: ‘It is almost universally cancelled and that sends a message’ (Organization 2, Middle Manager).

Structure: staff engagement

This element was coded as an area for improvement in Organization 1 and a weakness for Organization 2; this was the greatest degree of scoring difference observed between the organizations.

Organization 1

Staff participated directly in the resource allocation process by submitting proposals for investment or disinvestment. More intensive engagement occurred among staff who sat on the validation working group (VWG). This group consisted of middle and front line managers, and was headed by a senior manager. Each member received training in the organization’s PSRA process and in each budget cycle the group evaluated proposals using the organization’s assessment tool.

Soliciting input from physicians and support staff were highlighted as weak points in the organization’s engagement strategy. Several participants reported that ‘every manager has a different way of engaging their frontline staff’ and recommended greater standardization since some methods were ‘highly successful and doing extremely well and others [were] more “traditional”’ (Organization 1, Middle Manager). Lean – a quality improvement methodology – was mentioned as ‘another tool that really does help with engagement of front-line staff’ (Organization 1, Senior Manager).

Organization 2

In Organization 2, most respondents felt that there were relatively few, and inconsistent, opportunities for middle managers to contribute to organization-wide resource allocation decisions. The absence of a formal process appeared to limit staff participation. Consultation was often left to individual department managers, a cause of concern to some:

“There are some pockets that are really good, but others… Their idea of engagement is just telling people.” (Organization 2, Senior Manager)

The main way in which middle managers and other frontline staff might influence PSRA appeared to be via the organization’s Lean initiatives. ‘When we are setting priorities, we really engage our staff in that process. People participate in Lean and it is a wonderful tool for engaging your team’ (Organization 2, Senior Manager). Staff engagement is ‘mostly through Lean… and it is engagement around priority setting and decision making because the projects that we take on are all about our priorities and resources and how we use them effectively’ (Organization 2, Senior Manager). Middle managers generally concurred in this: ‘Lean lets staff feel involved’ (Organisztion 2, Middle Manager).

Attitudes/behaviours: culture of improvement

Interviews-related positive responses from both senior and middle managers related to the culture of improvement element. In both organizations this element was coded as an area of strength.

Organization 1

Senior managers reported a continual focus on the future and ensuring success of their formal PSRA model by ongoing refinement – both indicators of an improvement culture.

This year we’ve started the process earlier. Next year it’ll be better as well. We haven’t finished this year and we’re already talking about next year. So, as you see it’s getting better and better and it goes back to building it into the day-to-day activities as well, the expected way we do business. (Organization 1, Senior Manager)

These efforts seem to be recognized and felt by middle managers as they described a process that is ‘becoming more refined’ after multiple applications (Organization 1, Middle Manager). ‘We’re really a couple of years into this, but my experience this year was I think the process certainly ran smoother than it did last year, and I think people had a better understanding of it’ (Organization 1, Middle Manager).

Organization 2

Respondents very quickly and decisively answered questions about their organization’s culture of improvement in the affirmative, though without providing much detail. It was clear that most felt the organization to be highly supportive of learning and staff development. Many mentioned the organization’s willingness to fund managers to receive graduate training in Business Administration; some managers received exposure to models of formal health care priority setting, and its economic and ethical principles through this process. The organization did not offer internal training on the subject however. Most respondents also suggested that the organization had ample investment in decision support resources which were available to help prepare evidence-based cases and to assist in programme evaluation.

At the senior management level, efforts to improve PSRA were underway (e.g. development of explicit criteria) – and the willingness to participate in the development of the evaluative tool itself suggested a desire to advance and refine practice.

Tool refinements

Observations from interviews, analysis and interview debrief all informed efforts to revise the tool. In addition to noting a lack of time and some technical difficulties, the interviewers also observed that certain questions solicited shorter responses than others, perhaps due to difficulty with interpretation. To ensure that the tool was capturing a comprehensive picture of performance and to provide greater clarity for participants, the language of several interview questions was adapted and additional probing questions were added. Likert scales were also incorporated into many of the questions to facilitate responses.

Overlaps in content did take place during interviews. This was to be expected given the open-ended nature of the interview, and the strong relationships between elements of the high-performance framework used to create the evaluation tool itself. While no elements from the original framework were removed, analysis of participants’ responses led to the addition of two elements including a ‘programme budget’ sub-element in the Process domain and a ‘proposal quality’ element in the Outcomes domain.

Debriefs with participants post-evaluation were also useful for tool refinement. In general, participants felt that the evaluation tool was based on a comprehensive set of elements that captured the relevant aspects of a high-performing PSRA process. ‘I think there was a good global view and a good kind of global discussion. I think they’re (the elements of high performance discussed) all very pertinent’ (Organization 1, Middle Manager).

However, some participants suggested that certain elements could have been explored in greater depth had there been more time in interviews.

Post-evaluation debriefs also included asking interviewees whether any elements of high-performing PSRA processes were not covered. No suggestions were generated using this approach. Post hoc, investigators felt that this purpose would have been better served by asking interviewees what they thought the strengths and weaknesses of their organization’s process were before the interview. The responses given could then be compared against the existing tool to determine if any gaps existed.

In assessing the method of administering the evaluation tool, some participants expressed the view that ‘face-to-face is probably necessary’ (Organization 2, Senior Manager). Others felt that conducting every interview in this fashion might be too resource-intensive, and that video- or tele-conferencing would be viable alternatives. While a larger sample size could be achieved using surveys, respondents tended to suggest that they would not provide the same richness of responses.

Ensuring that interviewees are comfortable providing forthright responses is crucial for the effective evaluation of priority setting processes. Participants stated that they were comfortable sharing their organizations’ weaknesses in interviews. However, they did articulate mixed feelings as to whether they would be as comfortable sharing the weaknesses if the interviewer was someone internal to their organization.

Participant: The fact that you're not part of my organization is helpful. Interviewer: Do you think if it was someone from your organization that you’d give a different response? Participant: I’d probably be a little more guarded. I think some of those questions are delicate questions, especially when you start talking about alignment with strategic direction. (Organization 1, Middle Manager)

I like the idea of having someone external because then there is nothing to colour your views; I like the level of impartiality that adds … the external person makes it a lot more unbiased. (Organization 2, Middle Manager)

Discussion

Formal assessment of organization-wide PSRA is rarely conducted in Canadian health care organizations.⁵ The qualitative data from interviews in two organizations – the first multi-site application of an evaluation tool in the literature – confirms the applicability of the tool across health care organizations of different sizes and geographies. We have provided the tool as an appendix so that others may adopt and/or adapt it in different contexts with the intent of reporting on these experiences and thus continuing to expand this important area of research.

The tool proved to be sensitive to the conditions it attempted to assess. Strengths and weaknesses were identified with different levels and patterns of performance in each organization. Reports including colour-coded frameworks describing the strengths and weaknesses of each organization were delivered to senior management with recommendations for performance improvement. Both reports prompted in-depth discussion by senior leadership, and decisions to act on recommendations to improve their respective processes, which indicates that they were judged to be valid. This receptivity may be a result of the balanced scorecard protocol for development of the tool using visual methods to display performance and the ‘use focused approach’ to evaluation including the ‘users’ of the report in the evaluation.

The study also demonstrated that there are common challenges organizations face when setting priorities and allocating resources. Difficulties in communication and staff engagement have been identified in past studies.^8,24–26 Examples of good practice reported in the two organizations included a culture of improvement^24,27 and leadership²⁸ which is consistent with previous literature: ‘an organizational culture that is receptive to fostering change, and an openness to learning are all critical factors for the use of economists’ approaches to priority setting tools’.²⁹ This congruence with prior studies suggests a level of content validity in the current version of the tool. The fact that the two organizations shared struggles with other organizations also highlights the value of an evaluation tool that can be used to build collective knowledge of PSRA and facilitate improvement within and across organizations. Even partial use of this evaluative tool, to share results from the assessment of individual elements of priority setting, will provide more information about what contributes to effective resource allocation than would otherwise be known.

The tool was designed with open-ended questions to be administered via individual interviews. Other means of implementation are possible. Sibbald et al.⁷ suggested that to make the process of evaluation more streamlined, online surveys should be used rather than interviews – the rationale being that online surveys conducted by individuals internal to the organization would be more time and cost-effective. An online self-assessment tool should limit the number of open-ended questions since participants may be less likely to provide in-depth responses and would be likely to include more Likert style questions (e.g. how would you rate your understanding of your organization’s priority setting and resource allocation process? – 1–5 scale). Additionally, participant comfort (and perhaps engagement) could potentially be increased by third party administration and analysis of the data. Such a trade-off between usability and cost may be required in choosing an appropriate evaluation approach, but should be evaluated.

While application in two organizations is the largest documented application of a priority setting evaluation tool, it is still a small sample from which to reach definitive conclusions about the definition and valid, reliable measurement of performance in health care PSRA. This research is best seen as exploratory for others to apply the tool in different settings and jurisdictions. With wider dissemination, greater efforts could be made to measure the reliability and validity of the tool in varying contexts. Validity might be strengthened by engaging a broader group of stakeholders in the evaluation process such as junior managers, clinicians or patients.

Conclusion

Health care organizations across Canada are facing significant pressure to set priorities and make difficult funding choices in times of fiscal constraint. Improving this practice is thus crucial to encouraging transparency and maximizing the value from the limited resources available. Organizations, we suspect, are more likely to attempt evaluation of these processes when a tool is available. This research represents the first multi-site application of such a tool. While comparison across organizations is not yet possible, the responses from the two organizations studied here suggest that the tool could act as facilitator for improved PSRA. Broader dissemination should help to refine the tool, building a case for its greater validity and reliability, and most significantly catalysing discussions of how to improve PSRA practice.

Footnotes

Acknowledgements

The authors would like to acknowledge Dr Sam Sheps (Professor, School of Population and Public Health, UBC – Director, Western Regional Training Centre) for the support provided.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Canadian Institutes for Health Research (CIHR) through a Partnership for Health Systems Improvement (PHSI) grant.

References

Williams

Dickinson

Robinson

. Rationing in health care: the theory and practice of priority setting, Bristol; Chicago: Policy Press, 2012.

Donaldson

Mitton

. Priority setting toolkit: guide to the use of economics in healthcare decision making, London: BMJ Publishing Group, 2004.

Klein

. Rationing in the fiscal ice age. Health Econ Policy Law 2010; 5: 389–396.

Appleby

. The credit crisis and health care. BMJ 2008; 337: a2259–a2259.

Smith

Mitton

Bryan

. Decision maker perceptions of resource allocation processes in Canadian health care organizations: a national survey. BMC Health Serv Res 2013; 13: 247–247.

Gibson

Mitton

DuBois-Wing

. Priority setting in Ontario’s LHINs: ethics and economics in action. Healthc Q Tor Ont 2011; 14: 35–43.

Sibbald

Gibson

Singer

. Evaluating priority setting success in healthcare: a pilot study. BMC Health Serv Res 2010; 10: 131–131.

Peacock S. An evaluation of program budgeting and marginal analysis applied in South Australian hospitals. Melbourne Center for Health Program Evaluation, Monash University, 1998.

Smith N. High performance in healthcare priority setting and resource allocation: key elements (manuscript under review).

10.

Mitton

Dionne

Donaldson

. Managing healthcare budgets in times of austerity: the role of program budgeting and marginal analysis. Appl Health Econ Health Policy 2014; 12: 95–102.

11.

Daniels

. Accountability for reasonableness: establishing a fair process for priority setting is easier than agreeing on principles. BMJ 2000; 321: 1300–1301.

12.

Peacock

Mitton

Bate

. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy 2009; 92: 124–132.

13.

Kershaw

. Developing a BALANCED SCORECARD to Implement Strategy at St. Elsewhere Hospital. Manag Account Q 2001; 2: 28–35.

14.

Baker

Pink

. A balanced scorecard for Canadian Hospitals. Healthc Manage Forum 1995; 8: 7–13.

15.

Patton

. Utilization-focused evaluation, Thousand Oaks, CA: Sage, 2008.

16.

Questioning Techniques - Communication Skills Training from MindTools.com [Internet], http://www.mindtools.com/pages/article/newTMC_88.htm (accessed 20 June 2013).

17.

Questioning Techniques [Internet]. Gnosis Learning; 2009, http://www.gnosislearning.com/_document/Questioning+Techniques.pdf (accessed 8 August 2009).

18.

Questions and Questioning Techniques [Internet]. Yale Graduate School, http://www.yale.edu/graduateschool/teaching/forms/QuestionsandQuestioning.pdf (accessed 8 August 2013).

19.

Creswell

. Qualitative inquiry and research design: choosing among five traditions, London: Sage, 2007.

20.

Stake

. The art of case study research, London: Sage, 1995.

21.

Qualitative Research & Evaluation Methods [Internet]. Sage, http://www.sagepub.com/books/Book9906?prodId=Book9906 (accessed 21 October 2013).

22.

Cassell

Symon

. Essential guide to qualitative methods in organizational research, London; Thousand Oaks, CA: Sage, 2004.

23.

Holsti

. Content analysis for the social sciences and humanities, Reading, MA: Addison-Wesley Publ. Co., 1969.

24.

Reeleder

Goel

Singer

. Leadership and priority setting: the perspective of hospital CEOs. Health Policy Amst Neth 2006; 79: 24–34.

25.

Teng

Mitton

MacKenzie

. Priority setting in the provincial health services authority: survey of key decision makers. BMC Health Serv Res 2007; 7: 84–84.

26.

Robinson

Williams

Dickinson

. Priority-setting and rationing in healthcare: evidence from the English experience. Soc Sci Med 2012; 75: 2386–2393.

27.

Dionne

Mitton

Smith

. Decision maker views on priority setting in the Vancouver Island Health Authority. Cost Eff Resour Alloc CE 2008; 6: 13–13.

28.

Baker GR and Denis J-L. A comparative study of three transformative healthcare systems: lessons for Canada [Internet]. Canadian Health Services Research Foundation, http://www.cfhi-fcass.ca/sf-docs/default-source/commissioned-research-reports/Baker-Denis-EN.pdf?sfvrsn=0 (2011, accessed 5 May 2015).

29.

Peacock

Mitton

Ruta

. Priority setting in healthcare: towards guidelines for the program budgeting and marginal analysis framework. Expert Rev Pharmacoecon Outcomes Res 2010; 10: 539–552.