Systematic Quality Monitoring For Specialized Palliative Care Services: Development of a Minimal Set of Quality Indicators for Palliative Care Study (QPAC)

Abstract

Background:

A feasibility evaluation of a comprehensive quality indicator set for palliative care identified the need for a minimal selection of these indicators to monitor quality of palliative care services with short questionnaires for the patients, caregivers, and family carers.

Objectives:

To develop a minimal indicator set for efficient quality assessment in palliative care.

Design:

A 2 round modified Research ANd Development corporation in collaboration with the University of California at Los Angeles (RAND/UCLA) expert consultation.

Setting/Patients:

Thirteen experts in palliative care (professionals and patient representatives).

Measurements:

In a home assignment, experts were asked to score 80 developed indicators for “priority” to be included in the minimal set on a scale from 0 (lowest priority) to 9 (highest priority). The second round consisted of a plenary meeting in which the minimal set was finalized.

Results:

Thirty-nine of the 80 indicators were discarded, while 19 were definitely selected after the home assignment, and 22 were proposed for discussion during the meeting; 12 of these survived the selection round. The final minimal indicator set for palliative care consists of 5 indicators about the physical aspects of care; 6 about the psychosocial aspects of care; 13 about information, communication, and care planning; 5 about type of care; and 2 about continuity of care.

Conclusion:

A minimal set of 31 indicators reflecting all the important issues in palliative care was created for palliative care services to assess the quality of their care in a quick and efficient manner. Additional topic-specific optional modules are available for more thorough assessment of specific aspects of care.

Keywords

quality indicators palliative care minimal data set quality assessment quality improvement

Introduction

With a growing culture of quality assessment in palliative care (PC), one of the greatest challenges for specialized PC services is to evaluate their care systematically and continuously in order to improve it at the level of patients and their families.^1

-5 Quality indicators, that is, measurable aspects of care addressing a specific aspect of care, can be a helpful tool in achieving improvement and transparency at aggregated service level.^6,7 A number of internal quality indicators of palliative or end-of-life care have been developed.^8,9 However, these often focus only on patients with cancer or on specific dimensions for PC such as physical aspects of care.^10

-18 The Council of Europe has stated a need for quality indicators for PC that assess all relevant dimensions and hence reflect the multidisciplinary character of PC.³

In Belgium in 2009, we therefore started to develop a valid comprehensive quality indicator set and accompanying measurement procedure for PC¹⁹ to allow specialized PC services to evaluate the quality of their care delivery. In Belgium, these services include multidisciplinary palliative home care teams, PC units, and multidisciplinary palliative support teams in hospitals. We followed a scientifically rigorous method combining evidence and consensus (Figure 1) as suggested by Campbell et al for the development of quality indicators,²⁰ including:

Systematic literature review to identify existing indicators for PC.⁸

Identification of the important domains of PC (see Table 1) and the priority themes and indicators of quality of PC using a robust consensus methodology taking into account perspectives of relevant stakeholders, that is, patients, family carers, professional caregivers, and policy makers.²¹ This resulted in an extensive comprehensive set of structure, process, and outcome quality indicators for PC, partly composed of existing (validated) international indicators and partly of newly developed indicators (for the domains and themes that were not covered by existing indicators, following a Research ANd Development corporation in collaboration with the University of California at Los Angeles (RAND/UCLA) consensus method involving literature review and 2 expert panel rounds (this development process is described elsewhere^21,22).

Operationalization of the selected quality indicators into measurement instruments and methods. Five different questionnaires were developed²¹: (1) for living patients (or a proxy) receiving care from a specialized PC service, (2) for the most involved professional caregiver of these patients (nurse, physician, or psychologist of the team), (3) for bereaved family members of patients who died under care of a specialized PC service, (4) for the most involved caregiver of these patients (nurse, physician, or psychologist of the team), and (5) for the coordinator of the team (one-time only questionnaire for 4 structure indicators).

A feasibility study of the quality indicators and measurement procedure in 9 PC services in Belgium. This phase indicated good feasibility, usefulness, face validity, and discriminative power.²² Still the caregivers indicated that some changes in the measurement procedure were required and a desire to have fewer core indicators making the measurement and evaluation more feasible and better allowing them to focus on core working points for their team. The challenges for a final set, to be used in PC practice, was to be limited in terms of work-load, overload of information, and length of the questionnaires and to be feasibly measured and evaluated by the PC services themselves.

Table 1.

Overview of Representation of Domains of Quality of Care in the Comprehensive and Minimal Indicator Set.

	Number of indicators
Domains for quality of palliative care	CS	R1	MS
Physical aspects of care	11	5	5
Psychosocial and spiritual aspects of care^a	16	1	3
Communication and care planning with patient^a	20	4	7
Communication and care planning with family	6	2	3
Communication and care planning among caregivers^a	3	0	1
Type of care and circumstances surrounding death	6	2	5
Coordination and continuity of care	9	3	3
Care for family^a	9	2	4
Total	80	19	31

Abbreviations: CS, complete set; R1, round 1 of the RAND/UCLA panel (home assignment); MS, minimal set; RAND/UCLA, Research ANd Development corporation in collaboration with the University of California at Los Angeles.

^aThis domain was underrepresented in the provisional selection of the minimal set after round 1 (home assignment).

The aim of the current study was to provide PC services with a minimal core set of quality indicators for PC services and an accompanying measurement procedure for fast, efficient, and repeated comprehensive monitoring of the quality of their care delivery.

Methods

Design

In order to select a core minimal quality indicator set, a 2-round modified RAND/UCLA panel was conducted. This method combines scientific evidence with consensus among experts in the field.²³ Experts were asked to select a core set of quality indicators among the extensive set of indicators that were previously identified. In a first phase, the experts were assigned a home assignment to preliminarily score the indicators on priority (in this case to be part of a minimal set for PC); then a plenary discussion took place to reach consensus on the final selection of indicators.

Participants

The panel was composed of experts in the field of PC in Belgium. They were recruited from 3 areas: PC researchers, PC providers, and PC users. We balanced the number of experts per area according to the importance of that area. In total 13 experts participated: 1 researcher, 4 physicians, 4 nurses, 2 psychologists, 1 volunteer, and 1 representative of a patient organization.

Rating Process

During the development phase, the indicator set had been operationalized into 4 types of questionnaires (1 for the patient, most involved caregiver, family carer, and family physician). We predetermined that a maximum of 8 indicators should be measured per questionnaire for the minimal set. In this way, all relevant perspectives would be covered and the length of each questionnaire would correspond to 1 page, which had been identified by the caregivers as ideal. Hence, of the list of 80 previously developed and validated process and outcome indicators, a maximum of 32 (8 per respondent type) could be selected.

One month before the meeting, the panel members received this list of indicators, with descriptions of each indicator’s numerator, denominator, and measurement question(s) accompanied by a clear assignment and a document explaining the background and rationale for each indicator and results of the measurement in the feasibility study for each indicator. The experts were asked to score each indicator for priority to be included in the minimal indicator set on a scale from 1 to 9, with 1 equaling “lowest priority” and 9 equaling “highest priority.” Furthermore, they were asked to choose the 8 indicators with highest priority per questionnaire respondent type.

Analysis

After receiving all score sheets, the researcher calculated for each indicator the median score and the number of times the experts marked it with “highest priority.” The RAND/UCLA method prescribes that indicators with a median score of 7, 8, or 9 on which consensus was reached (ie, no more than 2 experts scored the indicator with a 1, 2, or 3) are retained immediately; indicators with a median score of 1, 2, or 3 on which consensus was reached (ie, no more than 2 experts scored the indicator 7, 8, or 9) are discarded immediately; all other indicators, that is, those on which consensus was not reached and those with a median score of 4, 5, and 6, are discussed with all experts during the meeting in order to reach consensus on the final selection. Following this method, 67 of the 80 indicators would have been retained immediately, exceeding the target number of 32. We therefore applied a stricter selection method: only the indicators with a median score of 7, 8, or 9 on which consensus was reached and marked by 7 or more panel members with highest priority were immediately selected for the minimal set. Indicators with a median score of 6 or less were definitely discarded. All other indicators scoring high (7, 8, or 9) but on which consensus was not reached or was marked by no more than 6 panel members with highest priority were discussed during the meeting.

Plenary Discussion Meeting

During a 1-day discussion meeting, the selection method and results of the home assignment were presented to the experts: which indicators per questionnaire respondent type were already selected for the minimal set and which still needed to be discussed and decided on. In order to preserve the coverage of all aspects inherent to PC, we informed the panel members which domains were yet underrepresented; they could take this into account though they were not obliged to select the indicators to complete the domains.

Measurement Procedure

The measurement procedure follows a snapshot approach: A cross-sectional inclusion method is used which requires the PC services to list 2 groups of patients on 1 given day (ie, the day of the assessment): all patients who are enrolled in the PC service are still alive on that specific day and all patients who were enrolled in the service have died in a given period. For the first group, a questionnaire is provided to the patient and the most important professional caregiver at that point in time and for the second group to the most important family carer and the most involved professional caregiver. We revised the procedure based on the results of the interviews that were part of the feasibility study.²² During these interviews with the caregivers who worked with the indicators, we analyzed the different steps in the measurement procedure: selection of the different respondents, sending out the questionnaires, and feedback of the indicator scores to the teams.

Ethical Concerns

In the recruitment letter for the experts, the confidentiality and anonymity of the research findings were guaranteed. As no personal information was collected from the experts, no informed consent was requested. The current study is part of a larger feasibility project for which the protocol was approved by the Ethical Review Board of Brussels University Hospital of the Vrije Universiteit Brussel (143201112708).

Results

In total, 13 experts participated in the expert consultation. All experts completed the home assignment; 11 of them were present at the plenary discussion meeting. Based on the first scoring round of the home assignment, 39 indicators were discarded immediately (consensus over low priority), 19 were selected immediately (consensus over highest priority), and 22 had to be discussed (no consensus). Of those, 13 could still be selected (in order not to exceed the targeted maximum of 32 indicators). At the end of the discussion, the minimal set for home and hospital PC in Belgium consisted of 31 process and outcome indicators: 5 about physical aspects of care; 3 about the psychosocial spiritual aspects; 11 about information, communication, and care planning; 5 about type of care; 3 about continuity of care; and 4 about care for family. Hence the minimal set covers all the original quality domains. An elaborated presentation of the minimal set including numerator, denominator, question, domain, and source can be found in Table 2. The other 49 process and outcome indicators were divided into 3 optional thematic modules: (1) biopsychosocial spiritual aspects, (2) communication and care planning, and (3) coordination of care. Finally, the 4 structure indicators (not involved in this study) were retained to be measured alongside the minimal set.

Table 2.

Presentation of the Minimal Indicator Set for Palliative Care in Belgium.^a

	Rater	Indicator description	Measurement question	Evidence of the indicator^b
		Physical aspects of care
1	Ca	Numerator: number of patients who were subjected to a general symptom assessment on a validated scale Denominator: total number of patients for whom this indicator was measured	Have you or any other professional carer evaluated the patient’s symptom burden since the admission or start of palliative care? □ Yes, with a scale □ Yes, without a scale □ No	Indicator developed by the researchers together with an expert panel Measurement question developed by the researchers together with an expert panel
2	Ph	Numerator: number of patients whose symptom burden was mostly or completely under control in the final week of life Denominator: total number of patients for whom a palliative care service was involved for at least 3 days	Did you have the feeling that the patient’s symptom burden (such as pain, anxiety, shortness of breath … ) was under control in the last week before death? □ Yes, totally □ Yes, largely □ No, not totally □ No, totally not □ I don’t know.	Indicator from Peruselli et al²⁴: “Symptom control in the last week of care” Measurement question by the researchers together with an expert panel
3	Ca	Numerator: number of patients who were subjected to a pain assessment, with or without pain scale Denominator: total number of patients for whom this indicator was measured	Have you or any other professional carer measured the patient’s pain since the admission or start of palliative care? □ Yes, with a pain scale □ Yes, without a pain scale □ No	Modified indicator from Twaddle et al²⁵: “pain measurement within 48 hours after admission” Measurement question developed by the researchers together with an expert panel
4	Pa	Numerator: number of patients who, after treatment, experienced significant improvement in pain Denominator: total number of patients treated for pain	In your opinion, did the treatment adequately relieve the pain? □ Yes, □ No	Modified indicator from Keay et al²⁶: “If patient had pain, this is followed to assess results of intervention and pain is reduced” Measurement question developed by the researchers together with an expert panel
5	Ca	Numerator: number of patients whose shortness of breath was relieved within 48 hours after admission or starting palliative care Denominator: total number of patients with shortness of breath	Was the patient suffering from shortness of breath at the time of admission or when palliative care was started? □ Yes □ No Did you or any other professional carer take action to relieve this patient’s shortness of breath after admission or after starting palliative care? □ Yes □ No How long after the patient was admitted or after palliative care was started was a relief of the shortness of breath achieved? □ (hours-days) □ No relief was achieved	Indicator from Twaddle et al²⁵: “Shortness of breath relieved within 48 hours after admission” Measurement questions developed by the researchers together with an expert panel
		Psychosocial aspects of care
6	Pa	Numerator: number of patients who indicated that the caregivers regularly assessed how they were feeling Denominator: total number of patients for whom this indicator was measured	Have your professional carers checked how you are feeling? □ Yes, regularly □ Yes, once □ No	Indicator developed through expert panel Measurement question developed by the researchers together with an expert panel
7	Pa	Numerator: number of patients with a score of 5 or more on a scale of 0 to 10 for quality of life Denominator: total number of patients for whom this indicator was measured	When you consider all aspects of your life—physical, psychological, social, spiritual, existential, and financial—your quality of life over the past 2 days was: (very bad) 0 ---------- 10 (excellent)	Indicator developed through expert panel Measurement question of “McGill quality of life questionnaire”²⁷
8	Pa	Numerator: number of patients who indicated that the caregivers most of the times or always respected their personal wishes and that they could most of the time or always plan their day and decide about the care provided^c Denominator: total number of patients for whom this indicator was measured	Do your professional carers give you the freedom to plan your day? □ Never □ Sometimes □ Usually □ Always □ Not applicable. Can you codecide about your care? □ Never □ Sometimes □ Usually Do your professional carers take your personal wishes into account? □ None of them do □ Some of them do □ Most of them do □ All of them do	Indicator from Claessen et al^10,28: “extent to which patients experience respect for their autonomy” Measurement questions from Claessen et al^10,28
		Communication and care planning with patient
9	Pa	Numerator: number of patients who received the right amount of information about their diagnosis Denominator: total number of patients for whom this indicator was measured	Are you getting enough information about the diagnosis? □ less than necessary □ just the right amount □ more than enough	Modified indicator from Claessen et al^10,28: “extent to which patients receive information about the expected course of the illness” Modified measurement question from Claessen et al^10,28
10	Pa	Numerator: number of patients who received the right amount of information about the course of the disease Denominator: total number of patients for whom this indicator was measured	Are you getting enough information about the course of the disease? □ less than necessary □ just the right amount □ more than enough	Modified indicator from Claessen et al^10,28: “extent to which patients receive information about the expected course of the illness” Measurement question from Claessen et al^10,28
11	Pa	Numerator: number of patients who received the right amount of information on palliative care options Denominator: total number of patients for whom this indicator was measured	Are you getting enough information with regard to end-of-life care? □ less than necessary □ just the right amount □ more than enough	Modified indicator from Claessen et al^10,28: “extent to which patients receive information about the advantages and disadvantages of various types of treatments” Modified measurement question from Claessen et al^10,28
12	Pa	Numerator: number of patients who indicated that they most of the time or always received clear and comprehensible and never or sometimes contradictory information Denominator: total number of patients for whom this indicator was measured	Do your professional carers explain things to you understandably? □ Never □ Sometimes □ Usually □ Always Do your professional carers give you conflicting information? □ Never □ Sometimes □ Usually □ Always	Modified indicator from Claessen et al^10,28: “extent to which patients indicate that they receive understandable explanations” and “extent to which patients indicate that they receive contradictory information” Measurement questions from Claessen et al^10,28
13	Ph	Numerator: number of patients with whom the physician discussed the care objectives Denominator: total number of patients for whom this indicator was measured	Have you ever talked to the patient about his or her wishes with regard to the care objectives? □ Yes, once □ Yes, several times □ No □ I don’t know	Indicator developed by researchers in cooperation with expert panel Measurement questions developed by the researchers together with an expert panel
14	Ca	Numerator: number of patients (or representatives) who were asked how they felt about end-of-life decisions and euthanasia Denominator: total number of patients for whom this indicator was measured	Have you or another professional carer inquired about the patient’s opinion with regard to end-of-life decisions? □ Yes, □ No Have you or another professional carer asked the patient’s opinion with regard to euthanasia? □ Yes □ No	Indicator from Claessen et al^10,28: “Extent to which relatives indicate that the patient was asked about her/his opinions with regard to end-of-life decisions” Measurement questions from Claessen et al^10,28
15	Fc	Numerator: number of patients who got a positive response to (all) their request(s) for a treatment or an end-of-life decision Denominator: number of patients who requested a treatment or an end-of-life decision	Indicate about which of the following treatment decisions your relative asked the doctor or another professional carer a question. □ Discontinuing a life-prolonging treatment □ Increasing pain medication with a potentially life-shortening effect □ Administering medication with the intention of rendering your relative unconscious (palliative sedation) □ Administering life-ending drugs on your relative’s own explicit request (euthanasia) □ None of the above decisions Did the professional carers take each of these questions about a treatment decision seriously? □ Yes □ No	Indicator developed through an expert panel Measurement questions developed by the researchers together with an expert panel
		Communication and care planning with family
16	Fc	Numerator: number of family carers who received the right amount of information about the patient’s condition and treatments Denominator: total number of family carers for whom this indicator was measured	Did you receive information about your relative’s condition? □ less than needed □ the right amount □ more than needed Did you get information about the pros and cons of different treatments? □ less than necessary □ just the right amount □ more than enough	Indicator from Finlay et al.²⁹: “providers keep family informed about the patient’s condition and treatment” Modified measurement questions of Claessen et al^10,28 who also incorporated this indicator in their set
17	Fc	Numerator: number of family carers who received the right amount of information about the patient’s approaching death Denominator: total number of family carers for whom this indicator was measured	Did you get information about the impending death? □ less than necessary □ just the right amount □ more than enough	Modified indicator from Miyashita et al³⁰: “explanation to family of patient’s impending death” Measurement question of Claessen et al^10,28 who also incorporated this indicator in their set
18	Ca	Numerator: number of patients about whom a conversation about their care preferences took place between the professional caregivers and family carers in the first week after admission or start of palliative care Denominator: total number of patients for whom this indicator was measured	How long after admission or after palliative care was started, did the professional carers and family have a conversation about the patient’s care preferences? □ (hours-days) □ No conversation has taken place (yet) with the family about the care preferences	Modified indicator from Twaddle et al²⁵: “a conversation takes place with the family about the patient in the hospital to decide on the care preferences” Measurement questions developed by the researchers together with an expert panel
		Communication and care planning among professional caregivers
19	Ca	Numerator: number of patients about whom multidisciplinary consultations took place at least once a week about their care objectives Denominator: total number of patients for whom the palliative care service was involved for at least 1 week	How often, in the past month, was a formal multidisciplinary consultation meeting held for this patient between professional carers to discuss the care objectives and/or options in terms of palliative care? □ once □ Less than once a week □ Approximately once a week □ Almost daily □ No formal multidisciplinary meetings have taken place (yet)	Indicator developed through an expert panel Measurement questions developed by the researchers together with an expert panel
		Type of care and circumstances surrounding death
20	Fc	Numerator: number of patients who scored 5 or more on a scale of 0 to 10 for the quality of death according to family carers Denominator: total number of patients for whom this indicator was measured	Please indicate on a scale from 0 to 10 how you perceived your relative’s death? (restless death) 0 --------10 (peaceful death)	Indicator developed by the researchers together with an expert panel Measurement question developed by the researchers together with an expert panel
21	Ph	Numerator: number of patients who scored 5 or more on a scale of 0 to 10 for the quality of death according to physicians Denominator: total number of patients for whom this indicator was measured	Please indicate on a scale from 0 to 10 how you perceived the patient’s death? (restless death) 0 -------------- 10 (peaceful death)	Indicator developed by the researchers together with an expert panel Measurement question developed by the researchers together with an expert panel
22	Ph	Numerator: number of patients whose palliative care started at least 2 weeks before death Denominator: total number of patients for whom this indicator was measured	When did you start the palliative care for this patient? □ (dd/mm/yyyy) □ I don’t know. When did this patient start receiving support from the palliative team? (only if the respondent is the family physician) □ (dd/mm/yy) □ I don’t know	Modified indicator from Peruselli et al²⁴: “the palliative care services respond rapidly to the request of care” Measurement questions developed by the researchers together with an expert panel
23	Ph	Numerator: number of patients admitted more than once to the emergency room since admission or start of palliative care Denominator: total number of patients for whom this indicator was measured	Was this patient admitted to the emergency department more than once since admission or since the start of the palliative care? □ Yes □ No □ I don’t know	Modified indicator from Earle et al³¹: “proportion with more than one emergency room visit in the last 30 days of life” Measurement questions developed by the researchers together with an expert panel
24	Ph	Numerator: number of patients admitted to the intensive care unit since admission or start of palliative care Denominator: total number of patients for whom this indicator was measured	Was this patient ever taken to intensive care since admission or since the start of the palliative care? □ Yes □ No □ I don’t know.	Modified indicator from Earle et al³¹: “proportion admitted to the ICU in the last 30 days of life” Measurement questions developed by the researchers together with an expert panel
		Coordination and continuity of care
25	Ph	Numerator: number of patients who were in touch with their family physician on a weekly basis in the last 3 months before death (personally or by telephone) Denominator: total number of patients for whom this indicator was measured	Did you have contact with the patient in the last 3 months of his/her life? □ Yes, by telephone □ Yes, in person □ Yes, both by telephone and in person □ No How often did you see or hear from the patient in those last 3 months? □ Every day □ A few times a week □ Every week □ Every month □ Less than once a month □ I don’t know	Indicator developed by the researchers together with an expert panel Measurement question developed by the researchers together with an expert panel
26	Ca	Numerator: number of patients whose caregivers were given the care objectives and resuscitation status during or after admission or starting palliative care Denominator: total number of patients for whom this indicator was measured	Did your GP, the physician, or other professional carers inform you of the patient’s care objectives? □ Yes, □ No Did your GP, the physician, or other professional carers inform you of the patient’s resuscitation status? □ Yes, □ No	Indicator developed by the researchers together with an expert panel Measurement question developed by the researchers together with an expert panel
27	Ca	Numerator: number of patients for whom the prognosis, psychosocial symptoms, functional status, symptom burden, and documentation about care wishes were entered in the file within 48 hours after admission or starting palliative care Denominator: total number of patients for whom this indicator was measured	Which of the following information was noted in the records within 48 hours after admission or starting palliative care? □ Patient’s prognosis □ Patient’s psychosocial symptoms □ Patient’s functional status □ Patient’s general symptom burden □ Documentation with regard to the patient’s desired care and treatment	Indicator from Twaddle²⁵: “entering in file the prognosis, psychosocial symptoms, functional status and symptom burden within 48 hours after admission” Measurement question developed by the researchers together with an expert panel
		Care for family
28	Fc	Numerator: number of family carers who indicated that the caregivers regularly talked to them about what it meant to them to care for their ill next of kin Denominator: total number of family carers who cared for their ill next of kin	Did the professional carers talk to you about what it meant to you to care for your sick relative? □ Yes, regularly □ Yes, once □ No	Modified indicator from Munday et al.³²: “identifying and meeting the needs of informal carers, before the patient’s death” Measurement question from Claessen et al^10,28 who also incorporated this indicator in their set
29	Fc	Numerator: number of family carers who indicated that the caregivers regularly asked how they were feeling Denominator: total number of family carers for whom this indicator was measured	Did the professional carers ask how you were feeling? □ Yes, regularly □ Yes, a few times □ No	Modified indicator from Claessen^10,28: “Extent to which, according to the direct relatives, attention was paid to their own psychosocial and spiritual well-being” Measurement question from Claessen et al^10,28
30	Fc	Numerator: number of family carers who felt they had adequate support after the patient’s death (inclusive evaluation meeting) and were informed of the possibilities of after-care. Denominator: total number of family carers for whom this indicator was measured	Did you feel supported by the professional carers immediately after the death of your relative? Yes □ No □ Not applicable. Was there a farewell- or staff appraisal interview that looked back on the care and treatment that was provided? □ Yes □ No Were the possibilities of after-care pointed out to you after the death of your relative? □ Yes □ No	Indicators from Claessen et al^10,28: “Extent to which direct relatives felt supported by the caregivers immediately after the patient’s death,” “Extent to which direct relatives were informed about the possibilities of aftercare,” and “Extent to which a final conversation or discussion was held to evaluate the care and the treatment” Measurement questions of Claessen et al^10,28
31	Fc	Numerator: number of family carers who were given as much assistance as necessary with the care process Denominator: total number of family carers who needed assistance with the care process	Did you feel that the professional carers gave you all the help and support you needed to care for your relative? □ Yes, I received all the help I needed □ Yes, I received help but not enough □ No, although I tried to get more help □ No, but I did not ask for more help either □ I did not need any help	Indicator from Peruselli et al ³³: “Satisfaction with family about assistance of caregivers in the care process for the patient” Measurement question from the VOICES questionnaire³⁴

Abbreviations: GP, General practitioner; VOICES, Views of Informal Carers - Evaluation of Services. ^aAll indicators and questions were translated by an official bureau using the backward–forward translation from Dutch to English, in close cooperation with the researchers in order to ensure the correctness of the translation and the content of the indicators.

^b Source of the quality indicator and measurement questions that was used during the development trajectory: the indicators were either based on existing international indicators or developed by the researchers in cooperation with an expert panel when international indicators were lacking for that theme within palliative care; the measurement questions were either based on existing questionnaires or developed by the researchers in cooperation with an expert panel when specific questions retrieving the necessary information to calculate the indicators were lacking.

^c This indicator refers to the autonomy of the patient. In the opinion of the experts, autonomy refers to the freedom of the patient both related to medical decisions and to privacy and freedom to plan their time. Therefore, it was part of the domain “psychosocial aspects of care.”

Quality Indicators Per Respondent Type

Patients

Only 3 indicators were selected immediately so 5 of 6 needed to be selected during the discussion (Table 3). Because of the overlap between indicator number 7 “Percentage of patients who indicated that the professional caregivers were most of the time or always attentive to their personal situation and needs” and number 8 “Percentage of patients who indicated that the professional caregivers regularly assessed how they were feeling,” the experts decided to discard number 7 and include the other in the minimal set.

Table 3.

Scores and Decision Per Round of the Indicators Included in the Questionnaire for Patients.^a

	Indicator title	Homework score		Decision after homework	Decision after discussion
	Percentage of patients:	Median score	Most prior	Decision after homework	Decision after discussion
1	With a score of 3 or less on a scale of 0 to 10 (average over 3 days) for pain	6^b	4	Discard
2	Who were most of the time or always treated for pain	7^b	4	Discard
3	Who, after treatment, experienced significant improvement in pain	9	6	Select	Minimal set
4	With a score of 3 or less on a scale of 0 to 10 (average over 3 days) for anxiety	7^b	2	Discard
5	Who were most of the time or always guided for anxiety	6^b	4	Discard
6	Who, after treatment, experienced significant improvement in anxiety	7^b	5	Discard
7	Who indicated that the caregivers were most of the time or always attentive to their personal situation and needs	8	4	Discuss	Discard
8	Who indicated that the caregivers regularly assessed how they were feeling	8	5	Discuss	Minimal set
9	Who were most of the times or always offered counselling when they were not feeling well or feeling less well	8	3	Discard
10	With a score of 5 or more on a scale of 0 to 10 for psychological well-being	6^b	4	Discard
11	Who indicated that the caregivers regularly talked to them about spirituality and religion in an understanding way	7^b	2	Discard
12	With a score of 5 or more on a scale of 0 to 10 for spiritual well-being	6^b	2	Discard
13	With a score of 5 or more on a scale of 0 to 10 for social well-being	6^b	0	Discard
14	With a score of 5 or more on a scale of 0 to 10 for quality of life	8	8	Select	Minimal set
15	Who indicated that they could most of the time or always be alone and talk to someone undisturbed if wanted	7^b	2	Discard
16	Who indicated that the caregivers most of the times or always respected their personal wishes and that they could most of the time or always plan their day and decide about the care provided	9	8	Select	Minimal set
17	Who indicated that most or all of their caregivers respected their philosophy of life	7^b	2	Discard
18	Who indicated that a priest or spiritual counsellor was available	5^b	0	Discard
19	Who received the right amount of information about their diagnosis	8	6	Discuss	Minimal set
20	Who received the right amount of information about their prognosis	8	3	Discard
21	Who received the right amount of information about the course of the disease	8	5	Discuss	Minimal set
22	Who received the right amount of information on the advantages and disadvantages of treatments	8	2	Discard
23	Who received the right amount of information on palliative care options	8	5	Discuss	Minimal set
24	Who indicated that they most of the time or always received clear and comprehensible and never or sometimes contradictory information	8	4	Discuss	Minimal set
25	Who indicated that they discussed their wishes about care objectives with a physician	8	3	Discard
26	Who are not receiving treatments they do not want	8	3	Discard
27	Who are not receiving drugs they do not want	6^b	0	Discard
28	Who were most of the time or always able to have some privacy with their family carers if wanted	6	2	Discard
29	Who believed that their caregivers were sufficiently informed of their medical care history to be able to provide the best possible care	6^b	0	Discard
30	Who most of the time or always felt they were given continuous care by their various caregivers	8	1	Discard
31	Who mostly or entirely knew who to talk to about their care	7	2	Discard

^aMedian score: on a scale from 1 to 9. Most prior: number of times found most prior by the experts.

^bThe experts did not reach consensus on these indicators in the first scoring round (home assignment).

Professional caregivers

Experts still needed to select 4 of 7 indicators during the meeting (Table 4). They decided to retain indicator number 8 “Percentage of patients who were asked how they felt about end of life decisions and euthanasia.” Furthermore, they choose number 16 “Percentage of patients whose caregivers were given the care objectives and resuscitation status during or after admission or starting PC,” 1 of the 3 indicators on transfers. Finally, they selected indicators number 9 and 10 on communication and care planning.

Table 4.

Scores and Decisions Per Round of the Indicators Included in the Caregivers Questionnaire.^a

	Indicator title	Homework score		Decision after homework	Decision after discussion
	Percentage of patients:	Median Score	Most prior	Decision after homework	Decision after discussion
1	Who were subjected to a general symptom assessment on a validated scale	8	7	Select	Minimal set
2	Who were subjected to a pain assessment, with or without pain scale	8	9	Select	Minimal set
3	Who were subjected to a delirium assessment, with or without delirium scale	7	4	Discard
4	Treated for delirium	6	2	Discard
5	For whom the delirium improved substantially or completely after the treatment	6	1	Discard
6	Whose shortness of breath was relieved within 48 hours after admission or starting palliative care	8	7	Select	Minimal set
7	Who were subjected to an anxiety assessment, with or without anxiety scale	8	6	Discuss	Discard
8	Who were asked how they felt about end of life decisions and euthanasia	8^b	8	Discuss	Minimal set
9	About whom a conversation about their care preferences took place between the caregivers and family carers in the first week after admission or start of palliative care	8	6	Discuss	Minimal set
10	Whom multidisciplinary consultations took place at least once a week about their care objectives	8	5	Discuss	Minimal set
11	Whose family physician was regularly or always involved in these multidisciplinary consultations	6	4	Discard
12	Whose wishes on resuscitation (DNR code) are known to the caregivers	7^b	6	Discard
13	Whose family physician, home nurse, and physiotherapist (when available) were involved in the care	7	4	Discard
14	Whose caregivers were given information about care and treatment prior to admission or starting palliative care	8	4	Discuss	Discard
15	Whose caregivers were given a summary of the patient’s medical records during the admission or after starting palliative care	8	5	Discuss	Discard
16	Whose caregivers were given the care objectives and resuscitation status during or after admission or starting palliative care	8	5	Discuss	Minimal set
17	For whom the prognosis, psychosocial symptoms, functional status, symptom burden and documentation about care wishes were entered in the file within 48 hours after admission or starting palliative care	8	7	Select	Minimal set

^aMedian score: on a scale from 1 to 9. Most prior: number of times found most prior by the experts.

^bThe experts did not reach consensus on these indicators in the first scoring round (home assignment).

Family carers

Five indicators were selected and 7 still needed to be discussed after the home assignment (Table 5). The experts selected indicator number 19 on quality of death and number 21 on after care. The last indicator number 23 “Percentage of family carers who were given as much assistance as necessary with the care process” was retained, as it was seen as covering a wide range of aspects.

Table 5.

Scores and Decision Per Round of the Indicators Included in the Questionnaire for Family Carers.^a

	Indicator title	Homework score		Decision after homework	Decision after discussion
	Percentage of:	Median score	Most prior	Decision after homework	Decision after discussion
1	Patients for whom the delirium improved sufficiently after the treatment	7	0	Discard
2	Family carers who indicated that caregivers were most of the time or always attentive to their anxiety and despondency	8	2	Discard
3	Family carers who indicated that the caregivers regularly talked to them about what it was meant to them to care for their ill next of kin	8	7	Select	Minimal set
4	Family carers who indicated that the caregivers regularly asked how they were feeling	8	10	Select	Minimal set
5	Family carers who were most of the time or always offered counselling when they were not feeling well or feeling less well	8	3	Discard
6	Family carers who indicated that the caregivers regularly talked to them about spirituality and religion in an understanding way	6	0	Discard
7	Family carers who indicated that most or all of the caregivers respected their philosophy of life	7	1	Discard
8	Patients who got a positive response to (all) their request(s) for a treatment or an end-of-life decision	9	11	Select	Minimal set
9	Patients who received the right amount of information about the treatment or end-of-life decision they requested	8	3	Discard
10	Patients who were involved in the final treatment or end-of-life decision (after a request of the patient)	8	2	Discard
11	Patients (or family carers in case the patient lacked capacity) who were sufficiently informed when a treatment or end-of-life decision was taken	8	4	Discuss	Discard
12	Patients (or family carers where the patient lacked capacity) who were involved in the treatment or end-of-life decision (without a request of the patient)	8	3	Discard
13	Family carers who received sufficient information about palliative lump sum and palliative leave	8	2	Discard
14	Family carers who received the right amount of information about the patient’s condition and treatments	8	9	Select	Minimal set
15	Family carers who received the right amount of information about the patient’s approaching death	9	7	Select	Minimal set
16	Family carers who received clear and comprehensible and noncontradictory information about the patient’s approaching death	8	4	Discuss	Discard
17	Patients for whom starting or terminating administration of artificial fluids and/or nutrition was discussed with family carers	7^b	4	Discard
18	Patients whose authorized representative was involved when the patient lost capacity	8	5	Discuss	Discard
19	Patients who scored 5 or more on a scale of 0 to 10 for the quality of death	8	5	Discuss	Minimal set
20	Family carers who were most of the time or always able to have some privacy with the patient if wanted	7^b	0	Discard
21	Family carers who felt they had adequate support after the patient’s death (inclusive evaluation meeting) and were informed of the possibilities of after-care.	8	5	Discuss	Minimal set
22	Family carers who were given as much assistance and training as necessary for practical issues and had someone available for help with these issues	8	6	Discuss	Discard
23	Family carers who were given as much assistance as necessary with the care process	8	4	Discuss	Minimal set

^aMedian score: on a scale from 1 to 9. Most prior: number of times found most prior by the experts.

^bThe experts did not reach consensus on these indicators in the first scoring round (home assignment).

Physicians (for the after-death questionnaire)

As the physicians’ questionnaire originally included only 9 indicators, experts were not asked to prioritize the 8 most important in this list. Based on the median scores only, 7 indicators were selected for the minimal set. The experts decided not to add any more to the minimal set, as there was no agreement on the validity of the remaining 2 indicators (Table 6).

Table 6.

Scores and Decisions Per Round for the Indicators Included in the Questionnaire for Physicians.^a

	Indicator title	Homework score	Decision after homework	Decision after discussion
	Percentage of patients:	Median score	Decision after homework	Decision after discussion
1	Whose symptom burden was mostly or completely under control in the final week of life	9	Select	Minimal set
2	With whom the physician discussed the care objectives	9	Select	Minimal set
3	Who were not started on artificial fluids and/or nutrition in the last month before death	7^b	Discuss	Discard
4	Whose palliative care started at least 2 weeks before death	8	Select	Minimal set
5	Who were not started on a new course of chemotherapy after admission or start of palliative care	7^b	Discuss	Minimal set
6	Admitted more than once to the emergency room since admission or start of palliative care	8	Select	Minimal set
7	Admitted to the intensive care unit since admission or start of palliative care	7	Select	Minimal set
8	Who scored 5 or more on a scale of 0 to 10 for the quality of death (according to family carers and physicians)	9	Select	Minimal set
9	Who were in touch with their family physician on a weekly basis in the last 3 months before death (personally or by telephone)	7	Select	Minimal set

^aMedian score: on a scale from 1 to 9. Most prior: number of times found most prior by the experts.

^bThe experts did not reach consensus on these indicators in the first scoring round (home assignment).

Measurement procedure

Based on the conclusions drawn from the feasibility study, 3 major changes were made to the measurement procedure: For measurement of quality indicators after death (family carer and physician questionnaire), the period of inclusion was changed from between 6 weeks and 6 months after death to between 4 weeks and 4 months after death; PC teams will collect the completed questionnaires themselves (instead of the research team) and deliver them afterward to the researchers for processing purposes. Three more steps were added to the procedure in order to embed the quality measurements into a continuous quality assessment trajectory: interpretation of results by the team, taking action to improve quality of care, and planning of the next measurement period (approximately 6 month later). This resulted in an optimized 6-step procedure¹⁹: Services appoint a coordinator (step 1) who makes the selection of all respondents (step 2). Anonymized responses are processed by an independent research team that calculates the quality indicator scores and feeds them back to the service in the form of a report (step 3). The care team members should then plan to meet to interpret the scores, distil working points, and set up an action plan to improve quality of care (step 4 and 5). In principle, services need to repeat the measurement and evaluation procedure every 6 months (step 6).

Discussion

Using a modified RAND/UCLA method, our study identified an operational minimal quality indicator set for PC services covering all aspects inherent to PC and an accompanying measurement procedure. As such we are able to present a comprehensive quality indicator set suitable for quick and efficient assessment of quality of care provided by palliative home care teams, PC units, and palliative support teams in hospitals.

We started from a comprehensive indicator set that was identified and tested following rigorous scientific methods.^21,22 The RAND method used to select the minimal indicator set and consisting of a homework assignment and a panel discussion is the most appropriate design to determine the extent of agreement among experts on a specific issue and is particularly useful in developing and identifying quality indicator sets.³⁵ The panel consisted of caregivers from different disciplines (ie, physicians, psychologists, and nurses from home and hospital settings) together with 2 experts representing the perspective of patients and their families.³⁶ However, some limitations of the study have to be acknowledged. Two experts who participated in the home assignment were not able to attend the expert meeting as a result of their uncertain working schedules. An analysis of their scoring results indicates that their absence likely would not have influenced the ultimate result as their scores in the homework assignment were similar to those of other panel members. Additionally, most panel members scored all indicators high for priority. As we anticipated this problem, we also asked the individual experts to pick the 8 most important indicators in order to differentiate more between the indicators. A possible solution for the high scoring propensity can be to define every point on the scale very precisely and to provide examples of how to score the indicators for priority.

At this time, all PC services in Flanders, Belgium, are invited to work voluntarily with the minimal quality indicator set and optional thematic modules in order to monitor their practice. By avoiding obligatory use of the indicators, caregivers will be highly motivated to perform the quality measurement systematically and will be more likely to apply the quality information in a well-considered manner. Palliative care teams should never be expected to reach certain levels of indicator scores. This might lead to unnecessary changes in care processes and lack of consideration for what is best for the patient and the family. The quality indicator set in its present form is designed for the internal monitoring of PC services. Still, measuring quality does not necessarily improve it. Therefore, the quality indicators need to be embedded into an improvement trajectory for which the current measurement procedure provides a good basis. Furthermore, the quality indicators may provide a useful tool for hospitals which want to measure quality of PC systematically and hence to be in line with the current requirements of accreditation programs. When implementing the minimal indicator set at wider scale, the usefulness and feasibility would need to be further evaluated and validated before it can be used as a tool to evaluate improvement programmes.³⁷ Therefore, the implementation study may provide useful information on the validity of the indicators and accompanying questions: When deemed necessary in the evaluation indicators as well as questionnaires need to be adapted based on other more recently developed and validated instruments that were developed to monitor the quality of PC.^13,38,39

Conclusion

In identifying and operationalizing a minimal set of quality indicators and accompanying measurement procedure, we have provided PC services with an indicator tool for quick and efficient assessment of the quality of the care they deliver in Belgium. Systematic measurement (every 6 months) with these indicators can generally identify the aspects of care that are going well and those that need improvement, hence serving as a barometer for the PC delivered by the specific service. Depending on the priorities different PC services set, action can be taken by the team to improve their care. Further evaluation of the usefulness and feasibility of the minimal indicator set is warranted once it has been implemented on a wider scale.

Footnotes

Acknowledgments

We thank all the panel members for their time and cooperation in the study. We thank Jane Ruthven for her help in language editing. This study is part of the “Flanders Study to Improve End-of-Life Care and Evaluation Tools (FLIECE-project),” a collaboration between the Vrije Universiteit Brussel, Ghent University, the Katholieke Universiteit Leuven, Belgium, and VU University Medical Centre Amsterdam, the Netherlands. Joachim Cohen and Lieve Van den Block are postdoctoral fellows of the Research Foundation Flanders.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by a grant from the Flemish government agency for Innovation by Science and Technology (agentschap voor Innovatie door Wetenschap en Technologie) (SBO IWT nr. 100036), and by the Flemish League against Cancer.

References

Aoun

Nekolaichuk

. Improving the evidence base in palliative care to inform practice and policy: thinking outside the box. J Pain Symptom Manage. 2014;48(6):1222–1235.

Aziz

Miller

Curtis

. Palliative and end-of-life care research: embracing new opportunities. Nurs Outlook. 2012;60(6):384–390.

Parliamentary Assembly of the Council of Europe. Resolution 1649 (2009): Palliative care: a model for innovative health and social policies. In: Directorate General I - Human Rights Directorate. Strasbourg: Texts of the Council of Europe on bioethical matters, Volume II. Council of Europe; 2014:68–70.

Lupu

Seow

. Progress towards systems of quality measurement that capture the essence of good palliative care. Palliat Med. 2012;26(4):291–293.

Evans

Harding

Higginson

. ‘Best practice’ in developing and evaluating palliative and end-of-life care services: a meta-synthesis of research methods for the MORECare project. Palliat Med. 2013;27(10):885–898.

Ostgathe

Voltz

. Quality indicators in end-of-life care. Curr Opin Support Palliat Care. 2010;4(3):170–173.

Strengthening of palliative care as a component of integrated treatment within the continuum of care, World Health Organization, 2014.

De Roo

Leemans

Claessen

. Quality indicators for palliative care: update of a systematic review. J Pain Symptom Manage. 2013;46(4):556–572.

Pasman

Brandt

Deliens

Francke

. Quality indicators for palliative care: a systematic review. J Pain Symptom Manage. 2009;38(1):145–156.

10.

Claessen

Francke

Belarbi

Pasman

van der Putten

Deliens

. A new set of quality indicators for palliative care: process and results of the development trajectory. J Pain Symptom Manage. 2011;42(2):169–182.

11.

deVos

Graafmans

Keesman

Westert

van der Voort

. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22(4):267–274.

12.

Eagar

Watters

Currow

Aoun

Yates

. The Australian Palliative Care Outcomes Collaboration (PCOC)–measuring the quality and outcomes of palliative care on a routine basis. Aust Health Rev. 2010;34(2):186–192.

13.

Kamal

Gradison

Maguire

Taylor

Abernethy

. Quality measures for palliative care in patients with cancer: a systematic review. J Oncol Pract. 2014;10(4):281–287.

14.

Lundstrom

Axelsson

Heedman

Fransson

Furst

. Developing a national quality register in end-of-life care: The Swedish experience. Palliat Med. 2012;26(4):313–321.

15.

Martinsson

Furst

Lundstrom

Nathanaelsson

Axelsson

. Registration in a quality register: a method to improve end-of-life care–a cross-sectional study. BMJ Open. 2012;2(4).

16.

Seow

Snyder

Shugarman

. Developing quality indicators for cancer end-of-life care: proceedings from a national symposium. Cancer. 2009;115(17):3820–3829.

17.

Wennekes

. Development and validation of quality indicators for cancer care[published online October 29, 2013]. Nijmegen. 2013.

18.

Woitha

Van

Ahmed

. Development of a set of process and structure indicators for palliative care: the Europall project. BMC Health Serv Res. 2012;12:381.

19.

Cohen

Leemans

. How can you prove that you are delivering good care? Monitoring the quality of palliative care using quality indicators. Eur J Palliat Care. 2014;21(5):228–231.

20.

Campbell

Kontopantelis

Hannon

Burke

Barber

Lester

. Framework and indicator testing protocol for developing and piloting quality indicators for the UK quality and outcomes framework. BMC Fam Pract. 2011;12:85.

21.

Leemans

Cohen

Francke

. Towards a standardized method of developing quality indicators for palliative care: protocol of the Quality indicators for Palliative Care (Q-PAC) study. BMC Palliat Care. 2013;12(1):6.

22.

Leemans

Deliens

Francke

Vander

Van den Block

Cohen

. Quality indicators for palliative care services: mixed-method study testing for face validity, feasibility, discriminative power and usefulness. Palliat Med. 2015;29(1):71–82.

23.

Fitch

Bernstein

Aguilar

. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corporation; 2001.

24.

Peruselli

Paci

Franceschi

Legori

Mannucci

. Outcome evaluation in a home palliative care service. J Pain Symptom Manage. 1997;13(3):158–165.

25.

Twaddle

Maxwell

Cassel

. Palliative care benchmarks from academic medical centers. J Palliat Med. 2007;10(1):86–98.

26.

Keay

Fredman

Taler

Datta

Levenson

. Indicators of quality medical care for the terminally ill in nursing homes. J Am Geriatr Soc. 1994;42(8):853–860.

27.

Cohen

Mount

Bruera

Provost

Rowe

Tong

. Validity of the McGill Quality of Life Questionnaire in the palliative care setting: a multi-centre Canadian study demonstrating the importance of the existential domain. Palliat Med. 1997;11(1):3–20.

28.

Claessen

SJJ

Francke

Sixma

. Op weg naar een CQ-index Palliatieve Zorg: meetinstrumentenontwikkeling. Utrecht, Netherlands: NIVEL; 2009.

29.

Finlay

Shreve

Casarett

. Nationwide veterans affairs quality measure for cancer: the family assessment of treatment at end of life. J Clin Oncol. 2008;26(23):3838–3844.

30.

Miyashita

Nakamura

Morita

Bito

. Identification of quality indicators of end-of-life cancer care from medical chart review using a modified Delphi method in Japan. Am J Hosp Palliat Care. 2008;25(1):33–38.

31.

Earle

Neville

Landrum

. Evaluating claims-based indicators of the intensity of end-of-life cancer care. Int J Qual Health Care. 2005;17(6):505–509.

32.

Munday

Mahmood

Dale

King

. Facilitating good process in primary palliative care: does the Gold Standards Framework enable quality performance? Fam Pract. 2007;24(5):486–494.

33.

Peruselli

Marinari

Brivio

. Evaluating a home palliative care service: development of indicators for a continuous quality improvement program. J Palliat Care. 1997;13(3):34–42.

34.

Addington-Hall

. VOICES (Vieuws of INformal Carers - Evaluation of Services). London, England: King’s College School of Medicine and Dentistry; 1998.

35.

Woitha

Van

Ahmed

. Validation of quality indicators for the organization of palliative care: A modified RAND Delphi study in seven European countries (the Europall project). Palliat Med. 2013.

36.

Kotter

Schaefer

Scherer

Blozik

. Involving patients in quality indicator development - a systematic review. Patient Prefer Adherence. 2013;7:259–268.

37.

Seow

Snyder

Mularski

. A framework for assessing quality indicators for cancer care at the end of life. J Pain Symptom Manage. 2009;38(6):903–912.

38.

Lorenz

O’Neill

. Cancer Quality-ASSIST supportive oncology quality indicator set: feasibility, reliability, and validity testing. Cancer. 2010;116(13):3267–3275.

39.

Schenck

Rokoske

Durham

Cagle

Hanson

. The PEACE Project: identification of quality measures for hospice and palliative care. J Palliat Med. 2010;13(12):1451–1459.