The Palliative Care Knowledge Questionnaire for PEACE: Reliability and Validity of an Instrument To Measure Palliative Care Knowledge among Physicians

Abstract

Background:

In Japan, a nationwide palliative care education program for primary palliative care (the Palliative care Emphasis program on symptom management and Assessment for Continuous medical Education: PEACE) was established in 2008. Effective delivery of such programs relies on adequate evaluations of program efficacy; however, such an instrument does not exist.

Objective:

This study aimed to develop and validate a measurement tool to quantify knowledge level of physicians about broader areas of palliative care, by which the effect of an education program could be measured.

Methods:

We conducted a cross-sectional, anonymous, self-administered questionnaire survey with a group of 801 conveniently sampled physicians in October 2010. To examine the test-retest reliability of items and domains, the questionnaire was reissued two weeks after the first survey was completed. This study used psychometric methods, including item response theory, intraclass correlation coefficients, and known-group validity.

Results:

The response rate was 54% (n=434). We included 33 items across the following 9 domains: (1) philosophy of palliative care, (2) cancer pain, (3) side effects of opioids, (4) dyspnea, (5) nausea and vomiting, (6) psychological distress, (7) delirium, (8) communication regarding palliative care, and (9) community-based palliative care. For these items, the intraclass correlation was 0.84 and the Kuder-Richardson Formula 20 (KR-20) test of internal consistency was 0.87. There was a significant difference in the scores between palliative care specialists and other physicians.

Conclusions:

We successfully validated a newly developed palliative care knowledge questionnaire to evaluate PEACE effectiveness (PEACE-Q). The PEACE-Q could be useful for evaluating both palliative care knowledge among physicians and education programs in primary palliative care.

Introduction

Palliative medicine has become an essential part of cancer care in the past 30 years.¹ To achieve high-quality palliative care, education for physicians is crucial, but most physicians worldwide including in Japan agree that current undergraduate and postgraduate programs do not provide sufficient education on palliative care.²

Recently, several countries established nationwide palliative care education programs.^3,4 In the United States, the Education for Physicians in End-of-life Care (EPEC^™) Project³ aimed to increase physician knowledge about palliative care, with 62% of the participants attaining improved knowledge. In Japan, the Palliative care Emphasis program on symptom management and Assessment for Continuous medical Education (PEACE) was established in 2008,^2,5 with more than 1000 PEACE education opportunities and more than 20,000 participating physicians reported throughout Japan from 2008 to 2010.⁶ To sustain the efficacy and significance of such programs, timely adjustment based on adequate and ongoing evaluations of the program is essential. Such evaluations should assess participant achievement and the effectiveness of the program among participants.

To date, the palliative care knowledge test (PCKT) was developed to measure the levels of knowledge about palliative care.⁷ The PCKT consists of five domains: philosophy, pain, dyspnea, psychiatric symptoms, and gastrointestinal symptoms. However, the PCKT was originally designed to quantify knowledge about palliative care among general health care professionals, especially nurses; and consequently, the reliability and validity of PCKT was not formally investigated in physicians. In addition, existing measurement tools to measure the knowledge of physicians in palliative care literature focus on pain management, and no instruments have been established to measure knowledge level of physicians about broader areas of palliative care.^8–10

The primary aim of this study was to develop and validate a measurement tool to quantify knowledge level of physicians about broader areas of palliative care, by which the effect of education programs could be measured.

Methods

We conducted a confidential and anonymous questionnaire survey among physicians, consisting of two phases: (1) instrument development and (2) psychometric analysis and selection of the 33 items for the final instrument. The ethical and scientific validity of this study was approved by the institutional review board of Saku Central Hospital. Consent to participate was indicated by the completion and return of the questionnaire.

Development phase

Initially, 12 palliative care specialists generated a total of 83 items for an item pool based on all modules of the PEACE program. Each such module includes nine domains and thus we designed our measurement instrument to include all these domains: (1) concept and philosophy of palliative care, (2) cancer pain, (3) side effects of opioids, (4) dyspnea, (5) nausea and vomiting, (6) psychological distress, (7) delirium, (8) communication, and (9) community-based palliative care.

To confirm the content validity, we adopted a modified Delphi method¹¹ with another group of 10 physicians (9 palliative care specialists and 1 psychiatrist). The experts evaluated the appropriateness of each item according to nine grades from 1 (inappropriate) to 9 (appropriate). If a participant rated the grade as less than 6, we asked him or her the reason. Items rated by eight or more members at 7 or more were selected; the remaining items were modified according to the members' opinions and then reevaluated. Following two such content-setting rounds we generated a provisional version of the instrument, the 79-item PEACE-Q (palliative care knowledge questionnaire for PEACE). Each question requires potential responses of right, wrong, and unsure.

Validation phase

The survey was carried out in October 2010 on 735 physicians from 15 conveniently sampled hospitals. The inclusion criterion was registered physicians whose specialty was not palliative medicine. In addition, to explore the known-group validity, 66 palliative care specialists were recruited. For this study, “palliative care specialist” was defined as a responsible physician in a certified palliative care unit belonging to Hospice Palliative Care Japan.

Procedures

The questionnaire with an accompanying cover letter was distributed to physicians by mail. Completed questionnaires were collected in a box specifically provided in each participating hospital. The survey for test-retest examination was conducted on the subjects who had consented to participate during the first survey in three institutions with two-week interval administrations. For the 79-item PEACE-Q, we then analyzed and compared physician backgrounds—i.e., age, gender, specialty, institutions, years of clinical experience, the number of terminally ill cancer patients seen in the past year, the number of cancer deaths per year, the number of patients they prescribed opioids to in the past year, and whether they participated in a PEACE program. No reminder or reward was used.

Statistical analysis and item selection

For analyses, ‘unsure’ responses were regarded as incorrect. First, to assess feasibility we calculated the percentage of missing data for each question; if the missing value accounted for more than 1% of all data, that item was regarded as inappropriate. Next, to assess sensitivity we calculated the percentage of correct answers for each question; if correct answers accounted for more than 90% of all data, that item was regarded as inappropriate. Third, to examine the test-retest reliability of each item, the kappa coefficients were calculated (cutoff: kappa coefficient of 0.3). Fourth, we estimated the difficulty and discrimination based on the two-parameter logistic Item Response Theory (IRT) model. IRT models are used for the statistical estimation of parameters that represent the magnitude of a latent trait attributable to the items. An advantage of IRT is that it potentially enables a researcher to improve the precision and reliability of an assessment.¹² We then determined the precision (cutoff: discrimination of 0.5). Finally, to confirm the content validity, the first 12 experts discussed the appropriateness of each item from both statistical and clinical viewpoints: (1) items in which sensitivity was likely to be higher and (2) items that could be viewed as providing lessons. Through this process, 33 items were selected to comprise the PEACE-Q.

Reliability and validity

Internal consistency was determined by calculating the Kuder-Richardson Formula 20 (KR-20) index. The test-retest reliability was explored by calculating intraclass correlation coefficients. To explore known-group validity, the unpaired t-test was used to determine a potential statistically significant difference in the total score of the PEACE-Q (33 items) and for each domain, between the palliative care specialists and the other physicians.

Statistical analysis was performed using statistical software SAS (SAS version 9.1; SAS Institute Inc., Cary, NC). The significance level was set at P<0.05 (two-tailed).

Results

Among 801 physicians included in the validation study, 434 responses (54%) were obtained and analyzed. For the test-retest investigation, 124 physicians agreed to participate and a total of 44 responses (35%) were obtained and analyzed.

The subject characteristics are summarized in Table 1. Approximately 70% were male, with an average of 16 years of clinical experience. Among them, 51% experienced more than 21 terminally ill cancer patients in the past year, while 17% cared for less than 5 cancer patients.

Table 1.

Characteristics of Participants (n=434)

	n	% ¹
Age, mean±SD	42±11
Gender
Male	306	70
Female	95	22
Specialty
Internal Medicine	121	28
Surgery	75	17
Resident	50	12
Gynecology	12	2.8
Urology	12	2.8
Anesthesiology	12	2.8
Other	106	24
Institution
University	37	8.5
Hospital over 400 beds	235	54
Hospital under 200 beds	47	11
Clinic	30	6.9
Other	43	9.9
Years of clinical experiences, mean±SD	16±10
Number of terminally ill cancer patients in the past year
None	20	4.6
1–5	54	12
6–10	37	8.5
11–15	33	7.6
16–20	18	4.2
21–	219	51
Number of patients prescribed opioids in the past year
None	49	11
1–5	75	17
6–10	68	16
11–15	40	9.2
16–20	23	5.3
21–	125	29
Number of cancer deaths per year
None	47	11
1–5	112	26
6–10	64	15
11–15	53	12
16–20	18	4.2
21–	89	21
Whether they participated in a PEACE program
Yes	92	21
No	301	69

The percentages do not add up to 100% due to missing values.

Item selection

Missing values totaled more than 1% in three items (1.2% each). However, we gave priority to content validity over psychometric properties based on the discussion among experts; thus, these three items were included in the PEACE-Q.

The percentage of correct answers ranged from 19% to 99% across the items, with the highest percentage of correct answers (99%) for, ‘Oral care should not be offered to the patients with nausea and vomiting, because mouth stimulation causes vomiting.’ Eighteen items showed a correct response rate of 90% or more, and these items were excluded. There were no items with 10% or less correct response rate.

The kappa coefficient in the test-retest reliability of each item ranged from −0.04 to 0.85, and was 0.3 or less for 14 items. However, we gave priority to content validity over psychometric properties based on the discussion among experts, and 6 out of these 14 items were included.

The results of the item analysis and IRT are shown in Table 2. The difficulty in all items ranged from −2.76 to 0.29, with a discrimination of 0.69 to 2.67. We determined that a subject who correctly answered an item with high discrimination would also have a high total score, whereas for an item with poor discrimination, the percentage of correct answers for that item would not relate to the total score.

Table 2.

Item Analysis and Item Response Theory Analyses of the PEACE-Q (N=434)

					IRT
Questions			Correct answer (%)	k-coefficient (n=44)	Difficulty	Discrimination
Philosophy of palliative care
1	Palliative care is synonymous with terminal care.	F	88	0.69	−2.76	0.77
2	In Japan, 50% or less of the general population believe that, if they have cancer, they want to feel secure about receiving cancer treatment and living without severe pain.	T	78	0.47	−1.54	0.97
3	The total consumption of opioids for pain is less in Japan than in the UK, Canada, and Germany.	T	85	0.42	−1.48	1.49
Cancer pain
4	When cancer pain is severe, one of the third-step drugs of WHO's Pain Relief Ladder is used as an initial analgesic.	T	53	0.48	−0.15	1.41
5	When opioids are initially prescribed, all non-opioid analgesics should be discontinued.	F	87	0.49	−1.26	2.37
6	Morphine is used safely in a patient with renal failure.	F	57	0.31	−0.32	1.20
7	The rescue dose of opioid is 5% of the total daily dose.	F	64	0.74	−0.45	2.22
8	Because the tolerance does not occur for opioid-induced nausea, an antiemetic should be prescribed for all patients.	F	83	0.43	−1.25	1.78
9	Total dose of daily opioids increases by 10% if pain is unpalliated.	F	55	0.85	−0.18	2.14
10	Opioid rotation or switching should be considered when it is difficult to increase the dose of opioids due to adverse effects.	T	87	0.68	−1.20	2.67
11	About 10% of the patients with controlled baseline pain have breakthrough pain.	F	49	0.58	0.00	2.57
12	Invasive dental procedures should be avoided during bisphosphonate treatment.	T	60	0.40	−0.68	0.69
Side effects of opioids
13	Opioid-induced nausea and/or vomiting occur in 80% or more of patients taking opioids.	F	50	0.36	−0.04	1.11
14	It is necessary to use a laxative together with oral opioids, because most patients who take opioids experience constipation.	T	89	0.09	−2.12	1.13
15	Opioids cause addiction in 0.2% or less of cancer patients under careful monitoring.	T	67	0.61	−0.59	1.73
Dyspnea
16	If a patient has dyspnea, the PaO₂ of the patient is under 60 Torr.	F	76	0.39	−1.18	1.23
17	Morphine is effective for dyspnea.	T	67	0.58	−0.59	1.79
18	If room temperature is maintained higher (hot), a patient with dyspnea often experiences relief.	F	41	0.47	0.29	1.52
Nausea and vomiting
19	The neurotransmitters in the vomiting center are dopamine, histamine, acetylcholine, and serotonin.	T	71	0.59	−1.05	1.03
20	When the primary cause of nausea is hypercalcemia, the administration of bisphosphonate is a useful treatment for alleviating nausea.	T	66	0.57	−0.53	1.99
21	Prochlorperazine sometimes causes akathisia.	T	67	0.73	−0.55	2.03
Psychological distress
22	When a patient has a high level of psychological distress, clinicians are recommended to examine whether the patient has suicidal ideation.	T	80	0.56	−1.87	0.84
23	When the patient has suicidal ideation, psychiatric consultation is recommended.	T	85	0.27	−1.95	1.04
24	An anxiolytic is one of the useful medications for patients with psychological distress.	T	86	0.34	−2.45	0.82
Delirium
25	Delirium occurs due to drugs or physical etiologies.	T	70	0.22	−1.28	0.73
26	Benzodiazepines should be used first for delirium.	F	52	0.26	−0.11	1.48
27	It is better to make the room pitch black for a patient with delirium, so that he or she can sleep well.	F	71	0.46	−1.05	1.03
Communication
28	An open-ended question means that it cannot be answered with a simple ‘yes’ or ‘no,’ and requires an unrestricted answer based on the subject's own feelings.	T	82	0.08	−1.57	1.15
29	When physicians convey bad news, they should ask the patient's concern and understanding about the disease.	T	88	0.25	−1.80	1.37
30	It is better to repeatedly use the word ‘cancer’ when telling the patient about his or her malignancy.	F	75	0.77	−1.58	0.78
Community-based palliative care
31	There is a consultation support center in all designated cancer centers.	T	56	0.38	−0.27	1.23
32	All terminally ill cancer patients 40 years of age can access long-term care insurance.	T	53	0.62	−0.17	1.16
33	All designated clinics with home hospice function have a 24-hour 7-day system.	T	73	0.37	−1.89	0.55

Based on the item analysis, IRT, and expert discussions, we determined 33 items for inclusion in the final version of PEACE-Q across the following 9 domains: (1) philosophy of palliative care, (2) cancer pain, (3) side effects of opioids, (4) dyspnea, (5) nausea and vomiting, (6) psychological distress, (7) delirium, (8) communication, and (9) community-based palliative care.

Reliability and validity

The KR-20 index of internal consistency was 0.87, and the intraclass correlation in the test-retest examination was 0.84. Regarding the known-group validity, there was a significant difference in the score for each domain as well as in the total score between palliative care specialists and other physicians (see Figure 1).

FIG. 1.

PEACE, Palliative care emphasis program on symptom management and assessment for continuous medical education.

Discussion

This study validated an instrument to measure the levels of knowledge about palliative care among physicians. This measurement tool enables us to evaluate the ongoing effectiveness of the PEACE program. The instrument showed good internal consistency, test-retest reliability, and known-group validity.

PEACE-Q would be useful for measuring the level of palliative care knowledge that all physicians engaged in cancer treatment should have. Furthermore, it could be used to measure the efficacy of undergraduate education programs about palliative care.

Previous measurement tools for quantifying the effect of palliative care education programs for physicians showed insufficient evaluation of reliability and validity,^13–16 and none other than the PCKT addresses a broad knowledge of palliative care.⁷ However, the psychometric properties of the PCKT were formally evaluated only among nurses. The PEACE-Q is a specific measure for physicians in both patient care and symptom management.

This study has several limitations. The tool was developed only for physicians who attend a PEACE-based seminar. In addition, several questions (2, 3, 31, 32, 33) are specifically about the Japanese medical system, and thus are not suitable for physicians in other countries; this limits the instrument transferability.

In conclusion, this study used psychometric methods to validate an instrument for evaluating palliative care knowledge among physicians who attend a PEACE-based seminar, which was developed as a nationwide education program in palliative care. This evaluation instrument was constructed to cover nine domains: philosophy of palliative care, cancer pain, side effects of opioids, dyspnea, nausea and vomiting, psychological distress, delirium, communication, and community-based palliative care. The PEACE-Q could be useful for evaluating both knowledge among physicians and education programs in primary palliative care.

Footnotes

Acknowledgments

The authors acknowledge financial support for the research, authorship, and/or publication of this article from a Health and Labor Sciences Research Grant for Clinical Cancer Research.

Author Disclosure Statement

The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

World Health Organization: Cancer Control Knowledge into Action: WHO Guide for Effective Programs in Palliative Care. World Health Organization: Geneva, 2007.

Tsuneto

. Past, present, and future of palliative care in Japan. Jpn J Clin Oncol, 2013; 43:17–21.

Katya

. Assessment of the education for physicians on End-of-Life Care (EPEC^™) project. J Palliat Med, 2004; 7:637–645.

Shvartzman

. Constructing a post-graduate palliative care curriculum: The Israeli National Palliative Care Training (INPACT) experience. J Palliat Care, 2011; 27:238–243.

Yamaguchi

. Recent development in the management of cancer pain in Japan: Education, clinical guidelines and basic research. Jpn J Clin Oncol, 2012; 42:1120–1127.

PEACE project (in Japanese). www.jspm-peace.jp.

Nakazawa

, Miyashita

, Morita

, Umeda

, Oyagi

, Ogasawara

. The palliative care knowledge test: Reliability and validity of an instrument to measure palliative care knowledge among health professionals. Palliat Med, 2009; 23:754–766.

Mortimer

, Bartlett

. Assessment of knowledge about cancer pain management by physicians in training. J Pain Symptom Manage, 1997; 14:21–28.

Furstenberg

, Ahles

, Whedon

. Knowledge and attitude of health-care providers toward cancer pain management: A comparison of physicians, nurses, and pharmacists in the state of New Hampshire. J Pain Symptom Manage, 1998; 15:335–349.

10.

Von Roenn

, Cleeland

, Gonin

. Physician attitude and practice in cancer pain management: A survey from the Eastern Cooperative Oncology Group. Ann Intern Med, 1993; 119:121–126.

11.

Fitch

. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: RAND, 2001.

12.

Streiner

. Health Measurement Scales: A Practical Guide to Their Development and Use, 4th. New York: Oxford University Press, 2008.

13.

Alvarez

. Systematic review of educational interventions in palliative care for primary care physicians. Palliat Med, 2006; 20:673–683.

14.

Montagnini

, Varkey

, Duthie

. Palliative care education integrated into a geriatrics rotation for resident physicians. J Palliat Med, 2004; 7:652–659.

15.

Okon

, Evans

, Gomez

, Blackhall

. Palliative educational outcome with implementation of PEACE tool integrated clinical pathway. J Palliat Med, 2004; 7:279–295.

16.

Chomiak

, Kramer

. Palliative care education: An intervention to improve medical residents' knowledge and attitudes. J Palliat Med, 2003; 6:391–399.