Discovering the hidden benefits of cognitive interviewing in two languages: The first phase of a validation study of the Integrated Palliative care Outcome Scale

Abstract

Background:

The Integrated Palliative care Outcome Scale is a newly developed advancement of the Palliative care Outcome Scale. It assesses patient-reported symptoms and other concerns. Cognitive interviewing is recommended for questionnaire refinement but not adopted widely in palliative care research.

Aim:

To explore German- and English-speaking patients’ views on the Integrated Palliative care Outcome Scale with a focus on comprehensibility and acceptability, and subsequently refine the questionnaire.

Methods:

Bi-national (United Kingdom/Germany) cognitive interview study using ‘think aloud’ and verbal probing techniques. Interviews were audio-recorded, transcribed verbatim and analysed using thematic analysis and pre-defined categories. Results from both countries were collated and discussed. The Integrated Palliative care Outcome Scale was then refined by consensus.

Setting/participants:

Purposely sampled patients from four palliative care teams in palliative care units, general hospital wards and in the community.

Results:

A total of 15 German and 10 UK interviews were conducted. Overall, comprehension and acceptability of the Integrated Palliative care Outcome Scale were good. Identified difficulties comprised the following: (1) comprehension problems with specific terms (e.g. ‘mouth problems’) and length of answer options; (2) judgement difficulties, for example, due to the 3-day recall for questions; and (3) layout problems. Combining the results from both countries (e.g. regarding ‘felt good about yourself’) and discussing them from both languages’ perspectives resulted in wider consideration of the items’ meaning, enabling more detailed refinement.

Conclusion:

Cognitive interviewing proved valuable to increase face and content validity of the questionnaire. The concurrent approach in two languages – to our knowledge the first such approach in palliative care – benefited the refinement. Psychometric validation of the refined Integrated Palliative care Outcome Scale is now underway.

Keywords

Patient-reported outcome measures outcome measurement cognitive interviewing palliative care

What is already known about the topic?

Outcome measures need to be specifically developed in palliative care populations to reflect the concerns of advanced illness and ensure minimal respondent burden.

The Palliative care Outcome Scale (POS) and Palliative care Outcome Scale–symptom module (POS-S) are brief, valid and reliable questionnaires developed for this population.

However, there is some overlap, and patient and clinician feedback indicates scope for further refinement.

Cognitive interviewing is recommended for questionnaire development but not adopted widely in palliative care research.

What this paper adds:

In response to patients’ and clinicians’ demands, we present a refined measure (Integrated Palliative care Outcome Scale (IPOS)) that has been shaped by the patients’ perspectives.

This study demonstrates once more the importance of cognitive interviewing in the refinement of outcome measures and that it can be done effectively in a palliative care population.

The concurrent use of cognitive interviewing in two languages – for the first time in palliative care – provided additional valuable information, resulted in wider consideration of the meaning and context of each item and led to a better overall refinement of the IPOS questionnaire.

Implications for practice, theory or policy

IPOS is now available for clinical use; it is one of the few brief measures validated in this population that covers all relevant domains.

The refined version of the IPOS is currently undergoing phase II of validation in Germany and the United Kingdom to demonstrate the psychometric properties.

Wider awareness and use of cognitive interviewing in measure refinement in palliative care is crucial, and cognitive interviewing in more than one language can have added value.

Background

Patient-reported outcome measures (PROMs) are questionnaires completed by patients to measure their perceptions of their functional status and well-being.¹ The Integrated Palliative care Outcome Scale (IPOS) is a PROM and a newly developed advancement of the Palliative care Outcome Scale (POS) which assesses symptoms and other palliative care (PC) concerns as perceived by the patient. There are versions for patients and healthcare professionals.

The POS was developed in 1999 with the involvement of patients and has been validated in a variety of settings: hospice in-patient care, day care, home care, hospital in-patient care and community and outpatient services.² It has been used to inform clinical practice, to monitor service interventions, to evaluate and improve the quality of care and as a research tool.^3,4 The POS team has currently endorsed 11 translations, and POS is being used in over 20 countries.^4,5 To describe the impact of PC-specific symptoms, a symptom module (the POS-S) was developed as an addition to the POS. Requests from clinical colleagues to merge POS and POS-S triggered the development of IPOS (see Figure 1). The IPOS is based on POS and POS-S, with changes to the wording of certain questions and response options. The 10 questions cover patients’ main concerns, common symptoms, patient and family distress, existential well-being, sharing feelings with family, information received and practical concerns.

Adopting a recommended quality criterion for questionnaires,^6,7 we aimed to involve patients in the questionnaire development through cognitive interviewing to ensure face and content validity and acceptability. Cognitive interviewing is increasingly used as a step in the refinement of survey questions and PROMs.⁸ A complex cognitive process takes place when research participants answer questionnaires. This includes comprehension of the question, retrieval of information from memory, judgement processes to estimate an answer and formulation of a response.^9,10 The European Organisation for Research and Treatment of Cancer (EORTC) guidelines for questionnaire development recommend cross-language pre-testing.¹¹ Accordingly, we aimed to base the IPOS refinement on views from German- and English-speaking patients and have the refined questionnaires available for concurrent psychometric validation in both countries.

Methods

Design

Cross-sectional, qualitative, bi-national and multicentre study enrolling purposively sampled PC patients in Germany and the United Kingdom, using cognitive interviews.

Setting and participants

In Germany, patients were recruited from a university hospital PC unit and its associated hospital support and community PC teams. UK patients were recruited from a university hospital support team. Inclusion criteria were as follows: age 18 years or above, capacity to give consent and sufficient fluency in written and spoken English or German language to answer the questionnaire. Exclusion criteria were as follows: being too unwell or distressed to participate as judged by the clinical team. Participants were purposively sampled according to the following criteria in order to achieve variation across key characteristics: gender, over and under 65 years, high, medium and low functional status as measured by the Australia-modified Karnofsky Performance Status (AKPS) scale¹² (see Table 1) and malignant/non-malignant main diagnosis. It was not aimed to have equal numbers in all cells, but to include participants with all characteristics. Screening took place at multidisciplinary team meetings or handovers, and a clinical team member asked whether the patients were interested and provided an information leaflet. Interested patients were approached by the researcher – except five who had deteriorated or been discharged. Of the 29 approached patients, 25 gave their informed consent.

Table 1.

Demographic and clinical details of participants.

		Germany	United Kingdom
Total number interviewed		15	10
Age (years)	Median, range	65 (22–85)	61 (43–83)
	⩽65	9	7
	>65	6	3
Sex	Female	10	7
Sex	Male	5	3
Nationality	German	14
	British		7
	Other	1	1
	Not recorded		2
	British		7
	Other	1	1
	Not recorded		2
Ethnicity	White British	Not recorded	7
	Black African		2
	Black Caribbean		1
Marital status	Married	7	6
	Partner	1	1
	Separated or divorced	2	2
	Widowed		0
	Single	5	0
	Not specified		1
Living situation	Alone	4	4
Living situation	With others	11	6
Working situation	Retired	Not recorded	4
	Working		5
	Unemployed		1
Primary diagnosis	Malignant	13	9
Primary diagnosis	Non-malignant	2 (cardiomyopathy and motor neuron disease)	1 (chronic obstructive pulmonary disease)
AKPS score	Median, range	60% (40%–100%)	52% (40%–70%)
	>70%	4	0
	>50%⩽70%	6	3
	⩽50%	5	7
Care setting	Palliative care unit	10
	Other hospital ward with support from hospital palliative care team	2	10
	Community	3

The Australia-modified Karnofsky Performance Status Scale (AKPS) grades functional status, for example, 70% represents ‘cares for self, unable to carry on normal activity or to do active work’ and 50% represents ‘requires considerable assistance and frequent medical care’.¹²

Cognitive interviews

Formal translation of the IPOS from English to German had previously been conducted using standard techniques.¹³ The interviews were conducted in the participants’ current care setting, that is, on the ward or in the participants’ home. Participants were asked to complete the questionnaire, using a combination of ‘think aloud’ and concurrent verbal probing techniques to elicit problems or comments regarding the completion.^14,15 ‘Think aloud’ techniques (uttering the thoughts during question answering) were demonstrated by the interviewer at the start of the interview. Probing techniques were both spontaneous, to explore verbal or non-verbal utterances like hesitations or irritation, and stand-ardised according to an interview guide (see Appendix 1). This was based on the four-stage question response model as described by Tourangeau¹⁰ (comprehension, retrieval, judgement and response formulation). Probes were developed for each of the four stages, and additional questions related to layout, missing aspects in the questionnaire and burden associated with its completion. The interviews were audio-recorded and transcribed verbatim.

Analysis of the interviews and refinement of the IPOS

Interviews were analysed by two independent researchers in each country using thematic analysis as described by Willis.¹⁴ In accordance with the interview guide, the transcript was also categorised following Tourangeau’s¹⁰ question response model (comprehension, retrieval, judgement and response formulation). Additional categories included the following: layout, acceptability/questionnaire burden and additional aspects. Results of all participants were compared and aggregated for each question and each category in both countries to identify the reasons for difficulties in answering the questionnaire and other emergent themes.

The German results were translated. Results from both countries were then collated and discussed between the research teams and the IPOS was refined accordingly, based on consensus. The German version was adapted according to the refined English version, taking German-language-specific results into account.

The study was approved by the relevant ethics committees (Ethikkommission of Ludwig-Maximilians-Universität Munich, reference number 169-13; National Research Ethics Service (London–Dulwich Committee) reference number: 13/LO/0573).

Results

Demographics

A total of 15 German and 10 UK interviews (16–54 min long) were completed between July 2013 and July 2014. The median age was 65 years in Germany (range 22–85 years) and 61 years in the United Kingdom (range 43–83 years). In Germany, 10/15 and in the United Kingdom, 7/10 participants were female. The predominant primary diagnosis was cancer. A third of the participants in Germany and two-thirds in the United Kingdom had an AKPS score of 50% or lower, which is defined by requiring ‘considerable assistance and frequent medical care’.¹² The majority of participants were in-patients. Details are shown in Table 1.

Findings from the interviews and IPOS refinement

The interviews in both countries demonstrated that for the majority of participants, most questions and answer options worked well. The identified difficulties were mainly comprehension problems and some pertained to judgement. No problems were identified with retrieval or response formulation. The interview results and the changes made based on them are shown in Table 2. The revised questionnaire can be found on the POS website.⁵

Table 2.

Issues regarding IPOS completion identified in the cognitive interviews in Germany and the United Kingdom and resultant changes made to the IPOS.

Prototype IPOS items	Prototype IPOS answer options	Aspects identified with supporting quotations	Revised IPOS items	Revised IPOS answer options
Q1. What have been your main problems or concerns at the moment?	Open question with three lines for answers	Good comprehension (18/25) of this question. Participants responded with personal problems as well as health-related problems2/15 German participants explicitly welcomed that the question asks for main problems AND concernsOh yes, that is good. Main problems is of course also associated with concerns. (Germany, male aged 55 years)3/10 of UK and 4/15 of German patients queried the openness of the question or were at first unsure about the layout with three free lines but consequently answered itCon … concerning what. (UK, female aged 65 years)	Q1. What have been your main problems or concerns over the past 3 days?	Open question with three lines for answers
Q2. Below is a list of symptoms which you may or may not have experienced. For each symptom, please tick one box that best describes how it has affected you over the past 3 days	Not at all I have not had this symptom in the last 3 daysSlightly Little or no effect on activities or concentrationModerately Some effect on activities or concentrationSeverely Marked effect on activities or concentrationOverwhelmingly Unable to think of anything else	3 days was not considered long enough by some patients (5/25)Why has it got to be the last three days? You know, like, in the last week, I’ve been really sick for five days, the last two days I’ve been alright. (UK, male aged 53 years)Judging severity of symptom over the time period of 3 days was sometimes difficult when people have had the symptom for one of the days or had a fluctuation of symptom severity (6/25)The subheadings in the answer options were confusing and/or too long (5/25)Poor appetite, well I mean ‘marked effect on activities or concentration’, how does it affect? But activities you just don’t wanna eat that’s all. So I can’t find the right one to tick. (UK, female aged 83 years)In contrast 2/25 of patients found that the explanations helped10/15 of German participants explicitly said the difference between response options was clear… that can be differentiated clearly. (Germany, male aged 22 years)8/25 of participants answered how severely they had the symptom rather than how it affected them	Q2. Below is a list of symptoms, which you may or may not have experienced. For each symptom, please tick one box that best describes how it has affected you over the past 3 days A 7-day version of IPOS is now also available	Not at all Slightly Moderately Severely Overwhelmingly
Pain		Good comprehension by all participantsOh, easy to understand. (UK, female aged 43 years)	Pain
Shortness of breath		Good comprehension by all participantsDifficulty in um not breathing as you normally do. (UK, female aged 47 years)	Shortness of breath
Weakness or lack of energy		10/15 of German participants said it was good that the question asks about weakness as well as lack of energy as they are different… they are very similar … but they are not the same. (Germany, female aged 49 years)2/25 of patients interpreted weakness or lack of energy as similar to drowsinessAnswer option wording ‘unable to think of anything else’ inappropriate, for example, for weakness (Germany, 1/15)	Weakness or lack of energy
Nausea (feeling like you’re going to be sick)		Good comprehension by all participantsWell nausea is you feel yuck, there’s something there and you’re likely to be sick … and vomiting is there is no doubt about it. (UK, female aged 83 years)4/15 of German participants described different types of nausea, and therefore found the explanation in brackets provided clarificationI sometimes feel sick but do not have the feeling that I have to vomit. (Germany, female aged 77 years)	Nausea (feeling like you are going to be sick)
Vomiting (being sick)		Overall good comprehension by participants (24/25)Vomiting is when you are actually sick. (UK, female aged 47 years)	Vomiting (being sick)
Poor appetite		Good comprehension by all participantsThat I sometimes do not feel like eating anything. (Germany, male aged 55 years)Answer option ‘Affects activity and concentration’ not appropriate for, for example, poor appetite (Germany, 1/15)	Poor appetite
Constipation		Good comprehension by all participants(regarding ‘poor appetite’) … is clear, clearly definable. Constipation, too. By different drugs, or caused by the chemotherapy. (Germany, female aged 66 years)2/10 of UK patients thought other bowel problems should be included here too	Constipation
Mouth problems		Too vague and unclear (5/25)Mouth problems? It’s funny they mention mouth problems, why would they ask about mouth problems? (UK, female aged 83 years)8/15 of patients in Germany did not necessarily think of dry mouth but of sore mouth/chewing/swallowing problems/tooth acheThe tongue is sore, there will be thrush or so, and with the teeth- they are not all own teeth – and so on there are problems with eating. (Germany, male aged 69 years)	Sore or dry mouth
Drowsiness		9/10 of UK patients had good comprehension of the word drowsiness, Um … when just … when you’re eyes are like going foggy and you just want to sleep … Oh it was easy to understand. (UK, female aged 43 years)2/15 of German patients were uncertain if it meant tiredness by day or by nightProblems going to sleep or problems sleeping through the night – or the drowsiness only relates to the daytime. (Germany, female aged 77 years)	Drowsiness
Poor mobility		15/15 of German participants had good comprehensionWell, if I can just stand up and walk away – or somehow really walk, without aids, let’s say. Now I cannot walk alone without the walking frame or the wheelchair. That is mobility for me. (Germany, male aged 69 years)Interpretation of mobility and weakness/lack of energy as similar to UK patients (3/10)	Poor mobility
Please list any other symptoms not mentioned above, and tick one box to show how they have affected you over the past 3 days.		2/15 of German participants commented that it was helpful having this question included and 4/10 UK patients wrote other symptomsYes, I find that good, because there may be things that are not listed. (Germany, female aged 65 years)	Please list any other symptoms not mentioned above, and tick one box to show how they have affected you over the past 3 days
Q3. Over the past 3 days, have you been feeling anxious or worried about your illness or treatment?	No, not at allOccasionallySometimesMost of the timeYes, always	Good comprehension by all participants	Over the past 3 days Q3. Have you been feeling anxious or worried about your illness or treatment?	Not at all OccasionallySometimesMost of the timeAlways
Q4. Over the past 3 days, have any of your family or friends been anxious or worried about you?		Variation in level of insight into family and friends worries and/or anxieties (3/25)That’s something I can’t really quantify because when I talk of family … you’re talking of a group of people … And different coping mechanisms how … I can’t quantify that. (UK, female aged 47 years)15/15 of German participants felt family/friends was clear; however, 2/10 UK patients were uncertain about who should be included in ‘family or friends’	Q4. Have any of your family or friends been anxious or worried about you?
Q5. Over the past 3 days, have you been feeling depressed?		8/10 of UK participants had a very good understanding of the word depressedTo be depressed … um … to be in darkness and see no light at all … And your whole being is black. (UK, female aged 70 years)4/15 of German participants had comprehension problems with the word ‘niedergeschlagen’ and made suggestions for alternatives	Q5. Have you been feeling depressed?in German version alternative term for ‘niedergeschlagen’: ‘traurig bedrückt’
Q6. Over the past 3 days, have you felt good about yourself as a person?	Yes, all the time.Most of the timeSometimesOccasionallyNo, not at all	Germany participants interpreted the term used in German mostly as feeling ‘physically well’ or ‘well in surroundings’, and if asked did not associate it with a spiritual aspect (9/15)For me feeling good would be standing up, walking … and playing tennis and normal living. … Relatively, I feel good, I do not have pain at the moment, but generally the situation is crap. (Germany, male aged 75 years)3/10 of UK participants found the question strange to be asked and confusingHow can you feel good about yourself when you’re lying in hospital … everything’s crap. (UK, 1007)Oh I understood the question but I just didn’t know what to put down … (UK, female aged 43 years)	Q6. Have you felt at peace?	Always Most of the time Sometimes Occasionally Not at all
Q7. Over the past 3 days, have you been able to share how you are feeling with your family or friends?	Yes, as much as I wantedMost of the timeSometimesOccasionallyNot at all with anyone	Good comprehension by 23/25 participantsYes … I’ve been able to share how I feel … Most of the time. (UK, female aged 65 years)Answer options caused confusion to 2/10 UK participants	Q7. Have you been able to share how you are feeling with your family or friends as much as you wanted?
Q8. Over the past 3 days, how much information have you and your friends been given?	Full information/as much as I wantedInformation given, most questions addressedInformation given at request, would have liked moreVery little information given, most questions not addressedNone at all, wanted more information	Questions and answer options considered by 6/25 patients to be too long or complicatedInformed ‘by whom’ and ‘about what?’ is missing (4/25)Question missed whether information was wanted or not (UK, 1/10) and whether information was understood at the right ‘level’ (UK, 1/10)1/10 of UK participants found it difficult to answer for both themselves and their family and friendsWell I don’t know if it was enough for my family I can’t answer for them. (UK, male aged 54 years)	Q8. Have you had as much information as you wanted?
Q9. Over the past 3 days, have any practical matters resulting from your illness, either financial or personal been addressed?	No problems/problems being addressedOccasional problems/problems being addressedSome problems/problems partly addressedSevere problems/problems not addressedOverwhelming problems/problems not addressed at all	Question and answer options confusing/difficult to comprehend (7/25)They’re very similar answers aren’t they … or is it my brain not working. (UK, female aged 43 years)‘Practical matters’ not ideal (Germany, 6/15)‘Problem’ in answer options inappropriate (‘funeral’ or ‘legacy’ is not a problem) (Germany, 2/15)	Q9. Have any practical problems resulting from your illness been addressed (such as financial or personal)?	Problems addressed/no problems Problems mostly addressed Problems partly addressed Problems hardly addressed Problems not addressed
Q10. How did you complete this questionnaire?	On my ownWith help from a friend or relativeWith help from a member of staff		Q10. How did you complete this questionnaire?	On my ownWith help from a friend or relativeWith help from a member of staff

IPOS: Integrated Palliative care Outcome Scale.

The figures refer to the number of participants who made these comments out of the total number of participants. The remaining participants did not make specific comments and had no difficulties with completion. Changes to the IPOS questions or answer options are typed in bold.

Comprehension

Patients were certain about their main problems and concerns (question 1), and many described these as the things that are always on your mind. Positive comments about the symptom list in question 2 included the following:

Oh, easy to understand. (United Kingdom, female aged 43 years)

and

I think question 2 is good, that it is all dealt with in a very ‘compact’ way. (Germany, female aged 25 years)

‘Pain’, ‘shortness of breath’, ‘nausea’, ‘poor appetite’ and ‘constipation’ listed in question 2 were well understood by all participants. Comprehension of question 3 (anxiety/worries) was also good for all participants.

Some comprehension problems were identified in both countries. First, difficulties arose with the wording of specific questions. For example, the term ‘mouth problems’ in question 2 was regarded as too vague, and in contrast to the intent of the question, many patients did not think of dry mouth when answering it. Furthermore, comprehension problems concerned answer options which were regarded as too long or complicated in question 2 (symptom list), question 8 (information received) and question 9 (practical matters). In addition, a third of patients in each country rated the severity of the symptom instead of the degree of being affected by it in question 2 (symptom list).

For some questions, interviewing in the two countries elucidated additional aspects that were important for the refinement of the IPOS: The question ‘Have you felt good about yourself as a person?’ (Q6) had been difficult to translate into German as there is no equivalent phrase that captures the same concept. The translation used for the interviews was understood by most participants as feeling physically well or feeling well in their surroundings. This complemented the comments of UK patients that this question was strange to be asked. A UK patient noted that question 8 (information needs) missed whether the received information was wanted. There were cases when comments made by the German patients diverged with those from the UK patients: Some UK patients interpreted ‘weakness and lack of energy’ and ‘mobility’ as similar, whereas all German patients had a clear understanding of ‘mobility’. Some German patients were unsure about the meaning of ‘drowsiness’, while UK patients had a good understanding of the word. In addition, there were a few translation-related difficulties commented on by the German participants, for example, comprehension problems of the German word used for ‘depressed’.

Judgement

A judgement problem was identified in both countries due to the 3-day time period over which symptoms are sought: some participants said they were unsure how to answer a question when they had a fluctuation of symptom severity; however, all answered the questions. Five participants commented that they would prefer a longer time frame, for example, 7 days. In addition, three participants said that judging the level of insight and anxiety of family and friends was difficult, but all answered the question.

Acceptability/questionnaire burden, missing aspects and layout

Overall, acceptability of the measure was good. Spontaneous comments included

Well I think it’s a good questionnaire I really do … it lets everybody like staff that they’ve only got to look at it and they know how you feel. (United Kingdom, female aged 74 years)

One patient became distressed when she talked about the worries of her family, but explicitly wanted to continue answering the questions because she thought they were important. One other patient found answering the questionnaire burdensome because she had problems concentrating. All other patients did not find it distressing or burdensome to answer the questions. In the United Kingdom, 8 of the 10 patients stated that they wanted to keep all questions in IPOS, one patient expanded on the importance and variety of all questions:

… when someone’s not well as I say it changes your whole system your whole … routine … and everything needed to be looked into properly. (United Kingdom, female aged 70 years)

One or two patients each suggested the following aspects to be additionally included in the questionnaire: ease of contacting the relevant team, experience of care, general well-being, insomnia and diarrhoea. We did not incorporate these in the refined IPOS for the following reasons: We recognise the importance of measuring patient experience of care, but intended this questionnaire to remain a PROM. The aim was to assess specific symptoms and concerns. Defining insomnia is difficult, often different types of it are described.¹⁶ Besides, an open list is provided for patients for not-listed symptoms, for example, insomnia and diarrhoea. The group agreed that this was sufficient, as we wanted to minimise respondent burden.

Finally, layout problems were identified, especially the difficulty of staying in the right line for questions 1, 2, 7 and 8. The layout changed automatically due to the item refinements, especially shortening/aligning of answer options.

Added value of cognitive interviewing in two languages for IPOS refinement

The difficult translation of ‘felt good about yourself as a person’ (question 6) into German and the fact that the chosen translation was mainly understood by patients as physical well-being led to detailed discussions between both research teams about the question’s meaning and intention. As the intended emphasis was on spiritual well-being, the question was replaced by the respective question from the African POS ‘Have you felt at peace?’ This question is a good measure of spiritual well-being in African PC and is highly relevant across diverse cultures and beliefs.¹⁷ Discussion in the team also helped to find concise and clear phrasing in both languages for ‘mouth problems’ (question 2) as well as for questions 8 (information received) and 9 (practical matters). The items ‘weakness or lack of energy’, ‘drowsiness’ and ‘poor mobility’ were left unchanged after extensive discussion. For this discussion, the partly conflicting results from the interviews in both languages and considerations for alternatives in both languages – which were deemed inferior to the current items – were paramount.

Discussion

The cognitive interviews conducted simultaneously in Germany and the United Kingdom showed that for the majority of patients, most questions and answer options of the IPOS worked well. The difficulties that were identified were mainly comprehension problems, which were addressed in questionnaire refinement. A judgement problem was reflected in some patients’ replies regarding the 3-day time period over which symptoms were sought. These patients found it hard to judge the score if symptoms were fluctuating. This corresponds with the findings of Bergh et al.¹⁸ in their cognitive interview study of the Edmonton Symptom Assessment System (ESAS), where patients found judging intensity ‘now’ difficult for fluctuating symptoms. The scoring of fluctuating symptoms seems to be a challenge for patients regardless of whether they are asked to score them ‘now’ or over a period of time. In accordance with the wish of some patients for a longer time frame than 3 days, a 7-day IPOS version is now also available. Generally, this problem of scoring fluctuating symptoms on a questionnaire illustrates that any questionnaire can only be one component of a comprehensive clinical assessment.¹⁹

Problems with retrieval or response formulation were not identified in this study. The absence of retrieval problems corresponds with the findings of a former cognitive interviewing study in PC and may be due to the short time period for which symptoms were sought.²⁰ However, the absence of retrieval problems is important to note for this population, which was mostly elderly and with low functional status. Another important finding was that except the two patients mentioned earlier, these PC patients with mostly poor functional status did not find it distressing or burdensome to complete the questionnaire, and that they wanted to retain all questions.

Overall, cognitive interviewing showed good comprehension and acceptability of the questionnaire, identified difficulties in its completion and helped to refine it. Enabling patients to shape IPOS greatly benefited the refinement. Cognitive interviewing of patients therefore proved a valuable method to reduce measurement error caused by misinterpretation of some questions and thereby to increase the questionnaire’s validity. Based on our findings, we want to emphasise the previously published view that cognitive interviewing is a valuable tool in questionnaire refinement and that it can be used effectively in the PC setting.^20,21

To our knowledge, this is the first study in PC showing that simultaneous interviews in two languages can provide additional benefits for questionnaire refinement. Multinational perspectives have generally not been well represented in cross-national survey instruments so far.²² This is also true for PC outcome measures despite their cross-national and international use. Studies that have used cognitive interview techniques in the development of PC outcome measurement instruments have done so in a single language.^18–21 In other areas of medicine, results of cognitive interviews in several languages have been published.^23,24 These studies, however, do not comment on the additional value of interviews in several languages to refine the tested outcome measures.

We want to emphasise that we still recommend cognitive interviewing in one language first for the development of a questionnaire from scratch. In this study, we wanted to refine an already existing questionnaire, the POS. The consistent results from the two countries and languages strengthened the credibility of the findings²² and therefore the evidence to inform the changes to the questionnaire. As Lee²² recommends, in multi-site interviews, the aim should also be to identify as many different problems as possible across different sites. As demonstrated, results that were only found in one of the countries, for example, regarding ‘felt good about yourself as a person’, and conflicting results between the countries (e.g. regarding ‘mobility’ and ‘drowsiness’) added important aspects to the common findings. Collating the findings from interviews in the two languages and discussing them from two linguistic angles resulted in wider and deeper consideration of meaning and context of each item. This helped to shape the final version of the questionnaire, based on the patients’ perspectives in both countries, to the optimum.

Methodological issues, strengths and limitations

In this study, we combined ‘think aloud’ and probing techniques as advocated by cognitive interview experts.¹⁴ Thus, we tried to maximise the information gained – such as unanticipated information by the ‘think aloud’ technique and specific information, for example, regarding the wording of certain questions, by probing techniques – and to minimise interviewer-imposed bias.¹⁴ As recommended for interviewing in multiple countries or languages, we used standardised probing as well as spontaneous probing, thus combining their respective advantages:²² Standardised questions maintain a level of objectivity and consistency of interviews across countries, while spontaneous probing allows flexibility to adjust the interview to the specific situations and different communication styles and facilitates more in-depth investigation into various aspects.²² We used concurrent probing as retrospective probing was likely to be hindered by recall problems in this elderly population.²⁰

It is debated whether ‘formal’ or ‘informal analysis’ of cognitive interviews is preferable.¹⁴ In this study, we used thematic analysis as described by Willis¹⁴ and additional coding applying the four categories from Tourangeau’s¹⁰ question response model, that is, a formal approach. However, as argued elsewhere, questions and answer options that worked well and the main difficulties emerged without formal analysis.^14,20 Analysis was performed by two independent researchers in each country to enhance the findings’ credibility.

A limitation of this study is its relatively small sample size. It is still debated what constitutes a ‘sufficient’ sample size for cognitive interviews.^15,22 However, small numbers are common as this method is work-intensive and yields rich data.^14,20 With our study, we met current recommendations of 5–15 respondents per language group.²² The majority of respondents had cancer which questions the transferability of the findings to people with other life-limiting diseases. However, this reflects the still high proportion of patients with cancer cared for by specialist PC in both countries. As only three community patients were included in the study, the results mainly reflect the views of PC in-patients. However, these patients have complex PC needs – as this is by definition the reason for a referral to a PC hospital support team or PC unit. Therefore, it seems adequate to base the development of a tool that aims to assess PC concerns mainly on the views of these patients.

Conclusion

Cognitive interviewing of patients for whom the questionnaire is intended proved a valuable tool to increase its face and content validity. The concurrent approach in two languages – to our knowledge the first such approach in PC – was beneficial. The refined version is currently undergoing psychometric validation in Germany and the United Kingdom. Wider awareness and use of cognitive interviewing in measure refinement in PC is crucial.

Footnotes

Appendix 1 Acknowledgements

The authors would like to thank the patients for their engaged participation and the staff at the participating centres for their help with recruitment.

Funding

This article presents independent research funded by the National Institute for Health Research (NIHR) under the Programme Grants for Applied Research scheme (RP-PG-1210-12015) (for the interviews in the United Kingdom). The views and opinions expressed are those of the authors and do not necessarily reflect those of the NHS, the NIHR, MRC, CCF, NETSCC, the NIHR Programme Grants for Applied Research programme or the Department of Health. In Germany, this research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References

Dawson

Doll

Fitzpatrick

. The routine use of patient reported outcome measures in healthcare settings. BMJ 2010; 340: c186.

Hearn

Higginson

. Development and validation of a core outcome measure for palliative care: the palliative care outcome scale. Palliative Care Core Audit Project Advisory Group. Qual Health Care 1999; 8: 219–227.

Aspinal

Hughes

Higginson

. A user’s guide to the Palliative care Outcome Scale. London: Palliative Care and Policy Publications, 2002.

Bausewein

Le Grice

Simon

. The use of two common palliative outcome measures in clinical care and research: a systematic review of POS and STAS. Palliat Med 2011; 25: 304–313.

POS Development Team. Palliative Care Outcome Scale (POS), http://www.pos-pal.org (2015, accessed 6 March 2015).

Terwee

Bot

de Boer

. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60: 34–42.

Higginson

. Quality criteria valuable with slight modification. J Clin Epidemiol 2007; 60: 1315; author reply 1315-4.

Willis

Royston

Bercini

. The use of verbal report methods in the development and testing of survey questionnaires. Appl Cogn Psychol 1991; 5: 251–267.

Jobe

. Cognitive psychology and self-reports: models and methods. Qual Life Res 2003; 12: 219–227.

10.

Tourangeau

. Cognitive sciences and survey methods. In: Jabine

Straf

Tanur

. (eds) Cognitive aspects of survey methodology: building a bridge between disciplines. Washington, DC: National Academy Press, 1984, pp. 73–101.

11.

Johnson

Aaronson

Blazeby

. EORTC Quality of Life Group: guidelines for developing questionnaire modules, http://groups.eortc.be/qol/sites/default/files/archives/guidelines_for_developing_questionnaire-_final.pdf (2011, accessed 6 March 2015).

12.

Abernethy

Shelby-James

Fazekas

. The Australia-modified Karnofsky Performance Status (AKPS) scale: a revised scale for contemporary palliative care clinical practice [ISRCTN81117481]. BMC Palliat Care 2005; 4: 1–12.

13.

Antunes

Daveson

Ramsenthaler

. The Palliative care Outcome Scale Manual for cross-cultural adaptation and psychometric validation, 2012, http://pos-pal.org/Resources.php

14.

Willis

. Cognitive interviewing: a tool for improving questionnaire design. 2005. Thousand Oaks, CA: SAGE.

15.

Beatty

Willis

. Research synthesis: the practice of cognitive interviewing. Public Opin Q 2007; 71: 287–311.

16.

Malik

Higginson

Gysels

. Sleep disturbances in advanced disease: a systematic literature review of assessment methods. Palliat Med 2008; 22: 449.

17.

Selman

Speck

Gysels

. ‘Peace’ and ‘life worthwhile’ as measures of spiritual well-being in African palliative care: a mixed-methods study. Health Qual Life Outcomes 2013; 11: 94.

18.

Bergh

Kvalem

Aass

. What does the answer mean? A qualitative study of how palliative cancer patients interpret and respond to the Edmonton Symptom Assessment System. Palliat Med 2011; 25: 716–724.

19.

Watanabe

Nekolaichuk

Beaumont

. The Edmonton symptom assessment system – what do patients think? Support Care Cancer 2009; 17: 675–683.

20.

Murtagh

Addington-Hall

Higginson

. The value of cognitive interviewing techniques in palliative care research. Palliat Med 2007; 21: 87–93.

21.

Ahmed

Bestall

Payne

. The use of cognitive interviewing methodology in the design and testing of a screening tool for supportive and palliative care needs. Support Care Cancer 2009; 17: 665–673.

22.

Lee

. Conducting cognitive interviews in cross-national settings. Assessment 2012; 21: 227–240.

23.

Brandt

Angun

Coyne

. LUTS patient reported outcomes tool: linguistic validation in 10 European languages. Neurourol Urodyn 2013; 32: 75–81.

24.

Wallen

Middleton

Rivera-Goba

. Validating English- and Spanish-language patient-reported outcome measures in underserved patients with rheumatic disease. Arthritis Res Ther 2011; 13: R1.