Abstract
With the advent of combination antiretroviral therapy for HIV, people living with HIV are approaching a normal life expectancy (Deeks, Lewin, & Havlir, 2013). Despite improvements in survival, however, comorbidities in this population have increased. Of particular concern is the prevalence of cognitive impairment, estimated to be as high as 69% (Simioni et al., 2010). Cognitive impairment is one of the most voiced concerns of people with HIV and affects, among other life areas, medication adherence, employment, financial management, household management, and driving (Letendre et al., 2004; Price et al., 1999). Cognitive impairment is a factor that contributes to social isolation and loneliness in older people in general (Richard et al., 2013) and is now being experienced by older people with HIV (Grov, Golub, Parsons, Brennan, & Karpiak, 2010).
Occupational therapy practitioners will increasingly be involved with the rehabilitation of people with HIV. Some clients may be referred directly because of difficulty with basic and instrumental activities of daily living and others because they have an indicator for rehabilitation arising from a chronic disease or an age-related event such as a stroke or fracture. Because cognitive impairment is prevalent in people with HIV, practitioners need to consider that it may be a factor underlying some of a client’s experienced difficulties in daily life. Practitioners are charged with assessing and optimizing occupational performance and community participation for their clients, and a cognitive assessment is part of that practice (Lapointe, James, & Craik, 2013).
Recognizing cognitive impairment in people with HIV is not straightforward. Review of the literature on HIV-associated neurocognitive disorders (HANDs) indicates that there is an unmet need for well-validated, low-burden cognitive assessment tools (Simioni et al., 2010). Based on consensus Frascati criteria, full neuropsychological assessment is the gold standard for diagnosis of HANDs (Antinori et al., 2007). Consensus recommendations on appropriate tests exist, but such tests require highly trained personnel not widely available outside of specialized settings. Access and assessment take time and resources that may not be in place before the occupational therapy practitioner joins the treatment team.
The tools commonly used by occupational therapy practitioners are better suited for assessing dementia than the mild cognitive impairment that is more common in HIV (Brouillette et al., 2015). Standardized self-report measures of cognitive impairment are rarely used in occupational therapy (Groves, Coggles, Hinrichs, Berndt, & Bright, 2010), but practitioners may observe a client’s behavior or query a family member or close associate of the client. The clinical community has doubts about the value of self-report cognitive questionnaires because some people with cognitive impairment cannot accurately report their condition because they lack insight (Vogel et al., 2004). In addition, people who are cognitively intact may overreport cognitive concerns because of heightened awareness of potential HIV-associated impairment. Poor measurement of self-reported cognitive concerns might explain some of the discrepancy that is sometimes found between self-reported results and actual cognitive performance. Moreover, cognitive performance as assessed by neuropsychological tests and self-reported cognitive concerns might reflect different constructs. Regardless of whether these assessments align, it is important to elicit the client’s point of view in developing a treatment plan.
Our research group challenged the conventional view of the informativeness of self-reported cognitive concerns, finding, using Rasch analysis, that self-report items fit with neuropsychological tests. The group found that both types of assessments measure the same underlying unidimensional construct in people with HIV who do not have dementia (Brouillette et al., 2015; Koski et al., 2011). Asking people directly about cognitive concerns has the advantage of accessibility, ease of administration, and coverage of all domains of cognition because a wide range of such concerns uniquely experienced by people can be tapped (U.S. Food and Drug Administration [FDA], 2009).
In 2008, the Canadian Working Group on HIV and Rehabilitation identified “enhancing outcome measurement in HIV and rehabilitation research” as one of six research priorities in HIV and rehabilitation (O’Brien, Wilkins, Zack, & Solomon, 2010). Assessing the cognitive concerns of people with HIV may be challenging because the current portfolio of self-report items may not be ideal for identifying the real-life situations of people with HIV experiencing cognitive concerns. Cognitive difficulties in HIV are often relatively mild, and current self-report items may not be suited to capturing such issues. Therefore, there is a need for an HIV-specific measure of cognitive concerns to facilitate communication between people with HIV and the clinical team, to focus treatment efforts on areas the patient perceives as problematic, and potentially to identify cognitive issues early.
This study sought to identify a pool of items that reflect the cognitive concerns expressed by people with HIV as a first step in identifying whether an HIV-specific measure is needed for this population. Although strictly speaking, this measure would be a self-reported outcome (SRO) measure; when people are queried directly about their experiences, such measures are more generally termed patient-reported outcomes (PROs).
Method
In 2009, the FDA released a new version of guidance for industry on the use of PROs to support labeling claims in medical product development. This study reports on the methods used for parts (i) and (ii) of this guideline: developing and adjusting the conceptual framework using literature and patient input and generating new items. Ultimately, the aim of this study is to develop a new PRO for people with HIV: the Communicating Cognitive Concerns Questionnaire (C3Q). Figure 1 shows the steps undertaken to develop the item pool of cognitive concerns.
Steps undertaken to develop the item pool of cognitive concerns.
Item Generation
Two sources of data were used to generate items for the C3Q. The first source—an anonymous, secure, cross-sectional online survey in English and French (hereinafter called the McGill survey)—was carried out using McGill University–supported LimeSurvey software. Participants were recruited through HIV community organizations in Canada, and a link to the survey was also distributed through social media (Twitter and Facebook). Two waves of recruitment were conducted through this survey.
The first part of the McGill survey (Wave 1) asked respondents to choose five aspects of their mental or cognitive function that they felt may have been affected by HIV. This format was an adaptation of a method that had been used in quality-of-life research for the same purposes, that is, the Patient Generated Index (PGI; Ruta, Garratt, Leng, Russell, & MacDonald, 1994). The second part (Wave 2), not linked to the first, asked respondents to provide specific examples for targeted cognitive domains (e.g., memory, concentrating, calculation). Finally, respondents were asked to indicate whether they would be interested in participating in further stages of item pool development by providing contact information. This question was not linked to any of the answers respondents had provided in the previous parts. To ensure confidentiality, no demographic information was collected.
The second source of data arose from a qualitative study regarding the development of the Patient-Reported Outcomes Quality of Life—HIV (PROQOL–HIV; Duracinsky et al., 2012), an HIV-specific multidimensional measure of health-related quality of life. During the development of PROQOL–HIV, in-depth interviews were conducted with 150 people living with HIV, treated and untreated, from nine countries, across five continents, and in 11 languages. The interviews aimed to gather information from participants that comprehensively reflected the day-to-day experience of living with HIV. All of the answers were translated verbatim into French and English and denominalized. The text related to the cognitive functioning domain was reanalyzed to develop a cognitive instrument that would go beyond the multidimensional tool.
Item Deduplication
In the first step of item deduplication for the C3Q, all the text threads from the McGill survey waves were transferred to a database and were mapped onto the cognitive domains typically considered in clinical assessments of mental status in neuropsychology and neurology (e.g., memory, attention, executive function) by members of the multidisciplinary research team representing the fields of cognitive neurology (L. Fellows); psychiatry (M.-J. Brouillette); and neurorehabilitation, epidemiology, and measurement (N. E. Mayo and C. Moriello). The team members independently reviewed the list of text threads to verify cognitive domain placement and identify literal duplications. Literal duplication was defined as identically worded items or items that, in each reviewer’s opinion, referred to the same cognitive concern but were worded differently such that including both in the C3Q would constitute duplication of content. The research coordinator (C. Moriello) collated decisions from each reviewer. Items for which there was reviewer agreement were reduced to a single item in the item pool. Any reviewer disagreements were resolved by consensus.
The second step of item deduplication involved grouping together all items that referred to the same aspect of a specific content theme (e.g., forgetting directions) but differed in time frame or meaning (e.g., I got lost, I can’t remember the directions, I don’t know which way I came from). Such items were grouped together, and the wording of all individual items within each group was retained. The purpose of this phase of deduplication was to retain all aspects of all content themes within the item pool while acknowledging that not all of the items in each group would ultimately be chosen for inclusion in the C3Q. The phases of the deduplication were sequential and started with Waves 1 and 2 of the McGill survey and finished with the content from the PROQOL–HIV data with the aim of identifying the extent to which new data sources yielded new information.
Item Reduction
Two phases of item reduction were conducted to determine which items would be retained in the draft C3Q. The first phase of item reduction consisted of a macro-level removal of items relating to content themes that were outside of cognition, such as anxiety, mood, and fatigue. Decisions regarding the removal of all items within a content theme were informed by the literature and by the multidisciplinary research team.
All items remaining after the first phase covered content themes (e.g., memory, attention, executive function) relevant to cognitive concerns of people with HIV. The team a priori identified that the measure would capture self-reported frequency with which a concern occurred. Items referring to “difficulty with” rather than “frequency of” performing an activity (e.g., reading a book) were removed, but the context (e.g., reading) was retained. This decision positioned the measure as an SRO rather than a PRO. Difficulty cannot be interpreted by anyone else, which is the true definition of a PRO, whereas frequency can have some external verification, although for many, self-report will be an accurate representation of real-world challenges. Difficulty may refer to an existing challenge owing to, for example, eye sight or learning disability.
In the second phase of item reduction, items relating to very specific tasks (e.g., “Asking a question twice after it’s already been answered”) were removed to reduce participant burden and to accommodate the inclusion of similar, but more widely applicable, items (e.g., “I can’t follow a conversation”). Items related to a change from an earlier state were flagged for separate consideration in the questionnaire. Item quality was assessed by the multidisciplinary team. Items with poor reading ease, potential ambiguity, or both were removed (e.g., “I can’t focus on one thought at a time”). Following this phase of item reduction, the wording of the remaining items was adapted to maximize consistency of both item wording and response format. Items were then formatted as the draft C3Q.
Assessing Content Coverage Against a Preexisting Theoretical Framework
The initial queries for the C3Q provided a comprehensive list of the concerns relevant to people with HIV, which were mapped onto existing generic cognitive challenges questionnaires or developed for use in other conditions to identify which cognitive constructs are represented and which have not been identified by members of the HIV community. We used a cognitive neurology and neuropsychology assessment framework (Baars & Gage, 2007) to assess the content coverage of the C3Q. This framework helped in categorizing all of the items and also provided clarity on how well the construct under measurement (cognitive concerns of people with HIV) is represented within the C3Q. According to the framework, cognitive ability has different domains, including attention, memory, language, executive function, and visuospatial skills. The content of each included item in the draft C3Q was assessed for its coverage of domains within this framework.
Simultaneous Translation
A working group of bilingual investigators, including three not involved in the earlier stages, met in person to harmonize the wording of the C3Q items because the wording had arisen from the respondents in English and French and the final instrument was intended for use in both languages, necessitating compatibility in both languages. The advantage of this simultaneous translation, compared with sequential translation (in which items are developed in only one language with subsequent translation into the target languages), is that problematic or idiomatic wording and discrepancies between languages are resolved during the item-generation process (Marquis, Keininger, Acquadro, & De la Loge, 2005). To minimize cognitive burden, the response options were limited to three.
Assessment of Prevalence, Clarity, and Importance
To ensure that people with HIV understand all items and to identify the items that reflect important concerns, a draft of the C3Q was sent to respondents who had agreed to participate further. An invitation was also sent to the community in the same way as in the first step to obtain new respondents. The survey listed all the concerns identified by respondents and concerns added by the research team to represent typically assessed constructs. Each respondent was asked to identify
How often a concern happened to him or her (using the scale frequently [almost every day], sometimes [once a week], and rarely [once a month or less]),
Whether he or she understood what the question is asking, and
Whether this concern or problem was important to him or her.
These data were used to generate a list of items that are relevant to the HIV population, have a range of severity, and have an important impact.
Results
A total of 292 respondents provided information on cognitive concerns: 68 from Wave 1 of the McGill survey; 74 from Wave 2 of the McGill survey, while information from the first wave was being processed; and 150 who participated in PROQOL–HIV development (Duracinsky et al., 2012). A total of 930 unique text threads were received and evaluated: 266 related to memory, 220 related to attention, 231 related to behavior and emotion, 94 related to executive function, 84 related to speech and language, and 35 related to visuospatial skills.
Figure 1 shows the steps carried out to trim the text threads to retain potential items for the C3Q. The literal deduplication phase resulted in the removal of the most text threads (i.e., 594). For example, 32 respondents exemplified their experience of “not being able to remember phone numbers” with different sentences such as “I forget the usual numbers,” “I can’t remember telephone numbers of my near family,” or “I forget my own phone number; I have to check my cell phone to confirm.” These 32 threads were harmonized to a single item: “I forget phone numbers that I know well.”
Threads that referred to the same aspect of a specific content theme but differed in time scale resulted in 12 text threads being harmonized into 1. For example, the three items of “I got lost,” “I can’t remember the directions,” and “I don’t know which way I came from” were reduced to one item of “I forget directions.” With reference to the data collected by the online survey, 73 threads were duplicated in the data from the PROQOL–HIV study sample and were removed, leaving a total of 252 unique text contents, which would be used to form items.
After excluding 94 text threads regarding noncognitive content (anxiety, mood, sleep, energy level, and physical weariness), 158 potential items remained. The micro-level reduction reduced the remaining 158 items to 125 items. Although the content coverage of items removed in this step was relevant for inclusion in the C3Q, they either were related to the performance of very specific tasks or had potential ambiguity (poor readability) and were removed to reduce participant burden and to accommodate the inclusion of other, more widely applicable, items.
Assessing Content Coverage Against Preexisting Questionnaires
As shown in Table 1, the potential items of cognitive concern were mapped to standard neurocognitive domains and subdomains. The 125 items were derived from three separate analyses (one for each of the two McGill survey waves and one for the PROQOL–HIV data). The first analysis on 68 people (McGill survey Wave 1) identified items covering all 15 subdomains plus 9 items for change. For the memory and speech–language domains, the items marked with a superscript “a” (see Table 1) could be mapped to more than one subdomain and hence cannot be summed. The second analysis on 74 people (McGill survey Wave 2) confirmed Wave 1 items and generated 23 new items, plus 5 new items for change. The third analysis on the 150 respondents from the PROQOL–HIV study again confirmed existing items and a few new items and had no new change items. The last column of Table 1 shows the distribution of items across cognitive domains.
Content Coverage of Cognitive Concerns Identified by Three Samples of People With HIV (N = 292)
Note. PROQOL–HIV = Patient-Reported Outcomes Quality of Life—HIV. — = not applicable.
Items are also in other subcategories of the same category.
The extent to which the item pool overlaps or is unique from items included in existing cognitive questionnaires is shown in Supplemental Table 1 (available online at http://otjournal.net; navigate to this article, and click on “Supplemental”). Our systematic literature search identified 13 generic cognitive questionnaires and 2 HIV-specific questionnaires. For each questionnaire, we identified the reporter, the number of items, the number and dimension assessed by the response options, and the cognitive domains covered. The most comprehensive questionnaires are the Multidimensional Assessment of Neurodegenerative Symptoms (MANS) Questionnaire (Locke et al., 2009) and the Everyday Cognition (ECog; Farias et al., 2008), neither of which are specific for people with HIV. The MANS Questionnaire is long, with 91 items that cover language, visuospatial, memory, executive function, and attention domains of cognitive function. The ECog has 39 items, including memory, language, semantic (factual) knowledge, visuospatial abilities, executive function, and attention. The C3Q is the only HIV-specific self-report questionnaire that covers all cognitive domains in its item pool, including memory, attention, executive function, visuospatial, speech and language, behavior and emotion, and cognitive changes.
Assessment of Prevalence, Clarity, and Importance
A sample of 80 people responded to the request for evaluation of the items selected for inclusion in the C3Q, resulting in each item being rated by 30–80 people. In the first step, all items that were rated as not important by >60%, as happening rarely by >60%, or as not clear by >5% of participants were removed from the item list. The new list of items was reviewed by our expert team to ensure that the content of the questionnaire remained adequate. The results are shown in Table 2. Sixty items were kept in the final draft of the C3Q (i.e., the C3Q–60), including 15 items for memory or behavior and emotion domains, 7 for attention, 6 for executive function, and 5 for speech and language. Change in these various domains was captured by 12 items. Items for visuospatial skills did not survive in this phase.
Content Coverage of the 60-Item Communicating Cognitive Concerns Questionnaire
All items that were rated as not important by >60%, as happening rarely by >60%, or as not clear by >5% of participants were removed from the item list.
Discussion
The C3Q–60 is a pool of items that relate to cognitive concerns of people with HIV, developed simultaneously in two languages, English and French. It covers almost all cognitive domains. As the first HIV-specific self-report questionnaire, major efforts were made to do due diligence to ensure that the items are relevant to the HIV population and reflect their cognitive concerns. This article documents the steps taken to develop the C3Q item pool according to the FDA (2009) guideline for developing PROs.
A systematic approach was applied to capture patients’ concerns in this questionnaire, which was different from other studies that relied on expert opinion and the literature for developing content. The items were generated from three different sequential samples. The open-ended format of the PGI (used in Waves 1 and 2 of the McGill survey) facilitated the generation of key issues regarding cognitive challenges from the perspective of people living with HIV. Very little new content was generated by the participants in the PROQOL–HIV study, supporting saturation. There was a concern that McGill survey respondents may have had better Internet skills than people who are not able to use the Internet. Therefore, we compared the item pool generated from the McGill survey with that generated by the PROQOL–HIV study, partly because the population in the study represents a wide range of socioeconomic status and education. There was considerable overlap in the areas identified (see Table 1).
The data collected in this study confirmed that participants are affected across all domains of cognition represented in cognitive frameworks. Note that these domains are not necessarily discrete when considered at the level of everyday life. For example, although “not being able to read a few pages of a book” might reflect problems with attention, it could also be the result of difficulties with memory.
This article demonstrates a process of generating a pool of items from existing theoretical frameworks using input and feedback from the target population. However, the C3Q–60 is still too long to be practical for clinical or research use. Future work is planned to fit the items to the Rasch model to test dimensionality, identify redundant and correlated items, and develop a scoring algorithm that yields a strong total score from the ordinal response options (Tennant & Conaghan, 2007). That work will also allow identification of items that might be affected by demographic variables. The final version of C3Q would be not only a measure of cognition but also, given its inherent ecological validity, a tool to directly guide patient-centered intervention plans. Many of the concerns could be minimized using cognitive rehabilitation strategies (Clare, Woods, Moniz Cook, Orrell, & Spector, 2003).
Rasch analysis can be used to develop short forms to fit different purposes (screening, prevalence, and change over time), which would require further validation. The initial steps reported here show a rigorous approach to developing relevant content.
Implications for Occupational Therapy Practice
This study has the following implications for occupational therapy practice:
The C3Q can provide a comprehensive profile of clients’ strengths and weaknesses in all cognitive domains including memory, attention, language, and executive function.
All C3Q items reflect the voices of people living with HIV, which makes it an HIV-specific measure of cognition.
This new self-report measure is easy to administer and take less than 5 min to complete.
Occupational therapy practitioners can use the C3Q to find the specific domain affected by HIV and tailor their interventions on the basis of the patient’s needs.
Conclusion
This study has identified cognitive concerns of people living with HIV, some of which were not captured by any existing questionnaires. The process of due diligence has contributed unique information that will serve as the basis of the development of an HIV-specific measure of cognitive concerns. It can help occupational therapy practitioners and other members of the clinical team to both measure cognition and guide further assessment and intervention.
Footnotes
Acknowledgments
We would like to thank the participants who completed the online questionnaire. We also are grateful to the following HIV community organizations for their support: Portail VIH/Sida du Québec, Coalition des Organismes Communautaires Québécois de Lutte Contre le Sida, Aids Committee of Toronto, AIDS Vancouver Island, Bureau Local d’Intervention Traitant du Sida, Canadian AIDS Society, Positive Living Society of British Columbia, Mouvement d’Information et d’Entraide dans la Lutte Contre le Sida, Maison Plein Cœur, Sida-Vie Laval, and Maison d’Hérelle.