The Use of Cognitive Interviewing Methods to Evaluate Mode Effects in Survey Questions

Background

Over the last few decades, the increasing use of cognitive interviewing has led to developments in the way in which the method is used. Commercial and government agencies have incorporated cognitive interviewing into usual procedures for questionnaire development (Beatty and Willis 2007). While these developments have moved the method forward, there is yet to be a consensus regarding best practice (Presser et al. 2004). Instead, practitioners have used the method, specifically the techniques involved, in a variety of ways and tend to tailor the approach depending on their needs. The two main techniques used are probing and “think-aloud” (Willis 2005).

Although cognitive interviewing is mainly used before the main survey as part of the questionnaire development stage, Willis (2005) states that cognitive interviewing can be conducted at various points. Other literature demonstrates that the method has wider applications. For example, cognitive interviewing has been used to evaluate the design of travel records (McGee et al. 2006), test informed consent procedures (Willis 2006), evaluate consents to link survey data to other data (Gray et al. 2008) and to explore other parts of the survey process such as reactions to the survey advance letter (Landreth 2001). Miller (2008) suggests that well-documented cognitive interview findings from the pretesting or piloting stage can be used after the survey fieldwork to assist in the interpretation of quantitative analytic results. More rarely, cognitive interviewing has been purposely placed after the survey fieldwork as a follow-up study. For example, Jakwerth and her colleagues (1999) used cognitive interviews to explore why students were leaving certain questions blank in the National Assessment of Educational Progress assessment questionnaire. Although the variations in uses have made cognitive interviewing difficult to standardize, the ways in which it has been adapted demonstrate that it is a flexible tool and extends beyond addressing the survey question and response process.

The cognitive interviews described in this article were implemented as the final stage of a study looking at measurement differences across the modes. The output of the overall study was a set of principles for designing survey questions that can be used across modes, minimizing measurement error. The first stage was a quantitative mixed-mode experiment that had involved comparing respondents’ answers to survey questions administered by computer-assisted personal interviewing (CAPI), computer-assisted telephone interviewing (CATI), and computer-assisted web interviewing (CAWI). The cognitive interviews that followed were used as a qualitative technique to uncover further how response processes can lead to mode effects. This article describes and evaluates the approach used in the cognitive interviewing phase.

Methodology

The sample for the cognitive interviews was drawn from a subset of those who participated in the earlier quantitative mixed-mode experiment. In brief, the mixed-mode experiment was preceded by a module of 15 questions embedded in NatCen Social Research's Omnibus Survey in 2009 (January–June), which is run as a face-to-face CAPI interview. Omnibus respondents who agreed to be recontacted were randomly assigned to one of the three experimental groups: with a questionnaire administered by (1) CAPI, (2) CATI, or (3) CAWI. The questionnaire included the same original 15 questions from the Omnibus survey (so that longitudinal effects could be explored as a separate part of the study) and an additional 67 questions chosen or designed to test hypotheses about mode effects. The questions were purposely designed to vary in task difficulty, question sensitivity, question type (including satisfaction, other attitudinal, behavioral, and other factual), and question format.

Question formats included short versus long scales, rating versus ranking, agree/disagree statements versus balanced forced choice questions, “yes/no for each” versus “mark all that apply,” unfolding versus one-step scales, fully labeled scales versus end-labeled scales, and show card versus no show card on long lists in CAPI. The response rates for the mixed-mode experiment were 73% in CAPI, 69% in CATI, and 47% in CAWI. Modeling was used to adjust for nonresponse bias differences in the three modes.

The quantitative analysis showed some hypotheses supported, some not supported, and some surprising findings. More detail about this stage of the research and the findings can be found in Jäckle et al. (2011), Lynn et al. (2011), Hope et al. (2011), and Nicolaas et al. (2011).

Cognitive interviewing typically uses small purposive samples designed to include respondents with characteristics that may have a bearing on how the survey questions are approached (Willis 2005). Using this principle for this study, we would ideally have recruited respondents who showed mode effects in the mixed-mode experiment. However, mode effects are mainly detected in aggregate comparisons between modes, rather than at the level of individual respondent behavior. Individual-level differences could only be detected if respondents experienced more than one mode, which was not the case in the quantitative experiment.

Fortunately, however, there were two aspects of respondent behavior (in the experimental data) that could be identified at an individual level and, when aggregated, varied across modes. Both these behaviors represent forms of satisficing (Krosnick 1991) in which survey answers are given, but the full processing needed to give an optimal answer does not take place. Krosnick (1991) identifies three factors that contribute to the likelihood of adopting a satisficing response strategy: task difficulty, respondent ability, and respondent motivation. Typically, task difficulty and respondent motivation tend to vary as a result of the mode of data collection (Roberts 2007). The behaviors observed at an individual level in the quantitative experiments were:

agreeing to opposite statements (an indicator of acquiescence behavior, which is the tendency of individuals to agree with something regardless of the content) with those in the interview modes (face to face and telephone), showing more agreement to opposite statements than those in the web mode; and

A regression analysis of the experimental data suggested that respondents with lower socioeconomic status and from minority ethnic groups were overrepresented in these two groups, termed acquiescers (agreeing to opposite statements) and rankers (non-differentiation in ranking tasks). It was important to have some contrasting respondents in the cognitive interviews, so a third group of respondents from the experiment were selected. This group (referred to as non-satisficers in this study) included those who had not acquiesced and had no ranking errors and who had an opposite demographic and socioeconomic profile to those who had exhibited satisficing behavior, which included individuals who were white, had high education and income levels, and were house owners.

The quota for the cognitive interviews was designed to obtain the following numbers of respondents in each group: 18 acquiescers, 9 rankers, and 9 non-satisficers. These numbers were chosen because we wanted the majority of respondents to be those who had demonstrated less than optimal behavior, and the acquiescence hypothesis was a higher priority than the ranking hypothesis. There was some overlap with members of the sample displaying both acquiescence and non-differentiation behaviors. Those who displayed both behaviors were allocated to the acquiescer group for the purpose of quota numbers as this was the more important hypothesis.

Owing to the techniques used, such as having respondents think-aloud and interviewers probing, a cognitive interview session takes much longer than the simple administration of survey questions. We needed to limit the interview to 1 hour, so it was not possible to include all 67 questions from the experiment. Therefore, only questions that had unclear quantitative findings were selected for the cognitive interviews. Sets of existing questions were divided into two parts (based on the results from the quantitative analysis with the goal of creating two equivalent groups of questions) or, in some cases, an additional version of the question was created. This ensured that no respondent was asked the same question in more than one mode. The cognitive interviews were carried out without reference to respondents’ previous answers in the mixed-mode experiment interview. This was because at least 5 months had passed since that interview and it would be impossible to expect people to remember how and why they had answered as they did.

Cognitive interviews were conducted in one-to-one face-to-face sessions in respondents’ homes in the United Kingdom. Interviews were conducted by researchers from the team and survey interviewers trained and experienced in using cognitive interviewing methods. Interviewers began by reading out a scripted introduction to the respondent. This made clear that the first part would be questions administered in three ways and the second part would involve being asked how they found the questions and came up with their answers. Respondents were assured that there were no right or wrong answers. Next the interviewers administered the survey questions. This part was designed to be as close to the real survey situation as possible. Our aim was to simulate convincingly the three modes, CAPI, CATI, and CAWI, in that order, for all respondents. ¹ Finally, the interviewers carried out a retrospective think-aloud. We now describe in more detail the survey question administration part of the cognitive interview and the retrospective think-aloud.

Respondents were first asked survey questions in person (to replicate CAPI using a paper questionnaire). Survey responses were coded by the interviewer and any behavior that signaled potential problems was noted.

A set of questions administered over the telephone (to replicate CATI) came next. Questions were administered over the telephone to replicate telephone survey conditions, using mobile phone (interviewer) to mobile phone or landline (respondent), with the respondent and interviewer in different rooms. It was considered important to replicate the survey conditions by undertaking “mode mimicking” (Beatty and Schechter 1994). Without this, some of the key features that distinguish telephone interviewing from other modes would not have been reproduced, particularly the absence of nonverbal communication and visual cues and the effect this can have on understanding and rapport. Respondent difficulties related to having no visual cues and feedback suggests this had successfully been achieved.

While phone interviewing did raise a few technology problems (poor reception and interference), respondents cooperated and were willing to move to another room even when it may have seemed unnecessary. It is our standard practice to audio record all cognitive interviews and, although there was no probing over the phone, we still felt it was important to have a record of the words exchanged during the telephone portion of the interview. Respondents might have spontaneously verbalized issues, asked for the question to be repeated, or given other verbal cues to signify problems, such as laughter, sighing or hesitation. The interviewers, therefore, switched their mobile phones to speaker mode and placed the digital recorder close to the phone, so that the voices of both parties could be recorded. This was successful almost all the time.

Finally, interviewers asked respondents to answer the last set of questions by themselves on the interviewer’s laptop (i.e., CAWI in an offline version). It could be argued that mode mimicking was achieved here, as the respondent completed the web questions on the computer; therefore, the presentation of the questions and experience of using the computer to respond was replicated. Although the presence of the interviewer could have changed the web experience for the respondent, we tried to minimize any effects. Interviewers were instructed not to intervene while the respondent answered the CAWI questions, to sit so that they could not see the laptop, and to ask the respondent to proceed as if the interviewer was not there. Feedback from respondents suggests that privacy, a feeling of being able to go at one’s own pace and to answer without being judged, was achieved despite the interviewer’s presence.

Although the intention was to mode mimic as far as possible, it should be noted that for the cognitive interviews, the CAPI and CATI questions were administered using paper. It was not practical to program a CAPI or CATI script and it was felt that the delivery of the question would be exactly the same (i.e., read out by the interviewer with show cards where necessary). Furthermore, the retrospective think-aloud phase of the interview required the interviewer to jump around the questionnaires rather than follow question order, which was easier on paper. We felt that paper administration would work just as well owing to the lack of routing in the question sets, with no need for textfills or looping. The web questions were administered on the computer (in an offline version), as it is not possible to replicate the experience of a web questionnaire on paper.

Once the questions had been administered in CAPI, CATI, and CAWI, interviewers used retrospective think-aloud and probing. Interviewers read out a script to train the respondent to think-aloud retrospectively. For most questions, the retrospective think-aloud involved reminding the respondent of the question that he or she had answered, the data collection mode in which it had been asked, and the response provided. The interviewer also brought up any behavior the respondent had displayed while answering that signaled that there could have been a problem, such as hesitation or difficulty choosing between answer categories. The respondent then talked through how he or she had gone about answering the question and decided on the answer given: the retrospective think-aloud. When needed to explore specific quantitative findings, the interviewer asked the respondent structured open probes. For example, to explore the issue of non-differentiation, interviewers were instructed to probe: “I noticed you picked the same ranking more than once. If you had to choose, which would be the most important of these? How easy or difficult would it be to choose? Why is that?”

Think-aloud reports uncover respondents’ reasoning in answering the survey question (e.g., why they chose the middle category), and in analysis we could see if these reasons differed by mode (e.g., more explanations indicating satisficing for web). In doing this, we were relying on the assumption that think-aloud reports are a literal reflection of thought processes a respondent goes thorough when processing information and deciding how to respond. We used a retrospective approach because it was essential that the administration of the survey questions at the beginning of the cognitive interviews mimicked the real modes of data and thus be free from any cognitive interview intrusions. Being retrospective, we missed the option of a live think-aloud and became dependant on the quality of respondents’ recall processes. The limitations of this are discussed later.

We also used prespecified, scripted probes for interviewers. These are less typical, as a strength of cognitive interviewing is its flexibility to include spontaneous probes. The standardized probes were written to address specific issues, and we stressed the importance of asking all the prespecified probes to interviewers. This is because, while the interviewers were very familiar with cognitive interviewing to test survey questions, this was a new application. The focus of the probing was not to assess the performance of the questions in terms of how well they met their measurement aims. Instead, it was to gain insight into the thought processes of the respondent and whether the mode of data collection impacted in any way on how they had answered.

All cognitive interviews were audio recorded with respondent consent and then transcribed, word for word, to allow for a full account of the interview. The alternative would have been to use interviewer notes. However, interviewers are usually trained to record problems with survey questions against their measurement aims. Issues relating to mode effects would be alien to interviewers and are difficult to anticipate.

The cognitive interview (transcribed) data were analyzed using a qualitative data management program Framework (Ritchie and Lewis 2003; Spencer et al. 2003) to build a qualitative data set and then to facilitate analysis. Framework is used for rigorous and transparent data management and allows the analyst freedom to conduct across- and within-case interpretation. Several different types of information were entered into the Framework program. In addition to all relevant findings from the think-alouds and probes, the entered summaries included verbal comments that were made at the time the respondent answered the survey question and the respondents’ answers to the survey questions. Additionally, demographic and socioeconomic information about the respondent (e.g., age, education, sex, economic status, income, tenure, ethnicity, and level of Internet use) drawn from the original Omnibus survey data was entered. There was also a link made back to the transcript so that during analysis, the researcher could go back to the full record. Quotes were included in the Framework summaries to demonstrate points and provide examples.

Once the data had been entered into Framework (data management), analysis began. Framework allows the analyst to run data queries, asking the software to show all respondents who answered a question in a particular way or showing the data against certain respondents in the sample. Usually, when using Framework to facilitate analysis of cognitive interview findings, the researcher would run queries for each test question with the aim of reviewing the evidence on understanding and recall for cases within the sample.

The use of Framework for this study, however, was slightly different and much more focused. The cognitive interview analysis plan was created based on how respondents had answered the survey questions during the cognitive interviews and groups were created based on the findings from the quantitative experiment. For example, as the level of acquiescence varied across mode in the quantitative experiment, we grouped the cognitive respondents into those who had agreed to opposite statements on the survey questions in the cognitive interview versus those who did not. We looked at explanations around answers given, comparing these against the reasons for answering from those who had not displayed such behavior. We also looked for differences in response patterns across modes. The directed analysis sought to explore whether there were any themes across these respondents’ explanations and whether they sensibly justified the way they had answered.

Discussion

The cognitive interviews described in this article were implemented after quantitative data collection on a mixed-mode experiment. The interviews helped explain some of the findings in the quantitative data. They provided insight into the way in which respondents process questions under the conditions of the different modes while also highlighting that response behaviors are often related to question wording, question format, and other factors beyond mode. The approach we used to conduct the mixed-mode cognitive interviews, however, is not without limitations.

While it was our intention to mimic the three modes (CAPI, CATI, and CAWI), it was also necessary for the interviewer to be present to conduct the retrospective think-aloud. We would argue that realistic mode mimicking was achieved for the face-to-face questions. However, we believe that only partial mode mimicking was achieved for the telephone and web questions. We successfully replicated the key elements of a telephone interview (absence of visual cues and the use of body language and reliance on oral and aural communication), but conditions could be considered different from standard telephone interviewing procedures. Notably, face-to-face rapport was established between cognitive interviewer and respondent. For the web questions, although the questions were visually presented on a laptop, in most instances the interviewer remained in the room (although never hovering over the respondent and the laptop). Additionally, respondents were told in an introduction, “In the second part of today’s session, I will be asking you some questions about how you found answering the questionnaire and how you came up with your answers.” It is not clear if this was noticed or remembered by all respondents, but this possible influence should be noted.

Our use of retrospective think-aloud avoided some of the problems associated with concurrent think-aloud, for example, the process of thinking aloud can impact on respondents’ answers to the survey questions themselves (Russo et al. 1989; van den Haak et al. 2003). But both concurrent and retrospective methods could press respondents to produce verbal reports when they may not have access to their thinking (Conrad and Blair 2009). Retrospective methods suffer from the problem of recall. In particular, Ericsson and Simon (1993) believe verbal reports are less likely to be genuine if they are retrieved from long-term (as opposed to short-term) memory. This would be true despite our efforts to remind respondents of each question experience.

Cognitive interviewing did identify problems with some of the survey questions used in the original experiment (e.g., what was meant by “local” and the difficulty that end-labeled scales pose for behavioral frequency questions). This is an interesting discovery since most of the questions were taken from well-established, good-quality surveys. This highlights the importance of using high-quality survey questions for methodological experimental research, otherwise there is a risk that it is not possible to draw conclusions about the methodological hypotheses.

The cognitive interviewing protocol was fairly structured, and spontaneous probing was discouraged for reasons outlined earlier. However, interviewers reported feeling frustrated that they were unable to diverge from the scripted probes and discouraged from following up any ambiguities or problems with the way the questions had worked per se. Maybe we would have uncovered more in relation to mode had we given interviewers the flexibility this method usually allows.

The final limitation discussed here concerns the analysis process used. Themes were identified as in typical qualitative research. But to try to understand mode differences, which are usually manifest at the aggregate level, it was hard to avoid looking at the magnitude of the differences across modes. ⁴ These quantitative-like references stray from qualitative research and led to discomfort within the team. It should also be acknowledged that identifying mode effects is difficult and the team was not successful in finding them for all of the questions studied.

Conclusions

This study used cognitive interviewing in a novel way for a number of reasons. Cognitive interviews were carried out after a survey, and there was no intention of feeding the findings into a questionnaire development process. Next, the sample was not designed not to represent a survey population or respondent characteristics which may have a bearing on how people approach questions (where problems are anticipated). Instead, the sample was made up of respondents who had displayed behaviors that the quantitative experiment had shown were linked to mode differences. The interviewing had a number of very different, and somewhat unusual, features in that the cognitive probing took place after all questions had been administered in three mimicked data collection modes. We have already addressed how this process could have impacted on the cognitive data that were collected. Also, the reason for carrying out the retrospective think-aloud, and asking the few verbal probes, was not to explore how well the questions had performed against their measurement aims. Our aim was to unravel otherwise hidden differences that come into play as survey respondents process and answer questions under the conditions of the different data collection modes.

While the procedures differed from those we usually employ when we carry out cognitive interviewing, the basic techniques can be used to explore the question and answer process more broadly. In fact, cognitive interviewing methods could be tailored to explore decision-making processes in wider contexts over and above the evaluation of survey questions. This study demonstrated that qualitative research can generate findings that add to and illuminate the findings from quantitative methodological experiments. We advocate the use of cognitive interviewing and other qualitative methods alongside quantitative methods rather than replacing them, since in methodological work the larger numbers available in quantitative approaches allow examination of overall patterns across respondents.

If cognitive interviewing is to be used in novel ways, we advise that interviewers should be thoroughly trained to carry out tasks outside their usual areas of expertise. This is particularly important if experienced cognitive interviewers are to be used, since old habits may need to be unlearned and new ones adopted for the tailored techniques to be successful.

Finally, for those considering using cognitive interviewing to address and answer methodological questions, we advocate putting much care and attention into the design and testing of the questions being used. Poorly worded questions could well detract from the main focus of addressing mode effects or other methodological issues.