Abstract
This column focuses on pretesting surveys, which represent an important series of steps in the survey process that must be undertaken prior to administering an actual (final) survey to participants. The authors welcome comments from all readers who have suggestions about the way information is presented or questions about the content of this column. It is not comprehensive and does not replace information found in textbooks or peer-reviewed articles. Beyond their experience, the authors have used textbooks and articles as sources and recommend that you also refer to them for more detailed explanations of what is discussed here.
Gwen: Hello, Allan. I hope you received the survey items I sent you and have had a chance to review them—have you?
Allan: Yes, I have. They look great and I'm excited to get this project moving forward so we can start collecting data with this survey. Can we do that now?
Gwen: Not quite yet. First, as I mentioned last time, we need to pretest the survey items. Let's discuss pretesting and develop a plan to carry it out. Then, in a few weeks, we can discuss administering the actual survey to participants.
Allan: Ok, but is pretesting really necessary? After all, you and I are satisfied, and I've never heard of others doing this.
Gwen: Because everything rests on the quality of the data we collect, this is a very important step in our study. Given its importance, researchers are using this approach more frequently and more is being published about different ways to accomplish this step (Willis, 2005). Although the specifics of pretesting vary among researchers, the primary goal is the same: to make sure that the survey items can measure the domains of interest reliably and validly when used in real situations.
For our purposes, I am using Dillman, Smyth, and Christian's (2009) four-stage pretesting approach: (1) obtaining expert feedback about the initial version of the survey; (2) conducting cognitive interviews; (3) collecting preliminary data on the survey; and (4) reviewing the final survey.
By sending you the initial version of the survey and asking you to review it, I have taken one step in the direction of obtaining expert feedback. To minimize bias from your responses, I will obtain input from other experts. For our purposes, an expert can be a professional with detailed knowledge about the domain we want to measure with the survey. So other experts might be your hospital colleagues or employees, especially those with experience in facility design. Similarly, I can also ask some of my colleagues for input to review the survey.
Allan: What exactly happens when a survey is reviewed? What type of feedback do you expect to receive from me and other experts?
Gwen: One approach is to send the survey to all of the experts and ask them to read the questions to see whether the questions appear to be measuring what we are trying to measure. In addition to reading the questions for content, we can ask the experts to look for anything that is confusing or does not make sense. For example, sometimes the response scale (1 = Strongly Disagree to 5 = Strongly Agree) may not correspond very well with the survey item, or the survey item is too complicated. After receiving all of the expert responses back, changes to the survey items are made and a revised version of the survey can be resent to the experts. This iterative process can stop once the changes suggested by the experts are not substantive or are unlikely to result in significant changes to the survey items.
Allan: Ok, that makes sense, but it sounds like it could take a long time. Remember, this project is on a tight time line. How is the approach you just described different from Dillman et al.'s (2009) second step, where you seem to be conducting interviews?
Gwen: Good question! The second step consists of conducting cognitive interviews. Instead of sending the surveys to experts for review, the people you invite for cognitive interviews should be similar to those who will be asked to complete the actual survey. Instead of interviewing executives in facility design or healthcare experts, people who are similar to potential participants should be identified and invited to be interviewed for this stage. And regarding your concern about time, it's key to select experts for the first step who have a track record of completing their tasks on time.
Allan: What do we ask during these cognitive interviews? Do we ask about our survey questions? If so, why not just give them the survey questions to complete?
Gwen: The interview is our opportunity to understand what participants are thinking as they answer the survey questions. This will help us identify ways to improve the survey before we actually send it out to be completed.
One way to conduct the interview is:
An interviewer welcomes the interviewee, obtains informed consent, asks permission to audiotape the interview (and if permission is given, turns on the equipment to audiotape), and then gives the interviewee a printed version of the survey.
The interviewer explains the purpose of the cognitive interview. This includes: (1) telling the interviewee that he/she will be asked to read each survey item aloud; (2) answering each item; (3) describing what he/she was thinking while answering the item; and (4) identifying anything that was confusing. The interviewee is then asked if he/she has any questions before the cognitive interview starts.
A practice question, such as “How many electrical outlets are in your kitchen?” is then asked. If the interviewee is silent, the interviewer asks the interviewee to discuss his/her thoughts after being asked the question (i.e., “Please tell me what you are thinking about”). By practicing with this or a similar question, the interviewee learns about the expectations for the cognitive interview.
The interviewee then starts reading and answering the items in the survey while explaining what he/she is thinking. Whenever the interviewee becomes silent, he/she is prompted by the interviewer to discuss aloud what he/she is thinking. The interviewer also looks for areas where the interviewee is confused by the survey and/or finds something unclear. During this process, the interviewer takes notes (in case there is a problem with the audiotape) about the comments/thoughts of the interviewee.
Allan: Wow, that sounds like a pretty involved process. I assume inducements are used to find patients who would be willing to participate in this type of interview. What can we offer them in exchange for participating?
Gwen: Participants usually appreciate a small incentive, such as a gift card or free parking for their next visit to your hospital.
Once the cognitive interviews are completed, the researchers examine the results. Specifically, the researchers focus on whether the participants interpreted the questions in the way that was intended and which questions were confusing. The researchers then make the necessary changes to the survey questions.
Allan: This sounds a bit tedious but I understand that you are being cautious and trying to create the best survey possible. Can we collect data now?
Gwen: The good news is that we can collect preliminary data. The bad news is that the data will be preliminary and might not be used as part of our final dataset.
Allan: What determines whether the preliminary data can be used in the final dataset?
Gwen: The key driver of this decision is whether we need to make changes to the survey based on the data we obtain in this step.
Allan: Ok, perhaps I am a bit impatient. What exactly is involved in this preliminary data step?
Gwen: The preliminary data collection step (also called pilot testing by some researchers) involves sending surveys to a small subset of participants. A good rule of thumb for the number of participants to include in a pilot study is 30 (Johanson & Brooks, 2010). Once we receive the surveys back from participants, we can conduct analyses that examine:
Dispersion of responses. Is everyone answering at the upper end (4s and 5s) on a one-to-five-point Likert-type scale, or are the responses spread out across all five response categories? We are interested in asking questions that yield variation in their responses because correlations may then be established among those questions. Without correlations, we have no chance of discovering potential causal relationships that can be used to inform our design decisions.
Reliability of scales. For survey items that measure a specific construct, is the construct being reliably measured? We can assess this by using Cronbach's alpha (Nunnally & Bernstein, 1994). Cronbach's alpha refers to the average intercorrelation of all of the items for the scale; to the extent that the items are related to each other, we view the scale as having internal consistency. Typically we want Cronbach's alpha to be .70 or greater.
Nonresponse to items. Are there certain items that were not answered by the participants? If so, what might be causing this, and how can the items be changed to obtain a higher response rate?
Response rate. What is the overall response rate that we achieved in the pilot study? If it is low, what can be done to improve it?
Allan: Ok, that seems reasonable. What happens in the fourth and final step of pretesting?
Gwen: The last step is another check of the survey after we have made any final modifications to it based on the pilot study. In this step, we should have some people who are unfamiliar with the survey attempt to complete it. Obtaining a fresh perspective about the survey will allow us to make sure that the final version of the survey is ready to administer for our actual study. Once we are satisfied with the survey, we can start administering it officially.
I think we have covered a good amount of material today. When we talk next, let's discuss developing a data analysis plan for our study. How does that sound?
Allan: Great!
Footnotes
Acknowledgment
Funding for the first author was provided by a K02 award from the Agency for Healthcare Research and Quality (Grant # 1 K02 HS017145-02) and The University of Texas at Houston—Memorial Hermann Center for Quality and Safety.