Understanding Evidence-Based Research Methods: Challenges and Considerations in the Analysis of Survey Data

This contribution to the series of methodology columns addresses frequently encountered issues in analyzing large data sets for research, including ensuring that the data are both usable and reliable. The authors welcome comments, suggestions, and questions about the content and presentation of this column from all readers. This column is not comprehensive and does not replace information found in textbooks or peer-reviewed articles. The authors have used textbooks and articles, in addition to their experience, for reference and recommend that readers also refer to these sources for further explanation of the content in this column.

Scenario: Allan, the Director of Facilities of Hospital XYZ, and Gwen, the senior researcher at a healthcare facilities design research consulting firm, have been discussing methods for conducting effective surveys to determine the relationship between newly constructed buildings and patient/staff attitudes and behaviors. After previously discussing pretesting, Gwen has called Allan to begin planning the data analysis. Readers can access the data file that Allan is using online at http://www.herdjournal.com/Media/DocumentLibrary/HERD_datasetwitherrors.xlsx. Their conversation follows.

Gwen: Hi, Allan, it sounds like pretesting went well, and you've administered your final survey.

Allan: We have! I got the surveys back and entered the results into Microsoft Excel. I think we're now ready to analyze the data.

Gwen: That's great, but we need to be careful before we jump into the data analysis. There are a few more steps to make sure the data are ready.

Allan: OK, where do we start?

Gwen: The first step before data analysis is to ensure that your data are properly formatted. Your records should be in a spreadsheet format such that each column corresponds to a single variable and each row corresponds to a single unit of analysis. In our case, the variables are the questions from the survey, and the units of analysis are the survey participants. Each variable should have a name that is simple and unique, does not have spaces, and does not begin with a number. In our data set, we had each participant answer three questions about how satisfied they were with their working conditions before and after the move. We could refer to these variables as SATISFACTION1A, SATISFACTION2A, and SATISFACTION3A for the responses before the move and SATISFACTION 1B, SATISFACTION2B, and SATISFACTION3B for the responses after the move. With these responses separated into different variables, it will be easier to perform analyses comparing each.

You'll also want to make sure that each participant has a unique numeric identifier in the first column. Participants from all populations should be entered as rows in the same spreadsheet, and you should add an indicator variable to note each participant's population. In our data set, for example, participants from the PICU on the fifth floor of the hospital would have a “1” in the POPULATION column, and participants from the PICU on the seventh floor would have a “2” in the POPULATION column. When using numeric codes for text values, you should create a separate key file that explains the value for each code.

When entering your data, you should format all or most of the data as numbers to allow computer programs to more easily describe and perform tests on your data. For example, even though the participants entered “M” or “F” when we asked them to report their gender, you would enter them as “1” or “2” in the GENDER column, indicating the corresponding values in your key file. Another consideration that you might encounter is collapsing continuous variables, those that have a range of values between two points, to categorical or ordinal variables, which contain a prespecified number of groups. For example, you might be tempted to group ages when you enter your data. It is best to always retain the continuous variables in your spreadsheet, in case you decide you don't want the data collapsed at a later point. If you do want to collapse the data into groups for your analysis, you can create an additional variable, which we'll call AGEGROUP, where “1” corresponds to ages 18 to 29, “2” corresponds to ages 30–44, “3” corresponds to ages 45–59, and “4” corresponds to ages 60 or older.

Allan: That makes sense for all of our numeric response and multiple-choice questions. We also asked open-ended questions, like “What aspect of the hospital did you like best?” How do I handle these?

Gwen: For any variables that are free-text entries, you should check for misspellings and synonyms so that you can group any words with the same meaning. For example, we have “clean,” “clen,” “Clean,” “very clean,” and “cleanliness,” which despite different spellings, capitalizations, lexical categories, and modifiers, all refer to the cleanliness of the building as something that the survey participants liked. Once you have completed this step, you can assign each distinct term to a unique numeric code, keeping the key in a separate document as you did in the prior step.

Allan: What if we have questions where the participants did not respond and the data are missing?

Gwen: Missing data is an important issue that can affect outcomes, and it can be caused by non-response, such as individuals neglecting to answer some or all survey questions; dropout, such as individuals responding to a first survey but not a second; or inadequate data collection. Each cause of missing data can bias the data differently, and it's important that you know the cause and approach the data accordingly. Your statistician may be able to apply analytical methods that can account for the missing variables. Imputation, or substitution, of the missing data is another option that your statistician can help you with to fill in the missing responses. Allan: OK, are we ready to analyze the data yet? Gwen: Not quite, Allan. The next step in data analysis is to ensure that your data are reliable.

Allan: Why would my data be unreliable?

Gwen: Just like missing data, unreliable data can have many causes. Some values may be entered in error either directly by the participants or by an individual entering data from paper or another source into the computer. For example, someone might accidentally hit a key twice, entering “100” instead of “10” or someone might hit the “0” key instead of the “9” key. Survey participants might also be dishonest in their reporting, whether in an effort to sabotage the research or maintain social desirability around sensitive subjects. Finally, you might have skewed data if there were errors in sampling or in the validity of our survey questions. It's important to check for these issues, because they can inflate the error rates and distort your statistics.

Allan: I see. Now I understand why pretesting is so important, as we discussed last time. How do we know if the data are reliable, though?

Gwen: First we'll want to check for outliers in the data. There are several methods that we can explore to do this. Visual inspection is a good place to start looking for logical minimums and maximums, considering the variables we are analyzing. One way to do this is to simply sort the data for each variable from high to low. With your data in Microsoft Excel, you can highlight your data, click on “Sort Largest to Smallest” in the “Data” tab in the “Sort & Filter Group.” When we sort the SATISFACTION1A variable, for example, we find that the largest value is 55. Because we know that 7 is the largest valid value, we know that 55 is an error, likely due to data entry.

We can also view a graph of the data, such as a histogram, which shows the frequency of distinct values for each variable. You can create a histogram with Microsoft Excel's charting functions, the “Analysis ToolPak” add-in for Excel, or you can import the data into SPSS to create the histogram. If we look at the SATISFACTION2A variable now, this time using a histogram, we can see that all of the values are in the range that we expect.

We can also use a statistical method commonly used to find outliers. First, we calculate the mean and standard deviation. To do this in Microsoft Excel, we can use Excel's functions to add a formula into an empty cell of the spreadsheet. The “AVERAGE” function calculates the mean for a variable, and the “STDEV” function calculates the standard deviation. For normally distributed data (i.e., the bell curve), 99.7% of the data lie within three standard deviations of the mean. Therefore, we can consider any observation greater than or less than three standard deviations from the mean an outlier.

Allan: Wow, that's really helpful to know! So what do I do if I find outliers in the data?

Gwen: I'd suggest the first thing is to go back and check the data entry. Because you have hard copies of the surveys, compare the data in your spreadsheet to the data in the surveys and make sure the values were entered correctly. If you still have outliers, then you need to determine the cause of the outlier first. If the outlier is not a valid choice, then you should omit it during your analysis. If the outlier is legitimate, you may still omit it, and many researchers do. However, you can also choose to transform the data, such as converting to a logarithmic scale. This maintains the relative ranking of scores and reduces the error variance, while still including the outliers in the analysis. Transformation is not always appropriate, though, and your results could be difficult to interpret. An alternative is to use robust analyses with the help of a statistician to prevent distortion from outliers. Finally, you can use nonparametric analyses, which have few or no distributional assumptions. For our purposes, any score outside the Likert range of 1 to 7 will be omitted because values outside the range do not correspond to anything meaningful.

This is a lot of information. Why don't we talk another time about data analysis methods? Allan: Great! I look forward to learning more.