Abstract
THE AUGUST 16, 2011, CHRONICLE OF HIGHER EDUCATION ARTICLE “Want Data? Ask Students. Again and Again” by Sara Lipka posits that in higher education we have a culture of oversurveying students and too often rely on surveys as our main, or only, way of assessing the impact of our programs and services on student satisfaction and learning. Because of this, students are experiencing survey fatigue, resulting in lower response rates and data of questionable validity. According to the American Association for Public Opinion Research, “A low cooperation or response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by nonrespondents.” Students who choose to respond to a survey even though they are not motivated to engage in the survey process may respond in either socially desirable ways or quickly and without much thought. As Lipka writes:
Two decades ago, 70 percent of students would answer a survey, campus officials recall. Now, by some standards, a 20-percent response rate is decent. In this year's National Survey of Stu- dent Engagement, more than a third of colleges had less than 30 percent of their students respond. The quality of responses, too, may be suffering. Databases show partial completers and straight-line respondents, students who scroll down a survey checking, say, only 3's. Even reading and answering each question doesn't necessarily represent much thinking. (para 5).
Lipka also notes concerns voiced by Patrick T. Terenzini, a professor emeritus of education and a senior scientist emeritus at the Center for the Study of Higher Education at Pennsylvania State University, who says that “Oversurveying is counterproductive…. This is a nontrivial threat to higher education and our ability to monitor what we're doing. Institutions are not getting the quality of information they need to make responsible, informed decisions” (para 9).
Lipka's article includes suggestions from various institutions and/or individuals for improving student engagement in the survey process, such as enticing students to participate and providing students feedback on how their responses may make a difference. For example, Don A. Dillman, a professor of sociology and deputy director for research and development of the Social and Economic Sciences Research Center at Washington State University, recommends, in addition to sound survey research practice, including a dollar bill or two in the survey request to signal the importance of a survey. He offers the example of Indiana University at Bloomington, which holds lotteries for survey participants with prizes like iPads; $10 gift cards for Amazon, Starbucks, or Target; personal training sessions; and Indiana University sweatshirts. Other institutions offer students free food. Thirty-nine percent of the 761 institutions that participate in the National Survey of Student Engagement hold drawings for participants to win gift cards, parking permits, or iPads.
While we agree with every point Lipka presents, we do not think she went far enough in presenting suggestions or potential solutions, if you will, to the overuse of surveys in higher education. We want to challenge others to think more creatively about the assessment process, so that we respect the time and energies of our students and also collect data that is more likely to be an accurate and more valid representation of what students experience, feel, and have learned.
What more can we do to respect our students’ voices and encourage their engagement in the assessment process, and to ultimately increase the credibility of our findings?
Consider other methods of data collection beyond surveys, and include qualitative, quantitative, and mixed methods. As stated in the Council for the Advancement of Standards in Higher Education Standards, “assessments must include qualitative and quantitative methodologies as appropriate, to determine whether and to what degree the stated mission, goals, and student learning and developments outcomes are being met” (p. 35).
Select a method that allows you to measure your goals and outcomes efficiently and accurately. These can include tracking of usage, review of existing documents and data, focus groups or interviews, observations, case studies, rubrics, portfolios, performance assessments, or one-minute assessments.
Include a combination of direct and indirect measures of student learning. For example, at Portland State University (PSU), the Assessment Coordinator works with programs to assess both their program and learning outcomes. Data collection often is a combination of usage numbers and existing data for program outcomes assessment and mixed methods for assessment of learning outcomes. An example of this is found in the Learning Center's 2011—2012 Assessment Plan: “The Learning Center will implement an effective social media campaign. Both formative and assessment data will be collected to develop and implement this campaign, including conducting focus groups; monitoring usage and content; and surveying students for feedback.”
Regardless of assessment method used, ensure that students know their voices matter by providing incentives that are meaningful to them, such as early registration for classes or discounts on book purchases or meal plans. Also, consider the timing when offering incentives. According to Dillman, providing incentives at the outset, along with the invitation to participate, has been found to be more effective than the promise of a reward upon completion of the survey.
Give students feedback about how the data are used and what changes have resulted from their responses. It is important that feedback is provided in a variety of forms using a variety of mechanisms. For example, student feedback on assessment results can be posted on program websites; in the school newspaper; and on Facebook, blogs, and Twitter, as well as presented at events students attend. In the We've Heard Your Voice section of the Lipka article, several universities indicated how they supply feedback on students’ survey responses. Techniques range from projecting assessment results prior to showing a movie at a student event to featuring assessment results in TV campaigns and newsletters and on flyers.
Consider feedback as an integral part of the assessment process. Use feedback throughout the assessment cycle to improve programs and to solicit input on how to be more effective in the delivery of services. Include students in the assessment process and use their formative feedback to help improve your programs. For example, in clarifying your mission, vision, and values, you can solicit student feedback through focus groups, interviews, and participation in meetings. Moreover, in the creation of your assessment instruments, sample a subset of the target student population to be assessed to establish if the instruments measure what you hope they do (face validity) and that the instructions on completing the assessment and actual questions/tasks are clear, specific, and understandable. Also include students in the interpretation of assessment findings and the development of recommendations for its use. By including student feedback in all phases of the assessment process, you are more likely to find students are engaged because they know their voices matter.
Even after identifying ways to expand your assessment repertoire, if you must survey students, consider some of these suggestions to respect students’ time and effort to maximize your data collection.
Collaborate with others in your efforts so that you do not survey the same students too often. For example, if New Student Programs is doing a survey on orientation outcomes, determine if it is possible to participate in their survey by adding a few questions. Also, check to see if there are other large-scale surveys being done on campus to avoid overlap on survey administration. Maintaining a campuswide assessment calendar would also be valuable. You do not want to go to the student “well” too often. At PSU, we contract with CampusLabs, formerly known as StudentVoice, to administer a central location for all of our surveys. Once a month, we can generate an Excel spreadsheet with a detailed listing of all surveys being conducted in Student Affairs. This has not only discouraged oversurveying, but has also encouraged data sharing.
Again, if you find that surveys are the most effective way to assess programs, we recommend that you consider keeping surveys short and doing the assessment more often but with fewer questions. Since students may not be interested in or concerned about our survey topics, we should not assume that they will complete a lengthy survey and/or respond thoughtfully to open-ended questions.
Present surveys in modes that consider how students best like to interact and that foster student engagement; for example, consider posting scaled or open-ended questions on Facebook, Twitter, or blog sites. Use technology to capture interest. Survey students by using visuals, such as the thumbs-up “like” icon on Facebook, or use technology such as iPods and iPads to capture student responses during an event. At PSU, we have a checkout program for iPods and iPads to allow Student Affairs staff to conduct short web-based surveys during events. We have found that students are engaged in this process and respond favorably to us “coming to them,” which is reflected in our increased response rates.
The final suggestion we would put forth is to control, as much as possible, rogue data collection. When deciding which programs and services to measure, select those that address program goals, those related to longer interventions, those with a greater number of participants, and those that use greater resources. If you have a number of activities related to one goal, you will not want to assess every activity related to that goal. When deciding which activities are most relevant for assessment, consider the following:
Length of intervention: When measuring program impact, typically, the longer the intervention, the greater the likelihood of achieving the outcomes for which we hope, especially learning outcomes. For example, if an outcome is increased student leadership skills, then a program that lasts one day versus two hours has a greater likelihood of increasing these skills.
Number of participants: All things being equal, if you have a program with 10 participants and a program with 100 participants (and they are the same length of time), then assessment of the larger program is recommended.
Cost/benefit: If you have programs and services that are particularly costly to implement (in terms of staffing, materials, time, etc.), then you would want to assess those to assure that resources are being used most appropriately and that you are getting the best return on your investment.
Time and effort invested: There are very real differences in the investment we make in assessment when we select quantitative or qualitative assessment methods. Quantitative data collection typically involves a larger number of participants, is efficient and of a shorter duration, and provides more breadth than depth, whereas qualitative data collection typically involves a smaller number of participants, is more time-consuming and of longer duration, and provides more depth than breadth.
You may want to consider some of our suggestions by asking the following questions:
What do you want to know about your students? Are you interested in whether they are satisfied with a service or activity? In what they have learned because of an activity in which they participated? In whether their behaviors or attitudes changed because of an activity in which they participated?
Once you answer the above questions, brainstorm all the ways to gather evidence that may answer your question(s).
Finally, consider, and then reconsider, if the survey you want to administer is the best way to gather evidence. Does a survey limit or expand your understanding of the impact of your programs and services? When we ask survey questions, we typically provide fixed response scales, thus limiting responses to that scale and limiting the range in which to respond. If a survey will not expand your understanding, then it does not provide the evidence you desire! As we mentioned earlier, a more appropriate method for gathering the desired evidence might be tracking of usage, review of existing documents and data, focus groups or interviews, observations, case studies, rubrics, portfolios, performance assessments, or one-minute assessments.