The why and how of Chacking Open and using Assessment Results

Abstract

EVERY SO OFTEN, I would catch myself gazing at the glossy, thick binders of data sitting on the shelf across my office, wondering if I should crack them open and see what treasures they held. Invariably and luckily, the phone would ring, an e-mail would arrive, or a colleague would come into the office and rescue me from that intimidating task. Those reports are worth thousands of dollars, so certainly my colleagues at other campuses were not as wasteful as I was, roundly ignoring such important information!”

So goes the typical guilty response to questions about how educators use their data from the National Survey of Student Engagement (NSSE). We're all extraordinarily busy. Making professional decisions on our own is challenging enough; adding a data dimension to our work can make it seem impossible to accomplish anything. Yet many campuses have successfully used data from NSSE and other assessments to effect meaningful improvement in students’ higher education experiences. To understand more clearly why some institutions were succeeding while others struggled to make sense of their data, last year we interviewed two dozen faculty and staff who were responsible for administration of the survey on their campus. All had roles in student affairs, because our goal was to understand better how NSSE data are used in that specific context.

The project managers we interviewed had successfully used strategies to make sense of NSSE data (although all believed they could have done even more with sufficient time). All shared a strong interest in using NSSE data to develop a richer understanding of how their institution could better serve students. All understood the difficulty of interpreting results influenced by so many variables. Rather than piling on new data analysis projects, successful institutions were able to integrate data analysis into larger campus initiatives, usually because they had already established the right atmosphere for those initiatives to thrive.

While there is no one-size-fits-all approach to using NSSE data most effectively, we believe that our findings offer a reasonable starting place for campuses interested in acting on their results. The five most transferable themes from our interviews are (1) collaborating and communicating about results, (2) using additional data for triangulation, (3) using data to learn more about students, (4) using data to demonstrate goal achievement, and (5) enhancing the first-year experience.

Collaborating and Communicating to Create a Shared Vision

THE THEME OF COLLABORATION was woven throughout almost every one of our conversations with NSSE users. While we uncovered many examples of fast, simple programming changes based on the most obvious NSSE results, we also found an overwhelming sentiment that if the data indicated the need for substantial change, many people had to be included in the project. Where collaborative relationships among faculty, student affairs educators, and institutional researchers existed, more meaningful data-based action was likely to result.

The institutional research (IR) office is typically pivotal in coordinating effective use of NSSE data; indeed, institutional research staff members usually hold primary responsibility for administration of the survey. Whether this office consists of one person or many, making sense of NSSE data—or data from any complex survey—requires the participation of someone who works with numbers every day. This person can help translate numbers into understandable information and assist in identifying important themes. Regardless of how user-friendly the NSSE reports and data summaries are, a trained eye can discern the less obvious patterns that may help improve an institution's approach to educating students.

While IR participation is critical to the task of understanding data, faculty involvement is also essential. The mandate for student affairs collaboration with faculty members has been clear for a long time, crystallized in recent years by recommendation in the American College Personnel Association's The Student Learning Imperative and the National Association of Student Personnel Administrators and the American College Personnel Association's Learning Reconsidered series (edited by Richard Keeling). The line dividing in-class and out-of-class instruction is blurry, so student affairs educators and faculty must understand each other's roles in order to fashion coherent educational strategies. Making effective use of NSSE data in student affairs requires faculty members who are willing to trust and understand the data, as well as to work with others to use the data. Institutions that best model this relationship find ways to get faculty invested in NSSE and ready to work with the resulting data before the survey is administered.

A culture of collaboration is not established through work around just one assessment measure. On the other hand, creating these partnerships does not have to take years. Institutions that reported the most success in using NSSE data took steps as simple as creating broad representation on assessment committees or ensuring that the NSSE project manager reported data to both academic affairs and student affairs.

These collaborative partnerships really begin to pay off after the data have been gathered and returned to the campus. Given a prior understanding of what to expect when the data arrive, faculty and student affairs staff can collaborate during retreats or simple debriefings to determine the best course of action in light of their institution's findings.

For example, on one campus, the project manager made personal visits to key faculty and administrators to discuss the data NSSE would provide and why those data could be particularly useful to them. Once he had data in hand, he created simple, attractive reports for each of those individuals to highlight results that pertained to their specific duties. The meetings and individualized reports built sufficient awareness to allow the creation of a new initiative that involved key faculty as well as IR and student affairs professionals.

Using Additional Data Sources to Corroborate and Deepen your Initial Impressions

SCHOOLS with assessment agendas generally use more than one method to examine student learning and the educational environment. Confirming that findings are consistent across multiple surveys and assessment meth-ods—a process called triangulation—is an important step, especially if data will be used to drive pivotal decisions.

Our study revealed that project managers crosschecked NSSE findings with data from other assessments. In addition, some users linked students’ NSSE responses to other student data such as grade point average (GPA) or residential status. This practice allowed institutions to conduct additional analyses to determine the extent to which engagement varied across groups.

Triangulation also makes data more meaningful by providing a context for NSSE results. In this era of concern about retention, the institution that cross-references NSSE scores with lower GPAs, quickly identifying gaps in student support structures, is taking supreme advantage of data it has already gathered.

All of the institutional representatives we spoke with conducted more than one survey each year. Some of the assessments, like senior exit surveys, were homegrown, while others were national instruments like the ACUHO-I/EBI Resident Assessment, Cooperative Institutional Research Program Freshman and Senior Surveys, and the Noel-Levitz Student Satisfaction Inventory. Tri-angulation assists institutions in better understanding student attitudes and behaviors and identifying trends. For instance, one institution compared its average score on the NSSE student-faculty interaction benchmark against sets of similar items on another survey. If responses on a NSSE item set were consistent with responses to similar questions on other surveys, project managers felt more confident in reporting those data and, more important, in advocating policy or program change as a result.

Learning more about who Students are, Based on their Survey Responses

THOUGH EDUCATORS work with them daily, students can seem like strangers. Their responses on survey items and scales provide insights that might have escaped notice otherwise. NSSE data allow institutions to gain a broad perspective on the undergraduate population as well as to monitor the engagement of specific groups of students and learn about individual students.

For example, student affairs staff at one institution identified individual students who might not be making the transition to college well by examining students’ responses on key survey items. One project manager indicated in an interview that NSSE data provided the campus with “a better sense of who is vulnerable to departure and a better basis to educate those involved with retention initiatives.”

Entire subpopulations of students, as well as individual students, may be retention risks. Some institutions monitor the engagement of specific groups of students to anticipate such risks. For example, one institution closely examined NSSE data on student athletes and traditionally underrepresented students; another school closely tracked its transfer students’ responses. As one data user explained, NSSE enables administrators and faculty members to study the overall engagement of students while providing insight into student subpopulations in specific areas, such as faculty-student interaction, advising, and academic challenge.

Another institution investigated whether students followed through with interests they expressed as first-year students. For example, one survey question asks about students’ intentions or experiences with study abroad programs. The results of that item revealed that the majority of first-year students planned to participate in study abroad but that only a handful of senior students had actually completed that experience. Even without any additional data analysis, this information gave faculty and staff some insights into students’ expectations and experiences and suggested a new direction for addressing the affordability and the effectiveness of the marketing of study abroad programs.

Setting your Goals and Knowing Whether you have Achieved Them

THE CONVERSATIONS with NSSE project managers that helped formulate this theme described three main NSSE uses: identifying strengths and weaknesses, forming objectives, and assessing progress toward goals. For example, based on NSSE data, the division of student affairs at one institution created goals for improvement in the areas of residence hall education, global citizenship, and diversity awareness. At the same institution, academic administrators similarly used NSSE data to generate goals for general education reform, spirituality education, computer literacy, diversity activities, and academic rigor. Another institution used NSSE to assess its living-learning communities and first-year experience programs by oversampling students involved in those initiatives.

To maximize the effectiveness of NSSE or another survey, schools can use the instrument during each step of the process of setting goals and determining whether the goals have been met; the institution that used NSSE data to identify weaknesses in faculty-student interaction epitomizes this practice. Faculty and student affairs educators brainstormed about ways to increase the number and quality of faculty interactions with students, implementing the most promising strategies based on results from relevant NSSE items. The current year's data were used as the baseline, and the campus will analyze the next year's data to determine whether the adjustments were effective.

Enhancing the First-Year Experience

THE CRUCIAL NATURE of the first year of college in accomplishing desired outcomes in student achievement, critical thinking, social development, and persistence is well understood. NSSE data help to track all of these outcomes, and the survey has become a popular tool for improving institutions’ understanding of their students’ first-year experience.

For example, many institutions have used NSSE to assess their first-year students’ satisfaction against the intended outcomes of their first-year experience initiatives. While some of these institutions have been pleased to see their NSSE scores improve over the course of two or more administrations of the survey, others have found their scores consistently lower than they had hoped. Having identified the items or scales on which they underperformed, these institutions were able to trace low scores back to such issues as insufficient rigor in first-year coursework, lack of cocurricular opportunities, and inadequate new student orientation.

Faculty interaction is a key contributor to the college student experience. Establishing this interaction in the first year is crucial. NSSE data include a benchmark that indicates the extent to which first-year students work with faculty in formal and informal contexts. One institution with low first-year scores on this scale revamped its summer and early fall orientation programs to ensure better faculty participation. In addition, faculty members teaching first-year seminars were given more latitude in creating their syllabi.

Implications

MANY academic and student affairs educators are intrigued by NSSE data but intimidated by the time or effort involved in data analysis. The most valuable lesson we learned from our interviews is that anyone can understand and effectively use NSSE results. Whether or not your institution is a NSSE participant, it has student-level data that can inform everyday practice, instruction, and planning. Sharing these data with stakeholders is the first step in making them useful.

We were inspired to learn from so many colleges and universities about their efforts to gather evidence to improve undergraduate education. At the same time, we learned that many more campuses can and should be using their data more effectively. Our hope is that more institutions will open their glossy binders, dig into their reports, and take the first steps toward converting data into action.

References

American College Personnel Association. (1994). The student learning imperative: Implications for student affairs. Washington, D.C.

Keeling

R. P.

(ed.). (2004). Learning reconsidered: A campus-wide focus on the student experience. Washington, DC: National Association of Student Personnel Administrators & American College Personnel Association.

Keeling

R. P.

(ed.). (2006). Learning reconsidered 2: A practical guide to implementing a campus-wide focus on the student experience. Washington, DC: American College Personnel Association, Association of College and University Housing Officers-International, Association of College Unions-International, National Academic Advising Association, National Association for Campus Activities, National Association of Student Personnel Administrators, & National Intramural-Recreational Sports Association.