Maximizing Campus Responsibility for the Writing Assessment Process

EFFECTIVE WRITING is a typical goal in undergraduate education, especially in general education programs. In a survey recently conducted at the University of South Florida, one faculty member said, “I responded [to this survey] because I strongly believe that “writing skills are essential for all college students regardless of their major field of study. If they cannot communicate clearly, how can they be successful? “This attitude is shared by employers and faculty alike.

At the University of South Florida, as at many institutions, “writing proficiency has become a concern that challenges both faculty and administrators. Reduced resources and increasing enrollment have challenged our faculty's capacity to assign student writing that requires the drafts and revisions we know are necessary for improvement. Even “with these challenges, we expect our students to be able to “write “well-organized, well-developed, and reasoned papers that address audience and purpose through the use of appropriate language.

Some students’ resistance to “what they perceive to be too much “writing further complicates the issue. Casual faculty conversations around campus suggest that students believe writing should be required only in En glish classes and that students do not fully recognize the importance of “writing to their lives. Science faculty often report that students do not seem to take “writing assign ments seriously or complain “when “writing is assigned. More than one science faculty member has reported that students exclaim, “This is not a “writing class!” “when “writing is assigned. Similarly, one adviser reported an increase in the frequency of her students dropping courses “when they perceive the “writing load to be excessive. Students’ writing ability is yet another complication, with faculty survey respondents reporting problems “with students’ “writing that large class sizes make difficult to address. Past “writing assessments have, indeed, revealed that our students were not reaching institutional expectations. To meet the myriad challenges in “writing instruction, we determined that the entire academic community must be involved in finding a solution rather than relying solely on the English department to improve our students’ skills.

This is the story of the University of South Florida, a research-extensive metropolitan institution “with enrollment of more than 41,000 students that took the bull by the horns to address an educational issue that is critical in many colleges and universities. The lessons we have learned may help others in their efforts to assess and improve these important skills.

When we launched our initial “writing assessment efforts several years ago, our strategies looked much like a standardized test, “with students “writing on a specific standard topic under timed conditions at particular points in the curriculum. We employed raters external to the university “who also scored essays for the state's rising junior examination. We followed standardized scoring procedures and assigned scores from 1 to 6, ranging from “little command of the language” to readiness to “progress to upper level”—the last two years’ coursework.

Our assessment practices “were acceptable (some would even say admirable), but we “were not satisfied. We knew that some students “were more than satisfactory “writers, but many more “were not. Even after using this approach for several years, we had summative or product results but little formative or process information to help us address specific “weaknesses in student “writing. In other words, we “were able to assess students’ “writing proficiency levels, but we could not identify the elements of writing that were deficient—the crucial information for making programmatic changes. While knowing the achievement levels of our students is necessary, assessment's major contribution to learning is providing the information needed to enhance achievement levels. Our assessment process “was further flawed by its lack of inclusion of our faculty.

A university assessment committee “was formed to improve the process, to add more representative voices, and to increase awareness of our assessment efforts. The General Assessment Advisory Committee, composed of faculty from all colleges, administrators, institutional research representatives, advisers, and graduate and undergraduate students, agreed that we should maintain our assessment focus on “writing, but believed that we needed new methodologies. The committee adopted an instrument that had been developed five years earlier, the Cognitive Level and Quality of Writing Assessment (CLAQWA), “which “was initially created in response to needs identified in a learning community program. Additional information about this instrument is available at http://www.usf.edu/ugrads/CLAQWA and in a 2003 article I co-authored “with Elizabeth Metzger. In the learning community program, faculty from multiple disciplines team-taught cohorts of fifty students, “who completed most of their general education coursework together over a two-year period. Before the CLAQWA was adopted, program faculty graded “writing, “with considerable variation across courses. Along with offering a more consistent approach to “writing assessment, the instrument “was intended to measure student cognitive development.

OVER THREE years and in collaboration with faculty members, teaching assistants, and measurement specialists, Elizabeth Metzger (a member of the Department of English and the learning community coordinator) and I reworked the two-part CLAQWA instrument. The CLAQWA provides faculty whose discipline is not English with a method for assessing papers on seventeen skills typically considered important for effective “writing. It encourages faculty users to consciously consider the cognitive level expected for an assignment, provides a tool for students’ self and peer review and for understanding faculty feedback, and facilitates a multidisciplinary approach to “writing assessment.

After refinements, the CLAQWA instrument “was introduced into assessment of general education courses three years ago “with a “writing scale of seventeen items organized into five major categories:

Each skill within the categories is assessed along a five-point continuum. Thus, results can be skill-based or category-based.

CLAQWA's cognitive scale, based on the “work of Benjamin Bloom and his colleagues, provides a means to assess the cognitive level reached as evidenced in students’ essays. The essays are assessed according to the four levels of the cognitive scale: level 1 = knowledge; level 2 = comprehension; level 3 = application; and level 4 = analysis, synthesis, and evaluation. Within each cognitive level, we further assess low, medium, and high levels. For example, although one of our prompts is “written to elicit level 4, if a paper is judged to be level 2 (comprehension), the paper falls below expectations.

Use of the CLAQWA instrument is now cam-puswide and involves teaching assistants, undergraduate students, faculty, and administrators. Assessment is an institutional endeavor, “with different teams and committees involved in multiple assessment activities. The General Assessment Advisory Committee assists in all phases of assessment, such as selecting and developing instruments, analyzing results, and suggesting strategies for program improvement. The members also have been instrumental in communicating results to their colleagues. In addition, a team of graduate and undergraduate assistants representing many disciplines and majors has been trained to conduct assessments of student “writing each semester using CLAQWA and other assessment techniques. We “were gratified to discover that undergraduates assign scores as consistently as more experienced graduate students. Meeting “weekly, “writing team members score assignments completed at three key points in the curriculum “with the CLAQWA instrument. Rescoring previously read papers at the beginning of each scoring session helps achieve consistency over time and agreement among scorers. If a scorer is unable to shed his or her bias or cannot interpret the scale consistently with all other scorers, we dismiss the scorer, since reliable use of the instrument is critical to effective assessment. When the two scorers assessing each essay assign scores varying more than .5 on the five-point scale, they further examine the paper, discuss the discrepancy, and adjust one or both scores.

Two other teams also assess the same essays; in both, two faculty members score each essay. One faculty team uses the CLAQWA cognitive scale to determines the cognitive level reached as evidenced in students’ essays. The other faculty team assesses students’ intellectual development levels as reflected in essays, based on the “work of William Perry. Trained in the use of Lee Knefelkamp, Carol Widick, and William Moore's Measure of Intellectual Development (MID), another faculty team scores essays to determine the intellectual development level exhibited. The MID reflects students’ development as they progress from a dualistic perspective focused on acquiring the truth from authorities, through multiple perspectives in “which knowledge becomes increasingly uncertain and students take increasing responsibility for learning, to a contextual perspective in “which students are self-reflective and self-motivating, value others’ perspectives, know the difference between opinion and supported opinion, and think in context.

WHILE MANY of our students arrive at USF expecting to passively receive information from experts that “will be relevant to the next test, we hope to cultivate individuals who can think in context, who value multiple perspectives, and who are self-motivated learners. By scoring papers with the MID, we are able to assess this growth. Through one assignment, then, we have been able to assess writing proficiency and cognitive and intellectual development levels and gather information on favorable learning environments and where deficiencies in the curriculum may exist.

Our assessments have revealed two areas for improvement: our students’ written ideas have been underdeveloped, and the details they supply have not been consistent with the main ideas in their essays. These and other results have guided general education curriculum review and reform decisions. The university's General Education Improvement Committee, formed two years ago to review the general education program, responded to our assessment discoveries by revising the curriculum to reemphasize writing and thinking. The revised curriculum includes not just more writing but writing to facilitate learning and reader-centered writing in which feedback and revision beyond editing are major components. A newly required capstone course will stress writing within a student's discipline. In addition, graduate and undergraduate students will be trained to assist faculty members in our ever-larger courses by assessing writing and providing feedback to other students.

The process of using CLAQWA to assess writing has had a variety of instructional implications. For example, faculty scorers have begun using the instrument to score assignments in their classes and have initially received positive responses. They consider the cognitive levels in their assignments more carefully than previously, and they better understand areas to target in teaching student writing. In addition, many faculty members report improvements in their own writing.

Our focus on writing assessment has catalyzed other campus writing efforts. The Department of English is systematically introducing the CLAQWA instrument in first-year composition classes in an effort to increase consistency in assessment across on-line and face-to-face classes. The College of Engineering is planning to train teaching assistants to assist faculty with assessing writing and providing feedback to their students. Because of increased participation, more faculty and administrators support the process and value the results. In fact, faculty members are requesting that we assess writing in their classes and are asking us to speak with their students about their writing performance.

Our assessments have revealed that students value learning at deep levels and prefer teaching strategies that promote thinking. They appreciate rigor, provided that it enhances their learning. There are significant contrasts between our earlier and current writing assessment approaches; the new approach reaps substantial benefits that the earlier approach did not provide. Our current process is more participatory and, at the same time, efficient. Through assessment, we are learning as much from our students as we hope they are learning from us.