Engaging Employers in Assessment

Abstract

The authors describe a pilot project in which formative feedback from industry professionals creates a genuine learning experience for students.

EDUCATION, like this country, faces unprecedented challenges. In their paper America in the Global Economy, Ray Uhalde, Jeff Strohl, and Zamira Simkins report that globalization has created a market in which U.S. graduates will be competing with 4.5 million highly skilled young professionals from twenty-eight low-wage countries. In higher education, awareness of this competitive situation has resulted in a vigorous press for accountability and concomitant calls to raise standards. Cross-institutional comparisons and standardized measures like the Collegiate Learning Assessment have gained traction, and initiatives that support comparisons among institutions such as the Voluntary System of Accountability (VSA) have garnered some attention. The VSA was developed through collaboration on the part of higher education professional organizations that include the American Association of State Colleges and Universities and the National Association of State Universities and Land-Grant Colleges (http://www.voluntarysystem.org/index.cfm). The VSA was largely a response to the Spellings commission, which was the Department of Education's response to the perception that the United States has been losing ground to other countries and that a renewed focus on accountability would be an essential way to meet the global challenge. At the same time, Transparency by Design, a response similar to the VSA, was launched by distance educators led by the Presidents’ Forum at Excelsior College (http://presidentsforum.excelsior.edu/projects/transparency.html).

Raising standards by employing standardized measures and cross-institutional comparisons, however, may not provide the practices and measures that meet the needs of employers or society. Jonathan Kozol, lamenting the failure of the long history of standardized measures to lead to improved learning on a CounterSpin radio broadcast, suggests why: “Weighing the lamb doesn't fatten the lamb.”

The persistent disconnect between cross-institutional comparisons, standardized measures, and student learning has been further explicated in Hart Research Associates survey of employers commissioned by the Association of American Colleges and Universities in 2008 that reports employers’ general disinterest in these comparisons and measures. Underscoring Kozol's wry caveat, the Association of American Colleges and Universities survey reports that a majority of 301 employers say they have little confidence in transcripts, tests, or cross-institutional comparisons as helpful predictors of career success. Those measures don't provide evidence for what graduates can do. The common weights of institutional and student success like “enrollment persistence, and degree attainment” are no longer adequate documentation (p. 1). Employers report a pointed frustration with recent graduates’ lack of “broader skills,” including communication, collaboration, and problem solving. Those “360 degree skills” are what employers deem “necessary to promise greater success for both the individual and their employer” (p. 1). Further, employers recognize that traditional assessments are unlikely to promote development of such skills. More useful, they suggest, are “internships, community-based projects, and senior projects” (p. 1).

Employers are not waiting for educators to establish useful outcomes assessment. According to Paul-Baskin's article in The Chronicle of Higher Education, Boeing has implemented an internal ranking system that tracks employees back to their colleges. It is not a coincidence that Boeing's senior vice president, Richard D. Stephens, was a member of Margaret Spellings's Commission on the Future of Higher Education, a panel formed by Spellings, the U.S. Secretary of Education, in 2005. Baskin quotes Stephens, who explains that his experience sitting through many meetings of the commission “listening to long discourses by education professionals” helped him realize that Boeing “would have to take action on its own.” Boeing's effort, Stephens says, is “really about improving the dialogue on curriculum, performance, and how we can build a stronger relationship between the colleges, universities, and us, because, ultimately, their students become our employees.”

Accreditors are also questioning the value of standardized tests and cross-institutional comparisons. In September 2008, HEA Update from the Council for Higher Education Accreditation noted that for the first time, accreditation “may include different standards for different institutions or programs, as established by the institution.” The recognition of institutional diversity is a recognition of the need to individualize, deepen the conversation among stakeholders, and do more to promote activities like internships, community-based projects, and senior projects—activities that faculty, students, and employers value but that elude traditional measurements.

Assessment that will meet needs of the twenty-first century requires a broad new approach to collaboration. As Peter Ewell of the National Center for Higher Education Management Systems argues, the vision for assessment has always “demanded” three “intertwined and inseparable” things: “collective and detailed vision of attributes and abilities,” “coherence and connectedness” and “defensible evidence” (p. 1, italics added). Implementation of the vision, it is clear, will not be possible without engaging stakeholders—employers, faculty, and students. To that end, communication technologies are indispensable.

The work cannot rely on the traditional and exclusive labor of educators (and their assistants) alone. Accreditors repeatedly remind classroom faculty that accountability expectations are changing and that grades, if they ever were, are not now sufficient for meeting accountability requirements. It is a caution that is in part recognition that the isolated perceptions of any single group—even a group of expert educators—will not satisfy the many stakeholders who are invested in higher education. In classroom terms, grades (faculty judgment) are not sufficiently grounded or validated. Including professionals from the field in assessment practice, however, is valuable for more than simply validating an assessment; the opportunities to build robust community have additional explicit and implicit value. Professional participation in assessment activities can be mediated to introduce students—and educators—to the discourse of the community of professional practitioners, even when the field of study is not directly vocational or professional. And the reciprocal benefits that professionals gain from working with the academic community—the opportunity to reflect upon their own professional development while guiding and shaping the future leaders of the profession—are also a mul-tifaceted opportunity to add value to the educational enterprise. Further, as John Dewey called for more than a century ago, collaborating with professionals provides a vital way to reclaim education's shaping function in society.

Finally, students themselves are key stakeholders in building a broader community around issues of assessment. As Grant Wiggins argues in his 1993 book, Assessing Student Performance: Exploring the Purpose and Limits of Testing, learners gain substantially just from the demystification of academic expectations as well as from engaging in guided self-assessment.

The Challenge of Educative Assessment

ENGAGING A BROADER COMMUNITY of stakeholders also aligns with a transformational view of learning that places the student at the center of the learning experience. This learning-centered view is readily supported by new technologies, including electronic portfolios, personal learning environments, and similar applications, as described by Martin Weller and by Gary Brown and Nils Peterson. However, implementing these innovations requires a radically different set of assumptions than the traditional approach to school. The new context for learning is global, technology-rich, and ever changing. To meet these challenges and maintain the relevance of higher education, we need a new set of guiding principles that are supported by research. In the book How People Learn, John Bransford, Ann Brown, and Rodney Cocking synthesize research from multiple academic disciplines, including neuroscience, that can help guide these new principles. They highlight the importance of students’ monitoring their own learning goals, strategies, and learning progress and of the untapped potential of technologies to facilitate that monitoring, or metacognition. Their research also points to the need for education to focus on learning-centered strategies that help learners understand and develop competence and understand what mastery means. They emphasize the importance of “assessment-centered environments” (p. xvi) and strategies that promote “a community-centered approach” (p. xvii) that capitalize on “connections to the outside world” (p. 49).

In Donald Finkel and Stephen Monk's classic 1983 essay “Teachers and Learning Groups: Dissolution of the Atlas Complex,” the authors identified the Atlas complex of the title as the belief of many educators that they are solely responsible for teaching students. Such a perspective is not sustainable, according to Finkel and Monk, and it is not effective. They elaborate on the wealth of teaching resources students live among—most notably, their peers. As Grant Wiggins argues on the New Horizons for Learning Web site, “Our challenge as educators is to think of assessment as first and foremost educative,” adding, “Our aim must therefore be to create assessments that provide better feedback by design, and not think of improvements in terms of more accurate evaluation.” Suggesting that feedback needs to be understood more broadly, Wiggins quotes Peter Senge, who defines it in learning organizations as “a reciprocal flow of influence.”

From this discussion of accountability, assessment, and feedback, we have formulated Table 1 as a way to outline the distinctions that define and guide our approach to the challenge of improving feedback, assessment, and accountability.

Table 1

Traditional and Transformed Views of Teaching and Learning

Traditional View	Transformed View
The institution and its faculty are the central node.	Learner is the central node; learners include all members of the discourse community (or community of practice).
Faculty member is the grading authority.	Community of practice holds responsibility (and the social capital) for assessment, and assessment (from the roots ad and sedere, “to sit with”) is most useful when it is formative, when it is understood to be constructive feedback rather than an authority's judgment.
Faculty member defines the assessment criteria.	Community of practice, by virtue of expert consensus, validates the assessment instrument.
Feedback to the student about learning criteria is masked by the letter grade.	Students merit direct and unfiltered feedback from the community using criteria that the community articulates.
The institution is the established (credentialing) authority.	Community of practice is the implicit credentialing authority; the university is the facilitator of that credentialing and of community building.
The unit of analysis is the course, bounded in time, content, and brick or virtual space.	The unit of analysis is the problem; problems are not bound in time or content.
Problems for students to study are artificially constrained by course boundaries.	Communities identify authentic problems that reach beyond the definition of the course.
Students rarely share their work with or receive feedback from public audiences.	Electronic portfolios built over multiple years are owned by learners (not the university) and used to communicate (collect, select, connect, project, and reflect) with and get feedback from wider communities.
Faculty member is a gatekeeper between the students and the community of practice.	Students are joining communities of practice. Faculty may introduce students to the community.
The faculty member has all the social capital within both the classroom and the community of practice.	Learning is social; therefore, learners are building social capital in the community of practice.

The Cross-Stakeholder Approach

INDUSTRY is giving feedback to higher education that many graduates are not employable or that they need substantial postcollegiate on-the-job training. Higher education is struggling to identify criteria by which to ensure that graduates contribute to society. In these two conversations, industry wants employability without unpacking that into criteria. Higher education has developed criteria, such as tests and rubrics, without evidence that such measures correspond with success beyond college.

At Washington State University (WSU), a field experiment and pilot project explored the proposition that a community of students, educators, and industry professionals providing feedback based on explicit criteria will improve learning outcomes in ways that meet accreditation guidelines and align with professional employability. Further, the project examined how such a community of practice collects and shares data to inform their conversation and help to demystify expectations and criteria for all stakeholders. Working with a design team from Washington State University's Center for Teaching, Learning, and Technology, a faculty member from the university's Apparel, Merchandising, Design and Textiles Department interested in promoting authentic learning opportunities enlisted her professional colleagues in industry to join in the project and the study. Findings from this work have implications for educators and learners in and outside formal classroom settings and for a broad range of constituents in and outside the academy. These findings have much to say to those educating for, working in, or preparing to work in settings far beyond apparel, merchandising, design, and textiles.

The Project

EIGHTY-SEVEN STUDENTS enrolled in a market forecasting class, working in fourteen teams, were asked to use blogs to create an electronic poster for a portfolio presentation of their market forecasting project at midterm for formative feedback, and then they were asked to revise the electronic poster as a final project. No particular set of technologies was required. The students were encouraged to use and were capable of navigating an array of Web 2.0 technologies and negotiating within their groups to settle on tools. (For a longer description of Web 2.0 technologies, see Peter Magolda and Glenn Piatt's About Campus article “Untangling Web 2.0's Influences on Student Learning.”) The project required the students to engage in substantial market forecasting research on fashion trends, an activity that required skill development in critical thinking, information literacy, teamwork, and communication. The project engaged the teams throughout one semester. Six industry professionals representing companies with global distribution networks participated in assessing the students’ work. These industry participants were located on the West Coast and East Coast of the United States, Africa, and Europe (Paris). In addition to the course instructor, seven other faculty members in the program agreed to review student projects and provide feedback. Students used the same criteria as faculty and professional participants to conduct self-evaluations and peer evaluations. The criteria were developed from a rubric that was based on WSU's critical and integrative thinking rubric (http://wsuctproject.wsu.edu/ctr.htm), which uses seven dimensions, or integrated performance criteria (including issue identification, context and assumptions, evidence, and conclusions), and a six-point assessment scale that ranges from “absent” to “mastering.” A score of 4—“developing”—or better on the rubric was presented to the students as necessary for entry level into the apparel merchandising profession. In addition to assessing student learning outcomes, two additional questions were asked of employers to determine, first, their perception of the overall employability of students based on their projects and, second, to ascertain changes in employers’ perception of the criteria as they gained experience in responding to student work.

Widely Divergent Views across Stakeholders

STUDENTS ASSESSED each other's performance as, on average (mean score 4.45), employable. (A score of 4 was deemed employable.) They assessed their own work as even more so (mean score 5.11). By a small margin (mean score 4.01), faculty in the program agreed that students were employable. Only employers gave performance ratings on critical thinking that indicated that the students were not ready for independent employment (mean score 3.75).

Ratings of student projects were statistically significantly different across all groups. Rather than being disheartened by these divergent findings, we see in those differences a rich and potentially productive opportunity for conversations about them across stakeholder groups. We believe that working to encourage consensus about direct measures of learning is critical in building bridges between stakeholder communities and provides the context in which this study was conducted.

The agreement within groups is also interesting. There were marginally acceptable levels of agreement among industry professionals at 65 percent, with half of the projects receiving ratings that had better than 70 percent agreement (the conventional minimum). The marginal level of reliability among employers may result from the novelty of the procedure and the lack of calibration or formal norming of the rating of student work using the rubric. In contrast, students were remarkably aligned at 80 percent agreement. The level of agreement among students is notable and merits follow-up in the future. Faculty ratings were, unfortunately, underrepre-sented because there were too few projects with multiple ratings to provide measures for analysis.

Shaping Learning of Students and of their Potential Employers

IN ADDITION to using the rubric scale, we also solicited feedback on the employability of students on the basis of the evidence of their projects. We asked industry professionals twice—at midterm and then after the final project—to identify whether and at what level they would hire the various teams on the basis of their presentations. Though the small number of teams and the open-ended question did not support significance testing, there were gains over the semester. From midterm to the final assessment, there was a positive shift from five teams at midterm who were rated by industry as “not ready for employment” and five rated as ready for an “internship with close supervision” to only one team that was judged to be “not ready” and two teams as ready for “internship with close supervision” by the end-of-term assessment. Similarly, at midterm, three student teams performed at levels deemed ready for independent internships and by the final, that number had increased to four teams. The number of teams ready for full entry-level positions moved from three to five during the same period. Ten of fourteen teams demonstrated concrete gains, according to those who were in a position to actually hire them in a very competitive field. (In fact, one employer expressed interest in doing so.) Not only did the project demonstrate gains in student learning, but it seems to have changed the way employers view the kinds of criteria they would use to assess the work of their potential employees. In answer to a question about their perception of the importance of analytical reasoning as a hiring criterion, industry partners indicated an increased perception that this intellectual skill is, indeed, critical. Industry professionals’ perceptions confirmed what educators and students assume, which is that the criteria used to assess students align with skills essential in graduates’ future professional employment. Working with a common rubric corresponds with greater appreciation of the utility of formally articulating criteria.

Differences in Student Evaluations across Groups

IT IS PROBABLY NOT SURPRISING to find that students’ sense of the merits of their own performance exceeds their perception of the merits of their peers’ performances and even less surprising that their judgments of each others’ performance are, in the eyes of faculty members and employers, still inflated. More interesting and indicative of the challenge of assessment is the significant difference between how faculty mem- bers and industry professionals scored students’ performances. What is less clear is the extent to which the improvements noted reflect changes in performance or movement of perceptions, as the scoring procedures of different groups of stakeholders were not rigorously calibrated in ways consistent with formal assessment procedures. Furthermore, it is not clear whether the nature of feedback—either its quantity or its quality— corresponds to the ratings in any revealing way. To that end, we conducted three follow-up analyses to illuminate the differences.

We examined the differences in the quantity of words used to guide or respond to student performance that were provided by the three groups—faculty, industry professionals, and students. The frequencies suggest interesting distinctions between stakeholders, distinctions that suggests different qualities of engagement. Employers provided the greatest quantity of words of all stakeholders, which suggests that they are a potentially rich and valuable collaborative resource that might contribute to the scalability of this approach to engaging distributed stakeholder and harvesting feedback, a way to counter the Atlas complex. We also extended the exploration of the quantity of participation by calculating the number of words per comment. A high percentage of faculty members responded to student work with fewer than twenty words per comment. It appears that faculty in this study were not only more parsimonious in their feedback but also less likely to respond to student work at all (perhaps an indication that they may not suffer from the Atlas complex after all?). Peers and employers were much more likely to respond more often, and when they did, they were more likely to provide a greater quantity of feedback.

Of course, there is no evidence to suggest that quantity associates with the quality or utility of a response. There is no reason to suspect that saying nothing is not appropriate in many instances, especially considering that the faculty lead's project goal was to promote engagement among employers and students. In fact, counter to the Atlas complex, Wiggins suggests that feedback need not be labor-intensive in his 2004 entry on the New Horizons for Learning Web site. He says, “A common misconception about feedback in schools is that it is impossible to provide enough of it because good feedback seemingly requires intensive one-on-one tutorials. But much important feedback is derived from situational information in response to trying to accomplish a task. The challenge of designing learning, in fact, is to make it possible for students to self-assess and self-adjust effectively, with minimal intervention by the teacher. Put another way, instructional design is the art of maximizing self-directed learning and useful information from the situation.”

To get at the issue of quality, we conducted a tag cloud analysis on the text of the feedback in order to provide some preliminary insight that would help sharpen this complex issue. Tag clouds count word frequencies, and the more frequently words are used, the larger they are rendered in the cloud. Figure 1 makes apparent the different character of the language used by each group.

Figure 1

TAG CLOUDS FOR STUDENT EVALUATIONS BY VARIOUS STAKEHOLDERS, INDICATING FREQUENCY OF WORD USAGE

The tag clouds are revealing. At first blush, the tag clouds demonstrate some degree of adoption of the language of the rubric across groups. Industry experts, faculty members, and students frequently use the word evidence, for instance. Scrutiny of the tag clouds, however, reveals some subtle but distinct differences between groups. Note the faculty use of problems, while the employers’ focus is on problem. We point to this distinction as emblematic of a potentially key difference and one that may illuminate the challenge of cross-stakeholder assessment. The faculty references to problems are pointers to the many aspects of students’ projects that are incorrect (also signaled in the cloud of faculty discourse). The feedback that faculty members give to students is, we suggest, more atomized, more focused on errors, focused on, metaphorically, the trees. The attention to corrections is often, as Rich Haswell noted, “analogous to that of the teaching of grammar in composition courses—hundreds of thousands of hours spent, and being spent right now, on a task of little proven benefit.” Alternatively, employers tend to look at the forest, the larger problem that students have addressed in their projects. Both industry experts and students use the words market and perspectives. Faculty don't. (In fact, the only faculty comment related to “other perspectives” was concern that students’ consideration of other views might be “irrelevant.”) Like students (and unlike faculty), industry professionals frequently discuss context and data. Like the implicit pedagogies that promote what employers value—communication, collaboration, problem solving—the big picture focus of employer feedback generally aligns with what research by John Bean; Grant Wiggins, in Assessing Student Performance: Exploring the Purpose and Limits of Testing; John Bransford, Ann Brown, and Rodney Cocking; and Ernest Pascarella and Patrick Terenzini; and others has identified as good teaching practice.

The faculty stress on correctness, however qualified by the fact that this is a small study, hints at the deeper problem of the Atlas complex. It reflects a habit of providing feedback to students that, as Haswell notes, is difficult to scale upward. Fatigue may very well be one reason that faculty comments are under-represented in this study. The “too busy” lament may mask a “too tired” reality. More important, Atlas-related fatigue also delimits faculty's ability to engage themselves and their students with the larger problems facing the profession. To what extent does that disengagement feed the ubiquitous perception among students and others outside the academy that the world of higher education is not the real world? Combing through student work in order to highlight errors is, for most, a tedious and exhausting practice. Focusing on ideas is more invigorating, as indicated by the positive response from employers in this study. In addition, focusing on correcting student work, however much it may be deemed necessary, also promotes what Jerry Farber identified back in 1969 as “authority addiction.” It is not incidental that in spite of the gains that students demonstrated in this study, student evaluations of the experience were distinctly bi-modal Students praised the course experience or panned it, with little in between. Student resistance to innovation is well documented by Sahana Murthy and Maria Ruibal. That authority addiction, we argue, lies at the heart of the difficulties many students identify with innovative teaching practices like those in the project reported on in this article. Such innovations press students to assume greater responsibility for learning. In other words, in spite of the reported purpose for getting an education as a way to prepare for careers and the seemingly obvious contributions such activities might hold to that end, students are addicted to the authority of the instructor, a symbiotic dependency that contributes to the status quo.

Building Consensus among Diverse Participants

THE LEARNING GAINS reflected in the increase in the number of students whose work met professionals’ expectations for internship, entry-level, and even leadership positions confirm the potential of this approach. Even if the gains are artifacts of the changing perceptions of industry professionals, which we doubt, the project and process demonstrate a budding and vital collaboration.

Other implications are worth noting. As Wiggins has pointed out, there is a danger when we conflate high standards with standardization. This process illustrates how students can be held to high and authentic standards while working on diverse teams doing diverse projects in diverse ways. As they would be in the real world, students in this project were responsible for constructing their own understanding of professional success and, with input from peers, faculty, and industry professionals, realizing that success. At the same time, industry participants reported gains in their own understanding of the criteria associated with that success.

Implications for Accountability

THIS STUDY was situated in the context of global competition and accountability. In that context, we suggest that the more salient distance in distance education may be not the space between faculty and students but the gulf between school and real communities, real issues, and real feedback. Engaging stakeholders in establishing and validating a shared discourse about principles of quality provides a rich and useful way to distribute the legitimate need for greater accountability in education. Strategies for continued improvement emerge clearly and are available immediately to those—faculty, employers, students—in positions to do something about what the assessment reveals. Responsible accountability is about improvement, not measurement. Furthermore, unlike standardized measures that abstract learning and separate responsibility for improvement from the assessment process, by leveraging the insights and energy of key stakeholders, we can capitalize on the wealth of diverse viewpoints and advance the urgent message that teaching and learning is everybody's business.

What is less clear, yet no less exciting, is how the process used for this project begins to assuage Dewey's fear that education might fall to only a reflecting role and forfeit its function of shaping society. Though this process challenges traditional faculty, student, and even employer roles in promoting relevant learning, it begins to break the fall.

References

Association of American Colleges, and Universities. (2008). College learning for the new global century. Retrieved October 31, 2008, from http://www.aacu.org/leap/documents/GlobalCentury_ExecSum_3.pdf.

Baskin

(2008, September 19). Boeing to rank colleges by measuring graduates’ job success. Chronicle of Higher Education. Retrieved September 19, 2008, from http://chronicle.com/weekly/v55/i04/04a00102.htm

Bean

J. C.

(1996). Engaging ideas: The professor's guide to integrating writing, critical thinking, and active learning in the classroom. San Francisco: Jossey-Bass.

Bransford

J. D.

, Brown

, & Cocking

R. R.

(2000). How people learn: Brain, mind, experience. Washington, DC: Commission on Behavioral and Social Sciences and Education of the National Research Council & National Academy Press. Retrieved March 3, 2008, from http://www.nap.edu/openbook/0309065577/html/index.html

Brown

, & Peterson

(2008). The LMS mirror: School as we know IT versus school as we need IT and the triumph of the custodial class. Journal of Online Learning and Teaching, 4(2), 190–197. Retrieved from http://jolt.merlot.org/vol4no2/brown0608.pdf

Council for Higher Education Accreditation. (2008, September 19). Accreditation and the Higher Education Opportunity Act of 2008. HEA Update, no. 45. Retrieved October 27, 2008, from http://www.chea.org/Government/HEAUpdate/CHEA_HEA45.html

Dewey

(1897, January 16). My pedagogic creed. School Journal, 54(3), 77–80.

Ewell

(2004). General education and the assessment reform agenda. Washington, DC: Association of American Colleges and Universities.

Farber

(1969). The student as nigger: Essays and stories. North Hollywood, CA: Contact.

10.

Finkel

D. L.

, & Monk

S. G.

(1983). Teachers and learning groups: Dissolution of the Atlas complex. In Bouton

, and Garth

R. Y.

(Eds.), Learning in groups. New Directions for Teaching and Learning, no. 14. San Francisco: Jossey-Bass.

11.

Hart Research Associates. (2008). How should colleges assess and improve student learning? Employers’ views on the accountability challenge. Washington, DC: Association of American Colleges and Universities. Retrieved May 28, 2008, from http://www.aacu.org/leap/documents/PrivateEmployersFindings.pdf

12.

Haswell

(1983). Minimal marking. College English 45(6), 166–170. Retrieved March 3, 2009, from http://www.lagcc.cuny.edu/CTL/dfl/dil0304/midyearinstitute/HasweU%20Minimal%20Marking-ADV%20Set%2022.htm

13.

Kozol

(2006, January 6-12). Shame of the nation: Restoration of apartheid schooling in America. CounterSpin broadcast. Retrieved March 3, 2009, from http://www.fair.org/index.php?page=2791

14.

Magolda

P. M.

, & Platt

G. J.

(2009). Untangling Web 2.0's influences on student learning. About Campus, 14(3), 10–16.

15.

Murthy

, & Villasenor

M. R.

(2006, August). Exploring students’ attitudes towards interactive engagement courses: A comparison study. Paper presented at the national meeting of the American Association of Physics Teachers, Syracuse, NY. Retrieved from http://74.125.155.132/search?q=cache:qHQMdc-QwGgJ:paer.rutgers.edu/ScientificAbilities/DowTiloads/Papers/sahana_maria_post5.ppt+%22Exploring+students%E2%80%99+attitudes+towards+interactive+engagement+courses:+A+comparison+study%22…cd=1…hl=en…ct=clnk…gl=us…client=firefox-a]

16.

Pascarella

E. T.

, & Terenzini

P. T.

(1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.

17.

Uhalde

, Strohl

, & Simkins

(2006). America in the global economy: A background paper for the New Commission on the Skills of the American Workforce. National Center on Education and the Economy. Retrieved from http://www.ncee.org/index.jsp;jsessionid=aglg8nMlV5Ng?setProtocol=true

18.

Voluntary System of Accountability. (Voluntary System of Accountability Program). Retrieved October, 2008, from http://www.voluntarysystem.org/index.cfm

19.

Weller

(2008, June 16). SocialLearn: Bridging the gap between Web 2.0 and higher education. Retrieved March 1, 2009, from eLiterate Web site: http://mfeldstein.com/sociallearn-bridging-the-gap-between-web-20-and-higher-education/

20.

Wiggins

(1993). Assessing student performance: Exploring the purpose and limits of testing. San Francisco: Jossey-Bass.

21.

Wiggins

(2004). Assessment as feedback. Retrieved February 28, 2009, from New Horizons for Learning Web site: http://www.newhorizons.org/strategies/assess/wiggins.htm