Abstract
The value of experiential learning is widely acknowledged, especially for the development of communication skills, but students are not always aware of their own learning. While we can observe students practicing targeted skills during the experiential activity, the experience can also color their explicit understanding of those skills. Transfer of applied knowledge to managerial contexts requires an explicit grasp of the skills as appropriate solutions to the problems they encounter within the experiential team. This article reports the adaptation of assessment processes to encourage the reflection steps necessary for developing the desired managerial perspective on team communication.
Competent communication within group settings is essential in the contemporary workplace (Worley & Dyrud, 2001), widely recognized as essential to quality practices (M. W. Allen & Brady, 1997;Terziovski, 2002) and participative management more generally (Drehmer, Belohlav, & Coye, 2000). Although employees who have received training have been shown to exhibit better conflict resolution and team skills (Hartenian, 2003), team communication instruction remains one of the more challenging aspects of a university business curriculum. The communication used by effective teams is complex and sophisticated (Yeatts & Hyten, 1998), including knowledge, skills, and abilities that involve a sensitivity to a specific organizational environment (M. J. Stevens & Campion, 1994).
Previous investigations have demonstrated the utility of contextualized communication instruction (Hoover, Giambatista, Sorenson, & Bommer, 2010). Business courses can teach group process skills along with discipline knowledge (Young & Henquit, 2000), and MBA programs, especially, have long recognized the value of integrating conceptual knowledge and practical advice with experiential learning through workshops, case studies, or field experiences (Bogert & Butt, 1996;Navarro, 2008;Saunders, 1997), internships (Dillon, McCaskey, & Blazer, 2011), or external consulting projects (Glazer, 2010). Experiential learning has been shown to have positive effects on student self-awareness and collaboration skills (Brzovic & Matz, 2009), creativity and flexibility (Bush-Bacelis, 1998), commitment and motivation (Cyphert, 2006;Judge, 2006;Littlefield, 2006), and emotional intelligence (Sigmar, Hynes, & Hill, 2012). All of these effects demonstrably enhance students’ team communication expertise, but assessing the experience’s direct effect on learning goals is a more difficult proposition.
One recommended solution is to embed assessment within the experiential learning process (O’Toole, 2007), designing prompts for student reflection that guide learning toward desired outcomes even as they provide evidence that learning has occurred. Implementation within a complex, time-sensitive, and emotionally intense consulting project is not simple. Nevertheless, the experience of the University of Northern Iowa’s MBA program suggests that the integration of instruction, assessment, and practice within an experiential learning project can enhance student development in the elusive, but important domain of team communication. This article presents a model for integrating learning goals, learning assessments, and reflection prompts to enhance the development of a managerial perspective of team communication.
Managerial Communication Goals
The MBA program at the University of Northern Iowa aims to meet the needs of midlevel management or prospective management employees of small- to midsized companies in the U.S. upper Midwest. One learning objective involves “ . . . the ability to effectively participate as a team member, facilitate group processes, and manage team projects,” with a focus on managerial skill. As with any aspect of communication, managerial applications require an ability to encourage and develop others, along with strategic application of one’s own skills. To effectively assign, train, or coach teams within an organization, a manager must be cognizant of the impact of relationships, skills levels, and specific behaviors. It is not enough tobean adequate team player; a manager mustbe mindfulof the impact of various behaviors on team outcomes in order to effectively create and manage teams within a business environment and proactively develop effective team skills of her own employees. A qualitatively different realm of action is involved when the learning objective ismanagerialexpertise rather than simply team participation.
Peer assessments were collected within a learning assurance program that includes data from peer evaluations for each student, EBI Alumni Surveys, within-course evaluations of team performance, program evaluation surveys, and course evaluation surveys. Assessments collected annually over 5 years indicate generally proficient communication within self-managed teams (seeTable 1). On a scale of 0 (does not meet) to 7 (proficient), peer assessments cluster between 3.74 and 5.53, indicating that students exhibit personal team skills at an adequate level for each of the College’s team communication learning goals.
Peer Assessment Averages in Capstone Experience.
Note. Scale: 0-1 =does not meet, 2-3 =developing, 4-5 =adequate, 6-7 =proficient.
In any one year, teams might experience unusually difficult circumstances, and each year a few students fall below the norm (typically international students without work experience who face vocabulary or culture issues). Conversely, a few students exhibit proficiency in one or more areas because of career experience or specialized training. Overall, however, students in the program are demonstrably able to work within self-managed teams to plan and execute successful projects.
Graduating students and recent alumni of the MBA program affirmed that team experiences were a valuable part of the program, but questions remained about the nature of that value. Students’ ability to practice targeted team skills during the program do not predictably translate into a mindful managerial capacity to develop, supervise, or coach effective teams after graduation. Exit surveys suggested students were highly engaged and reported learning from the ambiguities, conflicts, and problems inherent in the experiential learning process but simultaneously failed to report (and seemingly even resisted any responsibility for) management steps designed to resolve those ambiguities, conflicts, and problems.
Instructional Resources
The general philosophy of the MBA program reflects a “learned-behavior approach” (Hunt & Sorenson, 2001), which emphasizes the development of various skills in an inductive learning process that leads to conceptual understanding. Because students are judged to be generally proficient in the target skills, there is no explicit attention to team communication instruction (i.e., demonstrations, role-play exercises, case discussions, etc.), and the curriculum instead prioritizes conceptual knowledge and mindful management of team processes. Opportunities for self- and peer assessment allow students to identify their own developmental needs, with the assumption that students in the MBA program “are capable of using printed material, demonstrations, and discussion to identify areas in which they need to improve” and are willing to take responsibility for enhancing their skills (Cyphert, 2002, p. 85).
Given the emphasis on management of team communication, the most important element of the instructional process is a culminating consulting project known as the Capstone Experience. This final application of the MBA curriculum content requires teams of three to five students to work with a regional business, government, or nonprofit organization, taking approximately 6 months to identify, analyze, and propose solutions for a business problem or opportunity. 1 Its educational intent is twofold:
Develop problem-solving and decision-making techniques in a dynamic business environment, giving students a real understanding of the complexities of applying theoretical knowledge in an environment characterized by incomplete information, unclear goals, competing stakeholders, and dynamic political and economic conditions.
Develop the managerial skills of MBA graduates in the areas of team dynamics, business writing, professional presentations, organizational dynamics, and cross-cultural communication.
The Capstone Experience features active coaching of team processes, intended as a remedy for the typical business school assignment, which asks for team products without providing training in effective team processes (Hansen, 2006;Vik, 2001). Instead, mentoring and coaching occur at appropriate stages within the Capstone Experience project. Following the model introduced byHackman and Wageman (2005), team formation guidance is provided as teams are first meeting, while process and strategy advice is delayed until status report discussions after the first half of the project. Teams disband at the end of the project, making moot the collective after-action review process recommended byHackman and Wageman (2005)as appropriate to workplace teams. Instead, individuals and teams are encouraged to meet with their assigned faculty mentors, the capstone coordinator, and the MBA program director to discuss team problems, options, and outcomes.
Within this mentoring and coaching structure, teams must nevertheless function autonomously as consultants to regional businesses and nonprofit organizations. The program’s core student, a midcareer business professional, might come into the program as a proficient team member, but within the context of a high-stakes consulting team, students must take a more proactively robust managerial role. Rather than simply giving students yet another team experience, the instructional goals of the MBA program require that the Capstone Experience allow students an opportunity to develop mindful team management skills.
In practical terms, instruction relies heavily on students’ self-directed learning. Because each team of students faces a different client situation, there is no way to anticipate exactly the instructional needs with respect to team facilitation or management skills over the course of a specific project. Furthermore, because students’ cultural backgrounds, communication competence, and work histories vary widely, there is no consistent baseline knowledge on which to build a meaningful curriculum. While the faculty can be sure that the Capstone Experience will teach students a great deal about team processes, there is no good way to predict exactly what the lessons should be for any one student.
Experiential Learning and Learning Outcomes
Research consistently shows experiential learning yields positive effects (Brzovic & Matz, 2009;Bush-Bacelis, 1998;Cyphert, 2006;Judge, 2006;Littlefield, 2006;Sigmar et al., 2012), but assessing a direct effect toward learning goals is not simple. The learning situations reflect variability in student background, preparation, and motivation, compounded in the experiential context by differences among the students’ projects, clients, locations, or team composition. There is no question that students learn though experiential projects, but there is little evidence that they are all learning the same thing. As a result, most experiential learning programs do not attempt to assess individual learning. Instead, program assessment is “the easiest and the most common type of assessment” to do (Qualters, 2010, p. 57); success is measured with quantitative data such as the number of participants, student characteristics, or clients served.
With more than a hundred business and nonprofit organizations served since 1999, Capstone Experience success measures include client testimonials and estimates of value of consulting for the regional economy. The faculty and alumni consider the consulting experience a “distinct advantage” of the program, providing students with an opportunity to integrate knowledge and skills under the pressure of working for a vested client (Wilson, 2011). Such measures can be gratifying, and the College regularly points to the impact of Capstone Experience teams within the regional business community. However, such accolades cannot indicate whether the Experience has enhanced managerial team communication skills.
A program exit survey frequently garnered comments about team processes, as did a final reflection paper required of all capstone participants. While these provide only self-reports, the students’ comments were reviewed for insights. Over 10 years, 118 students (72.4% of all graduates) completed exit surveys. The authors categorized comments related to “strengths” and “weaknesses” of teamwork in the MBA program.
General Strengths and Specific Weaknesses
The most obvious insight was that students both valued and complained about group work, or perhaps, some students appreciated it, while others did not. In either case, the nature of comments varied with respect to specificity. Students reported the value of the team experience with broad descriptors such as “project management skills” or “group work,” but never in terms of specific skills developed or used. Conversely, they described team-related weaknesses in terms of specific problems or events within the project. There were a few holistic complaints about “too much” group work, but most students named specific situations related to workload distribution, slackers, unequal skill levels, or communication with international students.
This result suggests that students did not recognize their development in terms of specific, strategically useful managerial skills, even though they might have developed more holistic skills such as self-confidence, empathy, or emotional intelligence. This is not to say they had not gained management skills, but until students have developed a strategic awareness of those skills, we could not claim to have met our managerial communication learning goals.
Recognizing Management Issues
A second observation was that the very things that students list as disadvantages of teamwork are the common problems that effective managers must solve:
appropriate size of teams
differences in resources allocated or work expectations across teams
interpersonal relationships within teams
cultural and language differences
within-group differences in work ethic, competitiveness, or performance standards
background differences in educational preparation or work experience
conflicting work schedules
organizational policies/work rules
For instance, students consistently listed unbalanced workloads as a “weakness” of teams, sometimes tied to incompatible or “challenging” work schedules and sometimes to variations in technical preparation. Yet there was never a suggestion that the program should offer better instruction for developing and managing collaborative work methods. Similarly, difficulties with a “lack of accountability” were noted with the implied—and in a couple of cases, explicit—understanding of performance management as the “firing” of nonperforming team members. The sentiment echoes many professionals’ secret wish to have management fire all their nonperforming colleagues so they do not cause any pain. However, managerial success requires taking steps to improve others’ performance, skills, or motivation. The comments suggested that students respond to immediate team challenges, perhaps even solving the problems in productive ways, but without developing an ability to objectively analyze and troubleshoot team issues from a managerial perspective. We again judged the goals not to have been met.
Team Performance as an Ethics Problem
Perhaps most telling were the responses to a question on the final capstone reflection paper about ethical issues in the consulting process. When asked to describe any ethical challenges experienced during the Capstone Experience, students reported, almost without exception, team management situations. The faculty considered such incidents to be tests of managerial skill, but students perceived them as ethics violations on the part of their classmates. Research with workplace teams supports the connection; shared perceptions of fairness are a predictor of team performance (Roberson & Williamson, 2012). Even so, while an employee’s good attitude, work ethic, or team orientation might be deemed desirable by coworkers, such traits as “initiative, trust, openness, helpfulness, flexibility, and supportiveness” are not generally amenable to managerial control (M. J. Stevens & Campion, 1994, p. 504).
The learning goal rubric for team communication had deliberately minimized attention to personality traits in favor of objective communication behaviors, but that instructional framework had obviously not eliminated students’ willingness to explain dysfunctional behavior in terms of ethical choices. We might dismiss this finding as inevitable: the result of a fundamental attribution error. That would be an error, however, that an effective manager should strive not to make. Still, in practical terms, a manager does need to foster teams’perceptionof a shared justice climate, which has been shown to depend on team homogeneity, interpersonal expressiveness, and interdependent work relationships (Roberson & Williamson, 2012).
Enhancing Managerial Development With Experiential Learning
The tenuous relationship between experiential learning and an assessment of students’ development as managers for a team environment did not deter the faculty from its commitment to the authentic consulting context, which offers a unique value for students. Classroom exercises and role-play might offer more opportunities for structured instruction and explicit conceptual understanding, but two key factors suggest the Capstone Experience remains the best option for developing team management skills: inherent emotional engagement with a project and the self-managed team environment.
The experiential learning model is successfulbecauseit triggers emotional cues to motivate and guide learning, especially implicit learning (Kolb, 1984), but this same feature can create stress, which makes explicit cognitive learning more difficult. Within the capstone project, we have seen this dynamic played out repeatedly. The factors that demand engagement are simultaneously sources of stress.
Every capstone team faces significant challenges, starting with the normal project issues of content and process. Furthermore, conceptual challenges are why MBA students are asked to apply their knowledge in the context of an existing business. Case studies present static, even sanitized, business questions, but practicing consultants must be prepared to face deceptive or imprudent clients, ambiguous instructions and requests, and situational changes.
Intellectual challenges might be the faculty priority, but the biggest challenge from the students’ perspective is the sheer stress of meeting the expectations of the Capstone Experience. The time and peer pressure inherent in a culminating project, the ambiguities of an authentic consulting project, and the complex sense making required in a diverse project context are usually unexpected and sometimes overwhelming. However, these stressors drive a level of student engagement impossible within the structured parameters of classroom instruction.
The special situation ofself-managed teams in the Capstone Experience similarly poses both unique challenges and unique learning opportunities. Team assignment depends primarily on students’ expressed interest in working with a particular client or industry, with some attention to previous professional experience and students’ requests to be placed on the same (or different) teams. No team leader is designated and, while each team is assigned a faculty mentor, the role is strictly advisory.
The absence of any supervisory alternative forces teams to perform managerial functions, but without an imposed supervisory structure, there is no explicit structure to guide students’ development in this area. In these realms, the inherent variability of experiential learning compromises the value of fixed instructional materials. Learning goals include task knowledge as well as conflict resolution and leadership skills, but without any specific application to a self-managed context. Once projects are underway, students will sometimes request managerial advice from their mentor, capstone coordinator, or other faculty members, but typically present themselves as facing “personal” problems with situations or individuals.
Mentors sometimes suggest solutions when teams deem themselves unable to resolve issues, but, even with coaching, students might not understand themselves as part of a team that is experiencing problem-solving, decision-making, or leadership issues. Their focus is on solving an immediate problem, rather than learning how to solve problems in general. The self-managed team environment does seem to call on managerial skills, but it offers little structure to guide students toward a strategic perspective.
With any project, resource, goal, and process, conflicts are sure to arise, and self-managed collaboration necessarily involves conflict resolution skills. The program goals name conflict resolution as a targeted skill set, but without any presumption of a specific method, perspective, or context. Once again, the self-managed environment invites the application of important skills, but it offers no systematic supervision, coaching, or instruction to encourage the development of strategic managerial expertise.
Reflection in Experiential Learning
The experiential learning environment offers both emotional and concrete incentives to develop skillful team behavior, but both the emotional engagement and autonomy can also preclude the development of explicit awareness of the managerial skills or appropriate strategies for applying them. The faculty determined that the experiential learning activity needed instructional enhancements in order to achieve its potential to develop critical insights surrounding team management skills. A revision of the assessment process to incorporate student reflection prompts seemed to offer promise.
Concrete experience is just one part of the experiential learning process, which necessarily includes reflection, theorizing, and subsequent application (Kolb, 1984). Reflection is often incorporated into experiential assignments as an important part of the learning process (Mahin & Kruggel, 2006), and self-assessment prompts can be used to guide students’ reflection toward the essential steps of metacognitive theorizing and setting application goals (O’Toole, 2007). Reflection is generically understood as a moment of detached, dispassionate self-observation—a quiet moment away from the emotional complications of concrete experience.
The experiential learning works because of the inherent tensions. Emotions must be “available for use” in order for the learner to formulate and achieve goals (Moon, 2004, p. 46). Emotions guide students’ attention toward those aspects of the experience that warrant learning. Structured reflection on the emotion-inducing elements of the capstone project might thus allow students to more accurately identify managerial situations and assess their own capacity for skillful response. Furthermore, the reflection process can direct the student’s attention toward strategic choices made—or not made—at the point of emotional engagement. As with team formation steps and managerial resources, even consulting contexts that do not consistently require tools or foster good choices during the project can nevertheless provide learning opportunities for the students involved.
As one element of the College’s ongoing learning assessment process, the students in the Capstone Experience had been required to produce a final reflection paper, originally designed to assess student writing and technical knowledge relevant to the project scope. In view of the need for better development of managerial team skills, we revised this assignment and added additional reflection prompts designed to better capitalize on the educational potential of the Capstone Experience. Furthermore, both coaching and oral prompts to reflect on the management situation had been administered on an ad hoc basis, so systematic prompts were developed to optimize their instructional effectiveness.
Hackman and Wageman (2005)have shown that asking students to reflect on behaviors that are appropriate to the stage of the team’s work is most effective. The assessment process was thus redesigned to prompt reflection at key points across the students’ Capstone Experience.
Initial Individual Reflections
During an early course in the MBA program, students had been oriented to the team communication learning goals and skill vocabulary, and they were prompted to reflect on their own team management skills with a self-assessment instrument (seeAppendix A). These documents have not been consistently collected, however, making an independent assessment of that step problematic. A second prompt was introduced at the first meeting of the Capstone Experience. The team skills rubric is collapsed into three categories of communication, facilitation, and group skills within which students are asked to self-assess their own skills as well as their team management and coaching expertise on a 1- to 5-point Likert-type scale. Three open-ended questions encourage higher level reflection in the key areas of problem-solving, decision-making, and collaborative work (seeAppendix B).
With just 1 year of results, we draw no conclusions other than to note that the results support our own observations of teams over the years (seeTable 2).
Initial Rubric Self-Assessments.
Note. ESL = English as a second language.
All students rated themselves lowest in the area of individual skills (i.e., conversational skills, technical task competence, engagement, etc.), with the average falling midway between “I can do these things myself” (3) and “I know the goal but not how to coach a team” (4). The difference was much greater for English as a second language students, many of whom are uncomfortable with the fast-paced and sometimes heated conversations that characterize U.S. work teams, but all students felt unqualified to coach or train their future reports in these areas. Most U.S. employers label individual communication competence as a baseline entry-level skill (National Association of Colleges and Employers, 2012), which employees ought to have mastered during their educational preparation, and remedial skills training is not typically considered a managerial function.
Students, particularly native speakers, reported that they understood the team facilitation and process goals, but even native speakers approached managerial confidence only in the area of team facilitation skills, skill sets that are often discussed in management textbooks and practitioner publications. All students were generally unsure about coaching toward effective collaborative work processes, and their open-ended reflections suggest the cause might be a lack of strategic mindfulness—the very outcome we aim to achieve with additional reflection on their experiential learning.
Asked to reflect on effective and ineffective problem-solving methods they had used in previous MBA teams, students universally discussed foundational communication skills, particularly face-to-face conversation. Not one student named a specific problem-solving method, even though the MBA curriculum is filled with such tools. Similarly, they named no specific decision-making methods. Students focused instead on the relative merits of consensus, consultative, voting, or command methods, although without using that vocabulary. Nearly all students described elements of collaboration as effective methods, but mentioned no generalized organizational methods to foster collaboration.
Initial Team Reflection
Shortly after Capstone Experience teams form, students are asked to discuss and report orally on their own task readiness and team processes. This activity is now given additional emphasis as an assessment instrument (seeAppendix C) that includes a reflection prompt asking the team to offer itself objective management advice for enhancing its own performance. With just one round of results, we claim no new insights on teams’ development with respect to managerial team skills. We anecdotally note that teams highly engaged in the in-class discussion were also on track with team tasks and rated themselves at what appeared to be accurate levels, although the request for managerial advice was met with either superficial advice (e.g., “Don’t be afraid of creative solutions”) or a status justification (e.g., “Scope still under construction”). A third team, which was dealing with a nonperforming member and clearly avoiding discussion, rated themselves higher than their actual output warranted and gave themselves no advice at all.
These behaviors might turn out to be typical over many teams, but even within the highly variable contexts of experiential learning, we find both the assessment results and reflection process to be useful. Rather than depend on summative numeric data, we find ourselves discussing the dynamics of real teams in real situations. The reflection exercise gives the faculty a chance to reflect as well. Given this relatively unusual situation of a team with significant dysfunction, the formative assessment process gives us a tool for initiating an intervention. We expected the first few years to yield too little information for any further analysis; but, we have found ourselves with extremely useful qualitative assessment data.
Midproject Reflection
We have added a formal midproject reflection on the team’s process and managerial strategies (seeAppendix D), shown byHackman and Wageman (2005)to be a point when teams intuitively reassess team processes and often make major changes in their internal processes. Students have often commented at this point on their implementation of team formation steps, client relationships, and problem-solving frustrations. The formal reflection prompt encourages them to think systematically about the managerial skills of leadership, decision-making, and conflict resolution and their systematic application.
We can never anticipate the specific issues that a team will face, but research with teams (Yeatts & Hyten, 1998) has identified critical functions: internal processes to reach decisions and solve problems; the logistics of team meetings, including scheduling, technical support, and developing a clear agenda; and good working relationships with external stakeholders. The midproject prompt again asks students to review their success with a managerial troubleshooting mind-set.
Postexperience Reflection
A final prompt appropriately occurs at the end of the project, after the anxieties are largely over and individuals are better able to reflect dispassionately about their experience. This activity remains in place with some revisions to elicit more insightful responses and thus more learning.
First, the current open-ended questions typically yield description or descriptive reflection,Hatton and Smith’s (1995)two lowest levels of reflection. The prompt now solicits reflection about diverse team perspectives on the team’s processes and interaction, potentially encouraging the higher levels of dialogic and critical reflection. Secondly, a 360° peer review/self-evaluation tool is used to foster more accurate self-assessments of leadership capability (Mayo, Kakarika, Pastor, & Brutus, 2012). Those data are incorporated with qualitative reflections to assess context-specific influences on managerial lapses, such as attribution error, which is potentially causing students to confuse ethical and managerial issues, or cross-cultural challenges, which might require specialized management tools.
Finally, the artifact provides individual data on writing and content knowledge skills, providing a mechanism to identify underperforming students within the team environment. Students also identify gaps between project requirements and curriculum coverage, allowing faculty to assess the degree to which the MBA program prepares students for typical problems found among regional businesses. These subjective, open-ended responses have not been included in the College’s assessment of team skill outcomes, but a qualitative analysis of their content suggests these reflections offer powerful insights into the effectiveness of experiential learning.
Nine papers from members of three client teams written in 2006, the earliest available set of reflections, were compared with 18 papers from five teams in the most recent cohort available, 2015. All comments relevant to team dynamics and project planning were highlighted, generating 184 items (an average of 20.4 per student) in 2006 and 227 (an average of 12.6 per student) in 2015. Two of the authors had worked closely with teams in both years, while the third had participated in a capstone project as an MBA student in the program. All comments could thus be interpreted in the context of a specific known client and project.
As noted earlier, the extreme variability across experiential projects and contexts makes generalizations of any kind difficult, but a comparative analysis of their content yields several insights. Comments were first matched for causal or explanatory connections. When a description of events or behavior appeared within a sentence or two of the student’s assessment of how or why it occurred, the two comments were coded as a single reflection. Multiple explanatory factors were treated as a single reflective comment. The results of this first sort (seeTable 3) suggest that overall mindfulness has increased over the years, with most students now reporting on team processes with attention to causal factors. This does not, in and of itself, indicate accuracy or appropriateness of those factors; items were paired regardless of the causal validity or managerial perspective invoked. However, it would appear that students have become more aware that team-related situations, events, and behaviors can be traced to explanatory factors of some kind.
Percentage of Observations With Explanatory Reflection.
Comments were then categorized in terms of learning goals. Our primary question was whether students observed team-related issues that were not captured within our assessment rubric. Although the students’ vocabulary differed at some points, we found a straightforward correspondence with the top-level learning goals (seeTable 4).
Reflections on Team Management Skills.
The Capstone Experience focuses on the management of team-level functions, but comments were often made about team members’ foundational communication skills or group facilitation skills. These were included as aggregate totals, referring almost exclusively to individual or collective task competence (90% of total across both years) or external communication (77% of total across both years), generally involving the client.
We observe variation across skill areas and across years but attribute these to the normal variations across experiential learning contexts. For example, two teams in 2015 found themselves dealing with a member who lacked relevant technical skills, driving a high percentage of comments in that category. Further analysis of the comments will allow faculty to assess instruction and coaching methods relevant to specific rubric elements.
Conclusions and Implications for Pedagogy
Assessment of communication skills in any context is challenging, but the complexity and variability of an experiential learning context adds complications. The systematic use of the learning goals rubric as a guide for reflection should make experiential learning more effective, but objective measures of effectiveness remain elusive. In practice, much reflection occurs during mentoring or coaching conversations, with only a small proportion expressed in written documents that are available for later analysis. Furthermore, learning continues to occur after graduation—sometimes long after graduation—and we expect some lag time between program changes and evidence from alumni or employer reports.
Still, we are confident that systematically incorporating reflection prompts into the assessment process yields both better assessment data and better student learning. We also recognize points where further improvements can be made in both instruction and assessment of our students’ team management skills. In fact, the degree to which we recognize the need for a continuous improvement mind-set draws from our first conclusion.
Conclusions
Closing the loop is an iterative process
On paper, an assessment program typically appears as a single feedback loop. Students score poorly on a goal, and changes are made to instruction, leading to some measurable change. In reality, even the simplest learning scenario involves double-loop learning (Argyris & Schön, 1996), and within the complex contexts of experiential learning, we can expect to find emergent knowledge structures. Self-referential, recursive feedback loops (Hofstadter, 1979,2007) create small but meaningful cognitive shifts in understanding with each task or assignment repetition.
In practical terms, this means new faculty insights also emerge with each round of assessment, and we must be patient with our search for a seemingly illusory end of the rainbow. In our case, for instance, after a first round of assessment several years ago, faculty focused on skills associated with self-managed teams and sought to align goals and instruction more closely to those specific skills. With the next iteration, students reflected on self-management issues in terms of ethics, which we first assumed to stem from fundamental attribution error. As these topics were more closely parsed during subsequent course instruction and team coaching, however, we suspected individual variations in task competence as the root cause of most team management issues. Principles of internal team dynamics or conflict resolution skills tend to assume a level playing field among equally competent peers. These are not merely self-managed teams; they are also student teams. In a workplace, weak team members might be reassigned or terminated; academic institutions issue lower grades. On our fifth iteration since the review of the team skills learning goal, we find ourselves returning to instructional resources—not for self-managed team techniques, however, but for ways to scaffold academically weak students through the experiential project.
Student reflections offer important insights for the definition and assessment of learning goals
Those who argue for the instructional value of assessment or scoring rubrics (D. Allen & Tanner, 2006;Panandero & Jonsson, 2013;Reddy & Andrade, 2010;D. D. Stevens & Levi, 2013) suggest designers utilize vocabulary and concepts that reflect student usage and understanding. Rather obviously, student reflections can offer insights into their use of terminology and concepts, but we found them equally useful for understanding student priorities, concerns, and challenges during the experiential learning process. We were correct in our initial assumption that students would each learn something different from the experiential project, but we had not realized there would also be similarity across the diverse contexts.
For us, the key step was mapping student descriptions of their team processes back to the assessment rubric. Outcome assessment had previously been done by reading the final reflection paper as a writing artifact, noting only whether team processes were understood and vocabulary was used appropriately. With a focus on the reflection itself as the assessment artifact, we found ourselves categorizingallstudent ideas. We were gratified to find they did fit within the goals rubric, but we were also able to discern patterns. For instance, all student comments regarding collaborative work processes dealt with just two of the four subgoals,initial planninganddecision-making processes, and we will now parse these topics more fully during instruction and coaching. We expect additional insights to emerge, causing yet another zig in our path toward that pot o’ gold, but we look forward to making thoughtful and important adjustments to our team communication learning goal at the next 5-year review.
Reflection prompts offer a tool for the integration of outcomes assessment and student learning
The value of assessment rubrics as instructional tools has been previously noted (Panandero & Jonsson, 2013;Reddy & Andrade, 2010), as has the instructional value of well-timed reflection prompts within experiential learning contexts (Brockbank & McGill, 2007;Kolb, 1984;Mahin & Kruggel, 2006;O’Toole, 2007). Our faith in the value of coordinating the assessment rubric with a reflection prompt appears to have been rewarded, and we remain confident that student learning has been enhanced with the addition of rubric-based reflection prompts.
At a more subtle level, we have also found faculty conversation around both learning goals and student performance has grown richer and more useful. As we interpret student reflections, we find ourselves doing so in terms of specific team members, client situations, and business contexts. Instead of looking at numbers that tell us the proportion of students who seem to understand team management skills, we can see the range of options a specific team had as it dealt with a client’s internal politics or a team member’s lack of technical skill. Systematic interim reflections have moved the faculty beyond anecdotal conversations about current projects toward our own reflective assessment of the timing and content of our coaching and instruction. Perhaps most intriguingly, assessment that occurs within the single capstone course has become interesting enough to spark conversation among all faculty in the program, leading to both better coaching within that culminating experience and better instructional preparation in the prerequisite courses.
Qualitative, formative data offer rich insights
Like many faculty, we have favored quantitative data in our assessment methods, both for its convenience in data collection and reporting, and for its promise of generalizable results. We began, as this article does, with the recognition that variable contexts make outcomes assessment difficult within experiential learning events, but we understood summative, quantitative assessment data as an unmet ideal. We were willing to assume that learning was taking place, even though we could not hope to measure it directly.
Implications for Pedagogy
As so many qualitative scholars will attest, convenience and generalizability offer no inherent advantage over systematic immersion in the data and richly detailed understanding of the idiosyncratic workings of specific projects and project teams. The key seems to lie in the quality of the underlying data. Having moved from ad hoc, oral, anecdotal conversation to systematic, carefully timed, and thoughtfully aimed reflection prompts, we are building a grounded theory of our own instructional processes and student outcomes.
The actual steps of integrating reflections with assessment are deceptively simple: create simple prompts that utilize learning outcomes goals and vocabulary, administer them at points in the experiential learning process in which the use of the targeted skills might be expected, and systematically analyze the results in terms of the assessment rubric. The results have been powerful, leading to instructional changes that give more than lip service about closing the loop.
Furthermore, the elements involved in assessing the outcomes within an experiential learning context seem to offer promise for other applications. Service learning, increasingly popular as a mechanism for engaging students in their educational process, is similarly context specific and often experiential in design. Learning goals that include subjective and idiosyncratic applications, such as critical thinking, creativity, and interpersonal communication, are similarly difficult to standardize for quantitative assessment. Learning in general, when understood as a cycle of activity, reflection, integration, and application, might be usefully assessed with attention to students’ subjective reflections on their own process.
Finally, as management faculty, we have recognized some of our own assessment biases as a broader bias toward objectivist managerial practice. Our learning goals assume the successful graduate of an MBA program will possess the capacity to diagnose and fix problems faced by his or her team-based reports. Drawing from a tool kit of communication principles, team practices, and established intervention points, the skilled manager would assess the situation and administer successful coaching and instruction. We now approach our own interventions as formative assessment, recognizing the team’s reflection process as an essential element of its own identity as a team; stepping in to fix a team destroys the team. Objectivist management traditions face the same challenge within any complex learning organization (Clippinger, 1999;Gharajedaghi, 1999;Stacey, Griffin, & Shaw, 2000). An iterated, multiple-loop assessment process never closes; the reflection raises new questions and leads to new knowledge with which to revise the learning goals themselves.
Footnotes
Appendix A
Appendix B
Appendix C
If you were your team’s manager . . .
Having worked together for more than two months, you should “take stock” of your own performance as a team. This is the time to make any needed changes in your internal processes and procedures to ensure timely completion of your Capstone project.
Rate your own team’s effectiveness in five categories of project management.
Appendix D
Authors’ Note
Preliminary results of the research reported here were presented at the annual convention of the National Communication Association, Assessment Division, Chicago, IL, USA, 2014.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
Author Biographies