Abstract
The increasingly common practice of engaging consulting firms to assist states with educational policy agendas requires an analysis of the role these consultants play in what is positioned as a democratic decision-making process. In this study, we examine the discourse of a state-level advisory committee formed to develop a new teacher evaluation policy under Race to the Top. We used discourse analysis methods to analyze audio recordings of 11 meetings of this committee. We identified two patterns of consultant talk as it related to committee decision making: making decisions through validationand deferring and redirecting decisions, and we describe their implications.
Keywords
Introduction
With 42 pages and a catchy title, a report called The Widget Effect (Weisberg, Sexton, Mulhern, & Keeling, 2009) refocused conversations about teacher quality and student achievement on evaluation policy reform. The report pointed out that across four states, 99% of teachers were rated “satisfactory” even though student achievement in those states demonstrated persistent low performance for significant numbers of students, especially high-risk sub-groups of students. Teachers, like widgets, were completely undifferentiated by evaluations, though educational outcomes varied widely. Although it is easy to agree that teacher evaluations should differentiate effective from ineffective teachers, it is difficult to agree on criteria with which to do so.
The argument put forth in The Widget Effect can also be found in the outline of the Great Teachers and Leaders portion of the Race to the Top (R2T) grant competition’s criteria (U.S. Department of Education, 2009). Up to 58 points (28% of available points) could be earned by states who promised to improve teacher and principal effectiveness by measuring and addressing performance. Twenty-eight of these points hinged on a state’s promise to use performance evaluations in decisions about hiring, firing, promotion, and tenure. States that sought the available infusion of funding through R2T not only had to agree to reform their teacher evaluation systems, but they also had to take on the difficult work of developing an evaluation system that would provide this differentiation and link evaluation ratings to human capital decisions.
In this article, we describe findings from a study of the process of developing a reformed teacher evaluation policy in Tennessee, one of the first-round R2T grant program-winning states. Because teacher evaluation is deeply connected to a range of controversial issues in education reform (uses of standardized testing; teacher effects on student achievement; teacher preparation, retention, and tenure; definitions of teacher quality; the role of teacher unions; etc.), it is often addressed by committees that represent various stakeholders. In Tennessee, the governor and the speaker of the state House of Representatives appointed a 15-member committee (the Tennessee Teacher Evaluation Advisory Committee or “TEAC”) to represent the views of teachers, principals, and educational administrators as well as businessmen, community leaders, and legislators. A press release explained that committee members were selected to represent the interests of varied stakeholders and represent the state’s geographical demographics (Tennessee Government, 2010). This distribution implied an attempt to represent varied interests and perspectives in the consensus of the committee.
After an initial meeting, published minutes from April 1 reflected that the TEAC members requested the services of an educational consulting firm to help facilitate discussions and present research, options and projections from which to develop their policy proposals. What happened next could be viewed as an example of what Koppich and Esch (2012) have referred to as a shift in the locus of teacher policy decision making. They argued that policy efforts to improve teaching effectiveness moved from local governments to state- and federal-level governing bodies in the 1980s and 1990s, with an increasing influence of philanthropic and other non-profit foundations in the last decade. In this case, a committee that was formed at the state level went through a bidding process to contract with a national for-profit consulting firm, Education First. Two consultants from Education First then led the TEAC through the policy-making process.
As Ravitch (2010), Hess (2011), as quoted by Brownstein (2011), and others (e.g., Dillon, 2010) have noted, education reform efforts in the United States are increasingly populated by education consulting firms and for-profit interests. Indeed, the scale of reforms encouraged by R2T were often beyond the capacity of existing state departments of education and resulted in tens of millions of dollars in sub-contracts for outside consultants. Thus, as we investigated the work of the TEAC, we were interested in the impact of consultants on its decision making. The purpose of this study is to examine the function of the educational consultant talk in constructing a policy in the context of these meetings.
When outside consultants are hired to facilitate meetings, they are often brought in under the premise that their presence and leadership will increase the efficiency and productivity of a committee (Schwartz & Davidson, 2005). In this article, we argue that consultant talk can also function to manage controversy, in part by determining what counts as a decision, and also by setting priorities through managing the focus of agendas and conversations. This may represent a great deal more power and influence over educational policy making than is generally assumed to be held by a non-voting, external consultant to an advisory board where matters are “decided by committee.” Building upon Boden’s (1994) assertion that “organizational decisions are talked into being in fine yet layered strips of interaction” (p. 91), and that verbal interactions are the site of action in group decision making (Susskind, 2010), we describe how group decisions around educational policy are constructed through talk. The research question that guided our analysis of the data was as follows:
Review of the Literature
Within this section, we briefly summarize research in two overlapping areas of inquiry: (a) studies of decision-making talk and (b) studies of democratic decision making. One of the themes across research on decision-making talk is the elusive nature of decision-making moments. Even in settings where an established protocol like Robert’s Rules of Order is used to formalize a decision, the nomination of ideas and construction of consensus that occur before a vote is taken, and the smaller decisions related to use of committee time and resources, are often difficult to identify (Boden, 1994). Instead, what counts as a decision is constructed through talk in patterned ways. That is, decisions are named or identified, in part, by the ways in which they are taken up and restated later in conversations.
Contrary to the popular notion that decision-making moments are easily identifiable, especially in retrospect, Huisman (2001) argued, “Many decisions that are made in meetings, or in verbal interaction more generally, cannot be straightforwardly identified in the talk. They are largely invisible: they cannot be spotted as concrete utterances in an interaction, and have an ephemeral quality” (p. 70). He added, “It is not instantly obvious whether in fact a decision has been made . . . decision-making is an incremental activity in which members of an organization move their agendas forward, step by step” (p. 70). Having investigated the linguistic features of interaction that were characteristic of decision making across four organizations, Huisman argued that what counted as a decision “depends on the communicative norms of the group that is talking” (p. 69). That is, conversational norms dictate how a decision is constructed, identified as such, and taken up in further conversations as a commitment to future action. The study of group decision making is not a matter of simply finding the moment a decision is made, but identifying the interactional resources and linguistic features groups use to construct decisions about controversial issues (Boden, 1995).
It is difficult to trace the evolution or birth of a decision and why one, and not another course of action, is chosen. This is in part because the process of group decision making cannot be presumed to be singular, linear, or particularly logical in any organized sense of the word. Indeed, as Hernstein (1990) has argued, theories of decision making, specifically rationale choice theory, are better at describing how humans and animals should behave in decision-making settings, rather than how they do behave. In real time, we find that decisions and their reasons are varied and fluid. Boden (1994) claimed, “As realtime phenomena, decisions are, in fact, largely invisible and thus empirically unavailable” (p. 22), whereas decision making can be located in interactions between individuals in a group whose goal is to make a decision. Drawing upon discourse analysis, we argue that phenomena, such as decisions, are made visible through talk. Thus, in this study, we specifically drew upon discursive psychology, a form of discourse analysis with roots in conversation analysis and ethnomethodology, to examine such talk-in-interaction. Rather than analyzing talk to extrapolate some hidden meaning or imagine the speaker’s inner thinking, we orient to language as a medium for social action, assuming that language is always doing something (such as constructing and/or resisting decisions) within an interaction (Potter & Wetherell, 1987).
Although it is difficult to locate theorized connections between the process and product of decision-making ventures, discourse analysts have noted patterns in aspects of group interactions such as the facilitation of common agreement (Barnes, 2007), the negotiation of agendas (Boden, 1995), and the construction of mission statements (Connell & Galasinski, 1998) as well as the ways in which participants use language to formalize ideas (Iedema, 1999), open and close meetings (Heritage, 1997), and transition between topics (Barnes, 2007). Throughout this article, we argue that attention to language use is important for those who study and engage in committee work, especially where policies are situated within and across controversial issues such as teacher evaluation.
We also drew upon studies of democratic decision making and decision making by committee. Unfulfilled expectations for democratic decision-making processes was a theme across these studies. Either decisions were evaded and made impossible because of logistics, lack of information, and misuses of time (Wood, 1989); or decisions were made, but without the democratic ideals of equal participation (Townsend, 2009). For example, Marsh (2012) examined the “micropolitics” of teacher compensation committees under the New York City Schoolwide Performance Bonus program. She found that even when most people described the process of creating a compensation system as democratic, principals in particular tended to exercise their power over others to shape decisions. Marsh found that principals used their “formal authority” (p. 172) to influence decisions of the committees in both overt and covert ways. For example, principals could influence decisions by placing greater or fewer union representatives on the committees, or by replacing members with others who would not disagree.
An example of a more covert exercise of power was the strategic use of data to make a point or lead the committee to believe a certain decision was warranted. Although other committee members also referenced data, a principal’s access to and experience with the authoritative presentation of data contributes to an imbalance such that “the subtle influence of the principal and his strategic use of data to advance his interests calls into question the democratic and consensual nature of the final decision” (Marsh, 2012, p. 175). Building upon Marsh’s investigation of the micropolitics of committee decision making in school settings, we are interested in investigating the impact of the consultant’s formal authority over committee decisions that are controversial.
Using a micro-analysis of talk in a range of settings, Huisman (2001) concluded that decision-making moments are elusive and not bound by formal protocols, but rather are visible in assessments and formulations that are both patterned and situationally bound. Similarly, Weaver and Geske (1997) argued that educational policy making at the state level happens in a variety of both overt and covert conversations. They interviewed state-level policy makers about the sources of information they relied on for their voting decisions. They found that certain policy issues were more likely to be handled by elites in the legislature and informed by a small number of specific, close sources. Other policy issues were likely to be decided based on balanced information from a broad range of sources. Taken together, these studies suggest that decisions made by committee or within representative bodies of government cannot be assumed to be democratic in nature—rather, elites or people in positions of power may interrupt the potential for democratic decision making.
R2T and Investing in Innovation Fund grants have recently stimulated the demand and supply of education services, including consulting services, as state departments of education have proposed reforms beyond the scope of their internal capacity. Bulkley and Burch (2011) have noted that those who provide services to districts have begun to engage in the political process, writing that “it is critical that researchers and policymakers pay greater attention to the practices and impact of private actors in public education” (p. 248). They argue,
As we gain greater clarity about the direction in which private engagement is moving (i.e. what is changing, what is not, where there is greater equity and where we see abuses of power), we can use this information to inform policy that protects democratic values, such as voice and equality, and rewards innovation in line with these ends. (p. 248)
Most research on the role of private contractors and outside consultants serving as intermediaries examines processes of policy implementation, rather than policy making (e.g., Honig, 2004). Yet, some findings from these studies may be relevant. For example, Honig (2004) noted that intermediary organizations can increase resources and capacity for partner districts, but are sensitive to shifts in budgets and funding timelines. Cucchiara & Gold (2011) found in their study of corporate influences on the Philadelphia School District, that “contracting [with outside organizations] sets up a client-contractor relationship that can preclude other types of interactions, particularly criticism of the district” (p. 2490). We agree that “it is likely that there will always be a tension between managerial and democratic purposes for education—between the goal of efficiency and that of involving citizens in the messy work of setting and balancing priorities” (Cucchiara & Gold, 2011, p. 2491).
One role that these outside consultants and contractors may play is that of facilitator during policy-making meetings. One study in particular examined the role of a facilitator within a larger analysis of the discourse of decision making. Barnes’ (2007) examination of medical school meetings focused on how committee chairmen used particular formulations to establish agreement within the committee. Barnes argued that repeating utterances that gloss the preceding talk worked to close the business at hand, signal a transition to a new topic, and establish something for the record so that shared understandings could be preserved for later meetings. Barnes’ study focused on a facilitator that was a voting member (committee chairman) and who was thus expected to have an explicit contribution to the construction and nomination of decisions. Our study extends Barnes’ investigation by examining the role of non-voting external consultants on advisory committee decision making, particularly around controversial topics.
Study Context
The Tennessee TEAC was appointed in January 2010 as part of the state’s First to the Top Act of 2010. The First to the Top Act was passed to make Tennessee more competitive in the federal R2T initiative, a federal grant competition for states proposing education reforms. R2T required that the state adopt a comprehensive framework for identifying and evaluating teacher quality. As McGuinn (2011) noted, R2T has created the political cover for states to initiate substantial changes under the banner of education reform, while also providing capacity-building resources and infrastructure. Thus, policies developed for R2T initiatives often represent significant departures from past policies and practices.
Under Tennessee’s First to the Top Act, the TEAC was charged with proposing a set of guidelines and criteria for the evaluation of teachers and principals in all pre-K to 12th-grade subjects. Their proposal was presented to the State Board of Education, a nine-member body of appointed representatives of the congressional districts. After field testing the instrument in the spring of 2011, the recommended teacher evaluation policy was adopted and implemented across the state for the 2011-2012 school year.
The TEAC had 15 members, including representatives for public school district administrators, principals, and teachers; the state representative from the House Education Committee; the chairman of the Senate Education Committee; the Commissioner of Education; the director of the State Board of Education; and representatives from stakeholder groups such as Zycron, Inc. (an information technology company), the Memphis Urban League, and the Knoxville Chamber of Commerce. They began with monthly meetings and assorted sub-committees in March of 2010, led by the state superintendent of schools as committee chairperson. Four sub-committees, formed on April 1, 2010, each focused on a different requirement of the evaluation policy: the 50% of the rating that would be comprised of “non-data,” the “other 15%” achievement data, pre-K-2, non-assessed grades, and subjects, and principal evaluation. After finding it difficult to gather information and make informed decisions individually, the members discussed the possibility of hiring an outside facilitator. Published minutes from the first meeting indicate that “use of an independent facilitator to provide oversight was discussed but no decision was made” (March 18, 2010). At the next meeting (April 1, 2010), the governor’s office liaison said that a facilitator would be selected for the upcoming (third) meeting in May.
A bidding process overseen by the governor’s office reviewed applications over the next month and contracted with Education First, a consulting firm, to facilitate the meetings of the TEAC until its work was complete. Education First had also recently been engaged in a consulting contract to prepare the state’s R2T proposal. According to a state department official speaking in the May 10, 2010 meeting, the successful outcome of this initial consulting experience was integral to the selection of Education First to work with the TEAC, as was experience consulting on teacher evaluation development in other states or cities.
By May 2010, the sub-committees had dissolved and the full committee began meeting on a bi-weekly schedule, with every other meeting held as a conference call facilitated by one or two Education First consultants, both female, and referred to in this study as Consultant 1 and Consultant 2. Committee meetings were announced in advance and open to the public, though visitors rarely contributed to committee conversations (see Table 1).
Timeline of TEAC Meetings.
Note. TEAC = Teacher Evaluation Advisory Committee; SCORE = State Collaborative on Reforming Education; TAP = Teacher Advancement Program.
Method
This study was part of a larger discourse analysis study situated within a discursive psychology framework. This theoretical and methodological stance draws upon ethnomethodology (Garfinkel, 1967), rhetorical psychology (Billig, 1991), and conversation analysis (Sacks, 1992). Discursive psychology assumes that talk is action and investigates how language is used to accomplish particular tasks. The larger study investigated the construction of teacher effectiveness within the talk of the TEAC. This article reports on one aspect of the process by which the committee worked to develop the teacher evaluation policy: the role of the consultant talk in group decision making around controversial topics.
Data Collection
Under the Tennessee Open Meetings Act of 1999, all public meetings can be attended and recorded by any citizen. From May 13th through October 28th, 2010, the first author attended all in-person meetings of the TEAC and used two digital audio recorders to record the meetings. One recorder was kept next to the researcher and another placed on the opposite side of the room to better capture the voices of committee members around the room. In addition to recording, the researcher took field notes to aid in identifying voices. There were no fewer than 15, and sometimes up to 40, visitors at each meeting. The committee held six optional, publically announced conference calls that functioned as work sessions between in-person meetings. Skype™ call recorder was used to record the only conference call during which a roll call vote on major decisions was scheduled (September 2, 2010). In other cases, summarized minutes from conference calls were downloaded from the First to the Top website for review.
Data Analysis
Recordings from each of the 10 meetings and one conference call were uploaded into the Transana™ software program for transcription. The program enables the transcripts to be linked with the audio recording, synchronizing the files so that the data can be listened to and read simultaneously. A modified version of Jeffersonian transcription (Jefferson, 2004) was used (see the appendix) and is included in the extracts used to illustrate the findings. Transcripts were then uploaded into the qualitative data analysis software ATLAS.ti™ for further analysis.
Our analysis focused on how language was used to manage, facilitate, and negotiate the decisions of the committee, particularly around controversial topics. We noted, for example, ways in which the consultants framed the work of the committee, managed time, and set agendas. Our analytical approach was as follows: (a) repeated readings of the transcripts in conjunction with listening to the audio recordings; (b) identification, selection, and organization of patterns; (c) generation of explanations; and (d) transparent and reflexive documentation of claims. In the initial readings and listenings, we became familiarized with the meeting talk. This familiarity allowed us to identify patterns and variation across the data, as we attended to those sections that we found striking and surprising. During this first step of analysis, we met regularly; sharing our initial individual memos with each other and discussing our joint reflections about sections we initially found most intriguing (Potter & Wetherell, 1987).
We next re-read each transcript, in conjunction with listening to its recording, several times, noting patterns and conversational features. As we re-read and re-listened, the following broad, discourse analytic questions sensitized our analytic process (Potter, 2004):
What are the participants doing/accomplishing with their talk related to making decisions?
How are they constructing their talk to achieve this?
What discursive resources are being used to achieve these actions?
We used ATLAS.ti™ to organize the data and systematize the analysis process, annotating the data using the memo, comment, and coding features. Throughout the analysis process, we kept both analytic and theoretical memos. Meeting regularly, we discussed and began to identify what we came to view as the regularity and variability across the data set. After several cycles of analysis, we formed tentative explanations as to what the various discursive features were doing, moving back to the data to ground our claims. As we did so, we noted where and how certain language features were made relevant by the participants.
Findings
In the following section, we describe two main patterns that describe the function of consultant talk. The first, making decisions through validation, is a pattern that functioned to identify or name a decision as well as to limit discussion of alternatives and to signal a transition. The second, deferring and redirecting decisions, demonstrates the ways in which the consultants managed potentially controversial topics in the midst of the decision-making process. As we present the analysis of these patterns, we do not critique the consultant’s contribution or assume that their talk always represents unwelcome interference. Rather, we argue that their roles as facilitators of discussion are necessarily constitutive of the substance of decisions and should be understood as such. In the following section, we illustrate each pattern with extracts from the data. The extracts below are some among many examples of each pattern, but are chosen to highlight the decisions about some of the traditionally controversial aspects of teacher evaluation policies, including measures of student achievement, protocols for teacher observation, and the use of value-added measures.
Pattern 1: Making Decisions Through Validation
Although Roberts Rules of Order were employed by the committee to formally ratify all decisions, we found that the consultants made many informal decisions, such as determining which of the committee members’ statements should be treated as decisions, considered with more discussion, or deferred until later. In this section, we explore how committee member statements were validated by the consultants and as such taken up as informal decisions.
Extract 1 from the fifth meeting began with a minor decision to add two options to the list of acceptable measures of student achievement that would be included in calculations of teacher’s overall rating. This was necessary to be inclusive of students’ decisions to enter the workforce or certification programs in lieu of attending 4-year colleges. One of the sources of controversy for teacher evaluation policy makers under R2T is that states had to agree that student achievement measures would be a large percentage of all annual evaluations. Although few would argue with the idea of linking evaluations to “student outcomes,” the details of such policies are often controversial because of the nature of standardized testing and the lack of available, comparable options for teachers across subjects and grades. Put simply, evaluating teachers based on student achievement rests on the assumption that these achievements can be adequately and equally measured by available assessments.
This assumption is difficult to defend given the range of critiques associated with standardized and high-stakes testing, and given the range of subjects and grades for which no standardized test currently exists. Although the United States has spent the last decade developing high-stakes state tests to measure adequate yearly progress under the No Child Left Behind Act, the TEAC committee estimated that more than 300 grades and/or subject classes (e.g., music, journalism, and kindergarten classes) are taught in Tennessee each year without an accompanying standardized test. As the state’s standardized testing begins in third grade and focuses almost exclusively on reading and math, thousands of teachers fall into the “teachers of untested subject” category. Thus, discussions of how these teachers’ work should be evaluated have no easy answers. Extract 1, from the fifth meeting, illustrates how the decision to include certain trade-specific exams as an acceptable measure of student performance was made during a committee meeting.
Extract 1. 1
1 State Representative
2
: I would request that you consider a couple of ideas 2 uh like the end of course situation that could very well be a a what’s 3 the the there’s gonna be an auto mechanic ASE
3
you know the- 4 Consultant 1: -the certification [tests 5 State Representative: the certification]. Career tech certification that’s 6 used. And also ought to consider work keys 7 Consultant 1: Work keys. Ok. Ok. 8 Entrepreneur: Work what? 9 State Representative: Work keys it’s a it’s a basic- 10 Chamber of Commerce President: -it’s a measure of skills 11 State Representative: Work skills 12 Consultant 1: Career ready skills 13 Consultant 2: Good so we’ll add both of those um career certifications 14 and work keys . . .
In Extract 1, the state representative made a request to add certification exams to the list of acceptable measures (Lines 1-3). The consultant then clarified this request in overlapping speech with the representative in Lines 4 to 5 (“the certification”) and again in Lines 6 to 7 (“work keys”). Another member of the committee clarified that “work keys” referred to a measure of work and career-ready skills (Line 10), and in Line 13, the consultant accepted the representative’s suggestion and treated it as a decision (“good so we’ll add both of those . . .”).
Once the committee members clarified what they meant by “work keys” (Lines 1-12), there could have been discussion or a vote by the committee, but instead, the consultant re-voiced the suggestion and treated it as a decision (Line 13). This statement of future action (“we’ll add both of those”) worked to signal a decision had been made. In this way, the consultant’s response to the representative’s request was to make a decision on behalf of the group through validating an initial request.
Although this request was quickly granted, its substance is not neutral or inconsequential. The decision to include work keys as a measure of student achievement rests on the assumption that it is fair to evaluate some teachers based on state standardized tests, others on ACT/SAT scores, and still others on work keys, each of which are designed to assess different kinds of knowledge for different purposes. This decision stood in the final policy, but had it been put to the committee for discussion, a different or additional decision may have resulted. Instead, this decision-making moment was managed by the consultant and presented as positive (“good”) and logical. By analyzing the role of the consultant in taking up this suggestion as a decision, we do not critique the decision itself. Instead, we note the constitutive nature of the consultant’s contribution to the conversation. More than presenting evidence and organizing time, her interactions with committee members constitute, and at times resist, decisions.
In Extract 2, from the fifth committee meeting, we see a second example of the consultant taking up a committee member request in a way that marked a decision had been made rather than calling for further discussion. This extract began with a committee member requesting that the policy language include the word “post-secondary” rather than “college.” In the same context as Extract 1, this request would include students who choose alternate paths after high school.
Extract 2
1 Representative: Could we use the term post-secondary? 2 Teacher: Yes please yes 3 Consultant 1: [post-secondary 4 Representative: instead of college] 5 Consultant 1: Absolutely. Um and we do need it to be inc- inclusive of 6 rigorous apprenticeships as well as 2-year insti- 7 tutions or 4-year institutions. (.) Um even uh certificate programs in fields that lead to good 8 paying careers that you can support a family on.
This extract stood out to us because it is a clear example of the committee members positioning the consultant as the arbiter, one who has the final say as the leader of the discussion. The extract began with two requests from two committee members politely (“please yes please”) asking the consultant if “we” (the committee) could use an alternate term when referring to steps after high school. In Line 1, we see a request by a member of the committee (“could we . . .”) to include “post-secondary” instead of college. This request was quickly echoed by a second committee member in Line 2 (“yes please yes”) and the consultant repeated the request in Line 3 as a clarification. Rather than putting the request to the group for discussion at this point, the consultant accepted it in Line 5 (“absolutely”), followed by reasons in Lines 5 to 8 as to why this decision should be made. Here again, as in Extract 1, we see that it is the consultant, not the committee members, marking that a decision has been made. In fact, rather than suggesting, arguing, or making a motion, the committee members asked the consultant if they could add these terms. Although this may be productive and efficient for the work of the committee, it too points to the constitutive nature of the consultant’s contribution to the policy crafted by the committee.
Pattern 2: Deferring and Redirecting Decisions
In this section, we explore a contrasting pattern in which committee member suggestions were not validated by the consultants as decisions, but were instead deferred. Extract 3, from the sixth committee meeting (a full-day retreat with an 8-hr agenda), is an example of this pattern. In this meeting, both of the consultants from Education First were present. The day’s agenda included reviewing and voting on a rubric that would be used by principals to evaluate teachers during classroom observations. Because observations were worth up to 50% of a teacher’s evaluation, the decision about which rubric would be used as the state-endorsed model would impact thousands of teacher evaluation scores. During this meeting, the committee worked in small breakout groups to review and discuss sample rubrics. Then, they listened to a series of guest speakers from districts that had already tried each of the rubrics under consideration. Extract 3 is taken from the end of this day, at which time the consultant asked whether the committee had enough information to make a decision on which rubric to recommend (Lines 1-2).
Extract 3
1 Consultant 1: Well let me ask you a new point. Do you have enough 2 information to make a recommendation right now? 3 Chamber of Commerce president: No 4 Consultant 1: No. so we need more time 5 Entrepreneur: I don’t. 6 Consultant 2: In terms of uh ( ) 7 Entrepreneur: Not the one with the flowery words 8 Consultant 2: You like you prefer TAP? 9 Entrepreneur: I want not the one with the flowery words 10 Consultant 2: That one’s Marshall 11 Entrepreneur: I like TAP. I don’t like Marshall 12 Consultant 1: That we talked about how that seemed more subjective, 13 relevant, appropriate 14 Entrepreneur: It’s got a bunch of subjective words like cutting edge, 15 what’s cutting edge? 16 Consultant 1: Firm grasp. 17 Entrepreneur: I think there’s more to work from with the other one 18 Consultant 2: And of course this is just a sampling of rubrics there are 19 still several others that we could consider ( ).
In Lines 1 and 2, the consultant asked whether the committee is ready to make a recommendation on the rubrics. One committee member (chamber of commerce president) answered no, and this was then restated by the consultant in Line 4 (“No. So we need more time.”). Like Extracts 1 and 2, this follows the pattern of the consultant re-voicing and stating a decision. However, this time another committee member (entrepreneur) contradicted the chamber of commerce president in Line 5 by saying that he or she did not need more time. This answer, however, was not restated or treated immediately as a decision by either consultant, perhaps because two contradictory responses were on the table.
The next few turns worked to clarify that the entrepreneur was ready to recommend the TAP rubric but not the Marshall rubric (Lines 7-11). 4 In Lines 12 to 16, the consultants repeated some of the committee’s previously voiced concerns about the Marshall rubric (that it was too subjective); however, instead of treating the entrepreneur’s statement that “there’s more to work from” (Line 16) in the TAP rubric as a decision, the consultant countered that there were lots of other options, and the decision was deferred to a future meeting. Her use of the phrase “of course” worked to make the deferral of this decision an obvious next step.
In this way, the consultant’s turn changed the course of decision making by deferring the decision until a later date. The final decision on a rubric would not be made until after negotiations with vendors several months later. Unlike the previous extract, the consultant’s contribution to this interaction delayed a decision by closing discussion on the topic. Instead of encouraging the committee members to debate their level of readiness or their opinions on the options, the decision was deferred. This deferral was another way that consultants’ interactions are constitutive of the shape and nature of committee decisions. Indeed, by deciding that the first committee member’s answer to their question of readiness (“no”) would stand, the consultant’s own decision changed the course of the conversation. Ironically, a later vote selected the DC IMPACT rubric, but the availability of existing training materials and support services for TAP eventually won them the state contract anyway.
Another example of consultants deferring decisions occurred during the second meeting in which a controversial topic was raised during the discussion. In this meeting, a committee member (principal) brought up concerns about the impact of introducing high-stakes assessments of student learning in Kindergarten through second grades. The principal noted:
Extract 4
1 Principal: There’s definitely challenges with these pre-K through second 2 grade kids in terms of what’s developmentally appropriate . . . my 3 experiences with ThinkLink
5
so far has been that they’ve mailed us 4 back inaccurate scores, they’ve mailed us back scores from the wrong 5 testing period . . . no one can assure us that the test that was given in 6 September was comparable to the test that was given in December. So 7 you couldn’t truly measure whether there was growth um so we were 8 using a test and spending a lot of money on a test that wasn’t 9 necessarily accurate. And so you know there’s concern there. Um 10 there’s a concern with DIBELS
6
. . . often times very uh bright children 11 who are thinkers don’t read as fast because they’re stopping to ask 12 questions and think . . . the trend I saw when our district adopted um 13 DIBELS was that all of a sudden our teachers were getting training on 14 how to teach to the DIBELS test instead of how to teach good reading 15 strategies . . . I caution I think there’s a lot of danger in trying to put um 16 a standardized sort of traditional test in at the pre-K through second 17 grade level. I worry about what that could do to instruction . . .
In this extract, the principal described what she called “challenges” (Line 1) with using either of the two assessments being discussed for pre-K through second graders. She described her personal experience with inaccurate scores (Line 4), testing period confusion (Line 5), and lack of accurate growth measures (Line 7). The principal noted that she had seen teaching practices change (Lines 13-15) and used words like “caution” (Line 15), “danger” (Line 15), and “worry” (Line 17) when describing the use of these assessments.
In the following turns, the consultant began by “summarizing the things that are being said” in Lines 1 to 9 (below), during which the principal initially agreed with her summary.
1 Consultant: So to summarize the things that are being said, quality, accu- 2 racy, you know things measuring what 3 Principal: Right 4 Consultant: We’re wanting them to measure and also uh the reliability 5 certainly but also the purpose using the test 6 Principal: Right 7 Consultant: For purposes other than what they’re intended like the 8 DIBELS if it’s not intended to be used for a growth purpose. So I think 9 one of the things- 10 Principal: Well it is DIBELS is intended for growth purposes I mean you 11 get a chart on a student saying they were here 12 Consultant: Yeah right 13 Principal: And now they’re here so you can you can track growth but 14 whether it’s most aligned with I mean there’s a whole book about how 15 DIBELS has sort of been the undermining of reading instruction since 16 it’s been adopted so. I don’t have the name of the book but I have read 17 it. 18 Consultant: A book (unclear) 19 Principal: Well there are just definitely definitely challenges there that 20 when we’re creating an assessments to just assess teachers whether 21 they could actually hurt the quality of what’s going on in the class- 22 room if we go overboard with them . . .
In Lines 10 to 16, however, the principal interrupted the consultant’s summary to clarify that the DIBELS assessment is intended to measure growth and went on to justify his or her stance by referencing a book that is “about how DIBELS has sort of been the undermining of reading instruction” (Lines 13-15). At this point, rather than validating the principal’s concern or bringing the issue to the group for discussion, the consultant repeated, in a questioning tone “a book” (Line 18), after which the principal downgraded her criticism to “there are just definitely challenges” (Line 19).
At this point, another committee member (teacher) spoke up in defense of the accuracy of ThinkLink, based on the experience in his or her school.
1 Teacher: We use ThinkLink um here is their brochure, which he (a 2 ThinkLink representative) said that he’d be glad to come and talk to ya 3 . . . as far as a predictive indicator for TVAAS scores that they have 4 80-90% accuracy. Now we had a few problems a few quirks like that 5 too, they say there’s not that many but it hap- 6 pened to you. It it happened to me on one form in the last 5 years. One benchmark um out of 7 15 benchmarks with them I’ve done I did have an error and I caught it, 8 they corrected it but it was the training that goes with that learning 9 how to use it and that type thing.
The teacher here positioned the problems that the principal had outlined as just “quirks” (Line 4) that do not happen often (“one form in the last 5 years”). The teacher minimized the concerns expressed by the principal and offered to have representatives of ThinkLink come and talk with the committee (Line 2). At this point, there seemed to be conflicting views of the assessments on the table. Rather than using this as an opportunity for deeper discussion of the implications of their impending decision, we see in the continuation of the conversation below that the consultant redirected the discussion by deferring it until breakout groups convened. Like Extract 3, this is an example of the consultant serving as an arbitrator of disagreement between members. Rather than mediating, however, the consultant reframed and deferred the discussion until later (Lines 1-6).
1 Consultant: So when we have a breakout session I think it would be help- 2 ful to try and talk through some of these challenges a bit more. But 3 also to then think about uh noting the challenges what might be a 4 process to address them. Where what’s the process to select appropri- 5 ate assessments what research would we need. Whether that would 6 that be this committee or a sub-committee that included others but. 7 That’ll be good I’ll- I’ll- prompt that again but I think we really need 8 to dive into these implementation challenges.
While on one hand this put the controversy and the associated decisions back into the hands of the committee (rather than the consultant making a decision through validation of one committee member’s request), this also meant tabling the concerns raised by the principal, which ultimately were never raised again with the whole committee. In fact, the very test the principal used as a negative example (DIBELS) was included as an option in the final teacher evaluation policy for Tennessee. Rather than taking up the principal’s concerns about the validity and consequences of certain assessments, the consultant signaled that “implementation issues” should have further discussion in breakout groups, thus taking up the version of concerns offered by the teacher who had brought brochures and greetings from the company’s representative to the meeting. This characterization, unlike the principal’s, implied that the surmountable questions of logistics (“implementation”) were at issue rather than fundamental flaws in an assessment’s construct validity or an assessment company’s integrity. Unlike in Extracts 1 and 2, the consultant did not validate a committee member’s suggestion, but reshaped the issue under discussion even as she deferred the decision for a later time.
Another example of deferment is shown in Extract 5, from the eighth meeting, during which the committee was discussing whether Tennessee Value-Added Assessment System (referred to as TVAAS) should be used as a measure of student growth or of student achievement. While the controversy of value-added measurement is well documented (e.g., Amrein-Beardsley, 2012; Baker et al., 2010; Gabriel & Lester, 2013b), the state of Tennessee required its use within its First to the Top Act, so the TEAC was tasked with deciding how, not whether, to use this controversial measure. Nevertheless, conversations about what TVAAS meant and how it should be used were prime sites for controversy, as shown in Extract 5.
Extract 5
1 Chamber of Commerce President: Is teacher effectiveness a subjective 2 term? . . . 3 Teacher: If student achievement is not taking place then that teacher is 4 ineffective . . . if there is no student achievement then that’s the bottom 5 line there. Then that teacher is ineffective. 6 State Representative: So, achievement growth? 7 Teacher: And that’s where TVAAS comes in that’s what’s so great about 8 the state of Tennessee is that we have that indicator we have that 9 assessment that shows growth. 10 State Representative: So if a teacher is teaching the fifth grade and that 11 person winds up with a student that’s had two or three ineffective 12 teachers is the fifth grade teacher responsible for achievement stan- 13 dards or is that teacher responsible for achievement growth? 14 Teacher: Both. 15 State Representative: Hm.
Here we see a potentially controversial topic being raised as the State Representative questioned the use of TVAAS and whether it reflected achievement and/or growth. In Line 14, the teacher claimed it represented both, to which the state representative’s “hm” is a noncommittal response indicating either uncertainty, or possible disagreement. The teacher continued,
1 Teacher: However to me growth is more 2 State Representative: So they don’t meet the 5th-grade achievement stan- 3 dard that teacher is not effective? 4 Teacher: No I think that’s where we can add our 35% and our 15%. 5 Thirty-five percent out of 100%
7
is not that great. Everyone if every- 6 one has that TVAAS across the board if everyone starts out and it took 7 me a while to realize this because I wasn’t big in favor of everyone 8 having that. Does that make sense. It starts everyone out on an even 9 level. Growth is as important as achievement. But they’re different. 10 Growth says everyone in this room has made progress 11 Consultant: Is it ok if we um table the conversation on growth for just a 12 second so that we can talk just to make sure that we stick to the rubric 13 for a minute cuz um there’s a a lot . . .
In Line 2, the representative requested clarification that not meeting an achievement standard would equate to ineffectiveness. The teacher refuted this conclusion and attempted to distinguish between growth and achievement in Lines 4 to 9, but the consultant in Line 10 chose to redirect this controversial conversation about interpreting TVAAS data with a request to “table the conversation” (Line 11) about the circumstances under which TVAAS could provide valid growth or achievement data. In this case, the consultant(s) neither validated nor presented an alternate view of TVAAS, but instead tabled and redirected the topic. This resulted in the committee sidestepping the complex issues surrounding the inclusion of TVAAS as a component of evaluation under law (Gabriel & Lester, 2013a). State law mandated that 50% of a teacher’s evaluation would be based on student performance, 35% of which would be from TVAAS scores where available, but the decisions about what would count as evidence for the other 15% hinge on whether TVAAS is viewed as a measure of absolute achievement or growth. Persistent confusion about what the TVAAS score represents, especially this late in the policy-making process (the 7th of 10 planning meetings) is troubling.
Indeed, questions of validity related to TVAAS and other value-added measures are as plentiful as they are complex (McCaffery, Lockwood, Mariano, & Steodji, 2005), and thus, it was likely neither efficient nor realistic to work them out within TEAC meetings. Still, if those responsible for crafting policy and implementation guidelines are unclear about the nature of the measures they include, their decisions are suspect. Deferring a decision may have been warranted in the name of efficiency and focus at this juncture late in the policy-making process, but it also reinforces a consequential pattern of minimizing discussion around value-added measurement, which is present not only in the talk of this meeting (Gabriel & Lester, 2013a) but also in the media (Gabriel & Lester, 2013b). In this way, Extract 5 serves as an example of consultant influence that may have been welcomed, but still had great consequence precisely because of its formative and substantive impact on the topic, duration, and conclusion of discussions.
Through analysis of the conversational data, we see that the consultants’ conversational moves functioned to position the outside consultant as the one who has the ultimate control—particularly in determining what counts as a group’s decision. Such tasks were not entirely left to the appointed committee members as would commonly be understood to be the case. As demonstrated above, consultant talk functioned to shape and manage discussions of some of the most controversial issues facing the committee, including what assessments are allowed to count as measures of student achievement, and which protocol for teacher observation should be used to operationally define effective teaching practice and the inclusion of value-added measures.
In the following section, we discuss possible implications of these patterns for understandings of the impact of outside consultants and the nature of controversial decisions made by committee.
Discussion
Our turn-by-turn analysis of decision-making moments within the talk of the Tennessee TEAC demonstrates that consultants can have a profound impact on decisions “made by committee.” This impact is managed in patterned ways, including: making decisions through validation and deferring and redirecting decisions. We have described how micro-level interactions function to position consultants as final arbiters and chief decision makers because of the very nature of their role as a group facilitator (Boden, 1995; Schwartz & Davidson, 2005). Our findings do not suggest that facilitators, or paid consultants, are inherently undemocratic. Instead, we aim to equip readers with awareness of cues that decision making is shifting in conversations by highlighting the discursive patterns that positioned the facilitator, rather than the group as a whole, as chief decision maker. Awareness of the patterns of discourse associated with facilitator-centered decision making may help assist in increasing transparency in the policy-making process and equipping participants with a means to interrupt these patterns if desired. As Stokoe, Hepburn and Antaki (2012) have argued, by illuminating interactional mechanisms through micro-analysis, discursive psychology can
reveal where interactional practices go “wrong,” and how they might be put “right.” Indeed, DP/CA recommendations will be an order of magnitude more precise and detailed than the kind of generalized advice one sees in text-books; based on folk theories or experiential reports of interaction, or on simulated encounters (“use open-ended questions”; “listen actively”, etc. see Stokoe, 2011). (p. 487)
We argue that awareness of the patterns of discourse involving the consultant in this particular study can be fruitful, if not generalizable, because it alerts researchers and participants of similar committees to the patterns they may wish to identify, encourage, avoid, or interrupt. This notion of the fruitfulness of awareness is central to the application of discourse analysis for “implementing social change, particularly when the matter at hand comprises recurring interactional business” (Stokoe, Hepburn, & Antaki, 2012, p. 487).
Tennessee’s TEAC was among the first of many similar committees across the United States that met to craft state-level teacher evaluation policies over the last 4 years. As a first-round R2T winner, Tennessee set a high profile example both for policy and process of Tennessee’s legislation, which set precedents for everything from the assessments used to measure student achievement to the protocols used for observing teaching practice, as well as the possible roles of consultants in teacher evaluation reform.
While at times useful and necessary, the constitutive nature of the consultant role can be problematic if not made explicit. First, the public face of a committee includes the names and organizations represented by those who were officially named/nominated to serve on the committee. Outside consultants are often beneath the radar of public scrutiny as they are not considered to be nor held accountable as voting members. This is similar to Bulkley and Burch’s (2011) assertion that contracting can distance the providers of educational services from the constituents and democratically elected bodies they serve (e.g., Burch, 2009; Gold, Cucchiara, Simon, & Riffer, 2005). In addition, their status as non-voting members is perhaps most problematic as it suggests they have little to do with final decisions, though our findings suggest they have more to do with the construction of decision points than any single vote of an appointed committee member. Thus, the public’s perception of “decision by committee” may well be inaccurate when meetings are facilitated by an outside consultant (Koppich & Esch, 2012; Townsend, 2009).
The influence of outside consultants can also be viewed as problematic because it may be in conflict with the ideals of a democratic process. When the responsibility to present evidence, guide discussions, and mark decisions is held by a few consultants, the structure and content of meetings are no longer equally shared, and neither are the outcomes. A committee led by outside consultants may no longer in actuality be making decisions “by committee” or representing the interests of stakeholder groups in a balanced way. The committee’s decisions are instead reflective of what the consultant has presented and organized and the ways in which they have structured the participation of committee members. Given the steadily increasing involvement of educational consultants and contractors, and the connections between such involvement and larger trends toward privatization (Bulkley & Burch, 2011), a clear understanding and explicit disclosure of the role of consultants in policy-making processes is increasinglyimportant.
Involvement of consultants may be a reality born out of the need for efficiency, but it should also involve a change in the public billing of such processes. Rather than labeling draft policies as the work of a committee, one might more accurately label such work as the decisions of the committee as guided or directed by their paid consultants. For example, the handbook for Connecticut’s pilot teacher evaluation system (Connecticut State Department of Education, 2012) indicates that it is “based on the Connecticut Guidelines for Educator Evaluation, developed by a diverse group of educators in June 2012,” but that the core design principles were “developed in partnership with Education First” (p. 10).
As the governor reported attending to the balance of race, gender, and perspective when appointing members to the committee (TN.gov. Newsroom, 2010), it follows that such attention should also be paid when selecting the person who will facilitate the talk of the committee. The consultants’ own ideas, perspectives, and commitments, including commitments to their firm’s continued success and involvement with the state, should be considered carefully in a setting where their talk is so fundamentally constitutive of the work of the committee. Likewise, the possibility that a paid consultant necessarily represents their firm’s interests, including the desire to be rehired for future efforts, could represent a conflict of interest. Indeed, Education First was selected, in part, because of their previous involvement as contractors working on the state’s initial R2T application (meeting minutes, May 10, 2010, retrieved from http://tn.gov/firsttothetop/programs-committee.html). This is especially important to consider in the high-stakes context of multimillion dollar education reform efforts. Since the completion of the work of the TEAC in Tennessee, Education First has consulted on the state evaluation policies of at least five other states, and is well positioned to continue to provide such support to advisory committees across the country.
By pointing this out, we do not imply any indictment of the organization or others like it, rather we point out the importance of attending to the influence of consultants on committee decision-making processes and outcomes, especially those in which a set of stakeholders have been brought together to represent a range of perspectives and interests. Within this data set, consultant talk often functioned to redirect rather than encourage deeper discussion of complex and controversial issues.
One of the limitations of this study is that it focused exclusively on talk as recorded in public meetings. We understand, however, that decisions may have been made, and policy language written, outside of these meetings. Indeed, it is what happens to policies outside of policy-making settings, especially during implementation, that matters most when considering the impact they have on educators, students, and opportunities to learn (Coburn, 2001; Spillane, Reiser, & Reimer, 2002). Although we may not have access to the full evolution of each decision for analysis, patterns in the role of consultant talk in transcripts of the committee’s public meetings suggest a need for more investigations of the micro- as well as macro-level influences of outside consultants on education policy-making processes.
The benefits of professional facilitation in terms of information gathering, organization and time management may well outweigh the possibility that a purely democratic committee decision might be tainted. In these cases, awareness of the ways in which consultants influence when and how decisions are made might support committee members in acknowledging and/or resisting such patterns if or when they do not serve the best interests of their constituents.
Footnotes
Appendix
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.