Abstract
Problem solving is fundamental to psychoeducational assessment practices and generally grounded in activities related to identifying problems, developing and refining hypotheses, generating solutions, developing and implementing actions, and evaluating outcomes. While the process is central to response-to-intervention practices as well, little research has addressed the form, content, or outcomes of decision-making teams as they operate in schools. One barrier to building a program of research on team problem solving has been the absence of a credible and feasible measure of team performance. We developed the Decision Observation, Recording, and Analysis (DORA) tool to document problem-solving behaviors during team meetings. We were interested in evaluating problem solving during team meetings that focus on academic and behavior concerns in school. We describe the development and preliminary psychometric data for DORA in this article. Our discussion focuses on the implications of DORA for expanding the study of team processes and for improvement of problem-solving practices in schools.
Most education professionals agree that schools should adopt a systematic, team-based process for using data to inform classroom instruction and support teachers’ efforts to meet the individual academic and social learning needs of their students (Newton, Horner, Algozzine, Todd, & Algozzine, 2009). For school psychologists, problem solving focused on “data-based decision making and accountability” is part of “practices that permeate all aspects of [professional] service delivery” (National Association of School Psychologists [NASP], 2010b, p. 4; Ysseldyke et al., 2006). As Tilly (2008) points out, this “problem-solving method” is logical, easy to understand, and “something people do every day”; and, it has been proposed under many names (pp. 18-19). For example, the IDEAL (Bransford & Stein, 1993), the scientist practitioner (Barlow, Hayes, & Nelson, 1984), and the Heartland Area Education Agency 11 (Reschly & Ysseldyke, 1995) models as well as applied behavior analysis (Baer, Wolf, & Risley, 1968), behavioral consultation (Bergan & Kratochwill, 1990), curriculum-based measurement (Deno, 1985; Shinn, 1989), functional behavior assessment (Repp & Horner, 1999), and positive behavior interventions and supports (Sugai et al., 2010) provide frameworks for this type of decision making.
In contemporary practice, driven in large part by reauthorizations of Public Law 94-142 now codified as the Individuals with Disabilities Education Improvement Act of 2004, problem solving is embodied in response-to-intervention (RtI) methods that have emerged as the preferred practices for addressing identification, prevention, and improved education for the “most vulnerable, academically [emphasis added] unresponsive children” in schools and school districts (Fuchs & Deshler, 2007, p. 131); in fact, four of six of RtI’s essential and “core defining features” relate to problem solving and data-based decision making (Brown-Chidsey & Steege, 2005; Sugai & Horner, 2009, pp. 225-226):
Standardized problem-solving protocol for assessment and instructional decision making.
Explicit data-based decision rules for assessing student progress and making instructional and intervention adjustments.
Emphasis on assessing and ensuring implementation integrity.
Regular and systematic screening for early identification of students whose performance is not responsive to instruction.
Schoolwide positive behavior support (SWPBS) has also emerged as a preferred practice for preventing and addressing academic and social behavior needs of students and the assessment and decision-making principles of RtI share important features (e.g., a team charged to regularly meet to address school-based problems) with it (Sugai et al., 2010; Sugai & Horner, 2009).
Decision-Making Models
Problem solving typically involves cyclical steps (e.g., collecting information from a variety of sources, transforming the information into testable hypotheses, selecting, implementing, and evaluating interventions that test the hypotheses, then adapting or revising the interventions based on fidelity and impact data). In a variety of formats, team-based consultation and using data to make decisions that is an integral part of problem solving has been recommended for many years (Bergan & Kratochwill, 1990; Boudett, City, & Murnane, 2006; Deno, 1985, 2005; Hamilton et al., 2009; Tilly, 2008).
Effective instructional decision making is collaborative, and the continuous analysis of data to identify and document the resolution of problems data is one of its central features (Brown-Chidsey & Steege, 2005; NASP, 2010a, 2010b; Newton et al., 2009; Tilly, 2008; VanDerHeyden, Witt, & Barnett, 2005). While “the problem-solving method” has been the “foundation for science-based practice in schools” for a long time (Tilly, 2008, p. 18), its longevity has not correlated with systematic investigation or evidence of its use or value in improving school-based decision making. For example, little scholarship has addressed the extent to which school professionals follow systematic steps during team meetings. There are also few investigations documenting activities engaged in during team meetings or changes resulting from systematic professional development to direct and improve decision-making practices.
We were interested in evaluating decision making during SWPBS team meetings. A variety of tools exist for evaluating the general fidelity of implementation of the core features of SWPBS; however, there are no instruments available for documenting the extent to which steps in the problem-solving logic set are followed during school-based team meetings (cf. Algozzine et al., 2010). For example, the Self-Assessment Survey (Sugai, Horner, & Todd, 2003) is administered to the entire school staff to evaluate schoolwide, nonclassroom, and classroom support systems as well as those for individual students. The Schoolwide Evaluation Tool (Horner et al., 2004) is a 2- to 3-hr schoolwide review conducted by an external evaluator to (a) assess critical features that are in place, (b) determine annual goals for ongoing behavior support, (c) evaluate progress, (d) redesign and revise extant procedures as needed, and (e) provide a basis for year-to-year evaluation efforts; and, similar purposes are served by the Team Implementation Checklist (Sugai, Horner, Lewis-Palmer, & Rossetto-Dickey, 2011), and the Benchmarks of Quality (Cohen, Kincaid, & Childs, 2007). The Benchmarks for Advanced Tiers (Anderson, Childs, et al., 2010) and the Individual Student Systems Evaluation Tool (Anderson, Lewis-Palmer, et al., 2010) are measures of the implementation status of Tiers 2 (secondary, targeted) and 3 (tertiary, intensive) behavior support systems within a school. While information from these measures has value in judging implementation fidelity and impact of SWPBS, it provides no indication of the extent to which effective processes are followed when school teams engage in data-based decision making. We developed the Decision Observation, Recording, and Analysis (DORA) tool to document activities and adult behaviors during SWPBS meetings and to provide a basis for conducting analyses of the relationship between teaching these teams how to systematically solve problems and achieve improvements in the quality of their school-based meetings (Newton, Todd, Horner, Algozzine, & Algozzine, 2009; Todd, Newton, Horner, Algozzine, & Algozzine, 2009).
In this article, we document the development and emerging technical characteristics of the DORA. We believe our work has implications for decision making related to SWPBS as well as for more broadly directed psychoeducational assessment problem-solving practices that “permeate all aspects of service delivery” in general, special, and remedial education (NASP, 2010b, p. 4).
Method
Our effort to develop an instrument to document team-based decision making was grounded in processes of (a) identifying problems, (b) developing and refining hypotheses, (c) generating solutions, (d) developing and implementing actions, and (e) evaluating outcomes (see Figure 1). The technical characteristics that we were interested in were validity, reliability, and sensitivity of data gathered using DORA.
Team-initiated problem solving (TIPS) model
The Decision Observation, Recording, and Analysis (DORA)
Based on a review of documents addressing the conceptual and practical guidance for effective team meetings (see Anderson, 1994; Bradford, 1976; Grocz & Denson, 1988; Lencioni, 2005; Mackin, 2007; Perkins, 2009; Tobia & Becker, 1990) and team-based problem solving, we included two sections in DORA. Critical features of the meeting foundations for effective problem solving that should be in place at the start, during, and at the end of meetings comprise the first part; and the five processes of effective team problem solving are represented in the second part of the instrument (see appendix).
Meeting foundations
The “structure” of meetings (e.g., how a team prepares, conducts, and manages the follow-up activities) is important to their effectiveness. Critical features to be observed at the start of a meeting include whether an agenda was distributed, team roles were established, team members were present, relevant data were reviewed, and the meeting started “on time.” During the meeting, quantitative data should be distributed or projected, the status of one or more previous decisions/tasks regarding student social or academic behavior should be reviewed, and the fidelity and impact of one or more implemented decisions/tasks regarding student social or academic behavior should be discussed. At the close of the meeting, the minutes should be distributed; the date and time of the next meeting should be confirmed; and, attendance at the beginning and end of the meeting as well as whether it ended “on time” should be recorded.
Processes of effective problem solving
Because the process of solving problems is iterative, we reasoned that observers using DORA would also record the cycles of problem-solving and decision-making processes used by team members as they address social or academic problems. Each “problem” is recorded in a single row that includes information about the problem being addressed by the team (e.g., who, what, when, where) and reasons or hypotheses for why it was occurring, the type of data reviewed, the purpose of the data review, whether the team generated possible solutions for solving the problem, the type of action(s) the team decided to implement, the specific action(s) the team decided to implement, and the type of evaluation accountability the team documented once a decision was reached and recorded.
Data from DORA are collected in real time by an observer who is present for a full team meeting (or at least 70 min). The instrument (see appendix) provides evidence of and results in an overall score for “meeting foundations” (i.e., percentage of 12 items observed during meeting) and a count of the number of problems identified during the meeting. In addition, problem-solving process subscale percentage scores are provided for (a) precision of problem definition, (b) use of data to refine the problem or build a solution, (c) elements of a problem solution, and (d) action plan for solution implementation as well as for “thoroughness of problem solving” (i.e., average of scores for four subscales). The intent in using and scoring the DORA is to document levels of critical features of effective problem solving rather than to record achievement of predetermined standardized or benchmarked scores.
Technical Characteristics
We focused DORA on the observable behaviors of team members as they (a) managed meetings, (b) identified problems, (c) developed solutions to those problems, and (d) built action plans to implement their selected solution(s). We used multiple methods to evaluate the extent to which repeated use of DORA produced similar results (i.e., reliability), the extent to which use of DORA produced data reflective of what we intended it to measure (i.e., validity), and the extent to which DORA provided evidence of changes in decision making resulting from professional development.
Reliability
Observer training was established, assessed, and improved by completing an “observe, review, and revise” cycle. We used sets of scores recorded by these trained observers to document overall agreement scores as well as item-by-item occurrence agreement indices (Gall, Borg, & Gall, 1996; Hawkins & Dotson, 1975; Suen & Ary, 1989).
Validity
Johnston and Pennypacker (1993) argued that establishing evidence of the validity of measures used to gather direct observations is unnecessary given that they generally involve documenting occurrences of behavior as a basis for drawing conclusions. Since we were interested in measuring observable behavior representative of team decision making, we documented the content validity of DORA to provide evidence that information collected was consistent with the underlying knowledge base (i.e., the scale contains items that accurately and adequately represent the content of interest). We also correlated and compared scores on DORA with team members’ perceptions of selected problem-solving abilities.
A tool for assessing decision-making behaviors should include items that professionals agree represent essential components of decision making. In reviewing the literature, we found that effective team meetings have a “structure” of environmental supports (e.g., an agenda, data summaries, roles assumed by team members, Action Plan or minutes from previous meetings) and consistent use of problem-solving processes (e.g., defining a problem, using data) regardless of the context in which they function (Newton, Horner, et al., 2009). We evaluated the content validity of the DORA using a variation of the “Content Validity Ratio” (CVR) approach recommended by Lawshe (1975). Our goal was to determine the extent of agreement between expected and actual content in our instrument (e.g., Are included items addressing areas that are recommended as critical and essential to the effective problem solving?). We cross-tabulated content representative of decision-making models with areas addressed in DORA. We assumed adequate content validity if our content represented 80% or more of that included in widely accepted decision making models.
Sensitivity
The Social Problem-Solving Inventory–Revised (SPSI-R) is a self-report survey that provides indicators of respondents’ problem orientations and approaches and styles for resolving everyday problems (D’Zurilla & Nezu, 1990; D’Zurilla, Nezu, & Maydeu-Olivares, 2002, 2004). The total SPSI-R score is a global indicator of problem solving and five scale scores are used to reflect strengths and weaknesses in orientations, approaches, and styles of problem solving. The Rational Problem Solving (RPS) scale reflects behaviors reflective of “. . . deliberate, systematic, and skillful application of effective or adaptive problem-solving principles and techniques” (D’Zurilla et al., 2002, p. 4). As reported in the Technical Manual (D’Zurilla et al., 2002, pp. 56-64), the psychometric properties of the measure and its subscales are very good (e.g., internal consistency ranged from .80 to .95 and test-retest reliability ranged from .68 to .91; concurrent and predictive validity coefficients were high; sensitivity was supported by positive correlations between changes in psychological and physical health and changes in SPSI-R ratings). The technical adequacy of the survey for research purposes and its limitations as a self-report measure have been documented (cf. Lindsey, 2005; Owens, 2005). The SPSI-R is available in two forms: The short version (SPSI-R:S) includes 25 items and requires about 10 min to complete and the long version (SPSI-R:L) contains 52 items that can be completed in 15 to 20 min. The authors recommend using the long form whenever possible because it “seeks more information, especially pertaining to specific problem-solving skills” and “it is able to provide a more comprehensive picture” of how people solve problems (D’Zurilla et al., (2002) p. 7).
We asked team members to complete the SPSI-R:L independently. Because collective decision making occurs during meetings, we computed a team’s mean total score and the team’s mean scores for each of the scales and subscales of the SPSI-R:L. Ideally, to assess concurrent validity and sensitivity, we would correlate scores reflective of team decision making with DORA scores; however, the absence of any formal or informal measure created the opportunity for a next best evidence approach grounded using summative measures across individual team members as the best approximation of group behavior. We do not believe this concession represented a significant limitation in our work. We correlated scores on the DORA with RPS subscale scores and compared changes on DORA and the SPSI-R:L before and after professional development.
Results
Because the “foundations” of effective meetings are well documented in the business, psychology, and education literature, we confined our content validity analysis to five stages of problem solving included in DORA. We observed complete correspondence between these processes and the critical features of five widely accepted problem-solving models (see Table 1). A high degree of congruence was also evident in our analysis of the content reflected in the Rational Problem Solving dimension of the SPSI-R:L and the stages of decision making assessed in DORA (see Table 2). The problem definition and formulation items on the SPSI-R:L reflect behaviors included in the “review status and identify problems” and “develop and refine hypotheses” steps of the TIPS model (see Figure 1). Key aspects of “discussing and selecting solutions” in our model are included in the generation of alternative solutions items on the SPSI-R:L. The decision-making items on the SPSI-R:L reflect behaviors included in the “discuss and select solutions” stage of the TIPS model. Key aspects of “developing and implementing” and “evaluating and revising” an action plan in our model are included in the Solution Implementation and Verification items on the SPSI-R:L.
Cross-Tabulation of Content in DORA and Problem-Solving Principles From Practice
Relationship Between SPSI-R and TIPS Activities
A completed DORA provides an overall Thoroughness score as well as scores for dimensions of problem solving (e.g., Problem Precision, Action Plan). These scores reflect evidence of the extent to which the observer documented specific behaviors during an ongoing team meeting. A completed SPSI-R:L has scores for five scales one of which (i.e., Rational Problem Solving) “. . . assesses the rational, deliberate, and systematic application of effective problem-solving strategies and techniques” and has scores for four subscales including (i.e., Problem Definition and Formulation, Generation of Alternative Solutions, Decision Making, and Solution Implementation and Verification) that reflect the respondents’ perceptions of how they “might think, feel, and act when faced with problems in everyday living” (D’Zurilla et al., 2002, p. 31). We correlated DORA overall Thoroughness scores with scores for its decision-making dimensions reasoning that positive interrelationships would support an “integrated” underlying construct for the areas being observed (as predicted in the literature) and zero-order relationships would reflect “independent” or discrete categories of information. The correlation was moderate to high between the overall Thoroughness score and DORA Dimensions of Problem Precision (r = .80), Quantitative Use of Data (r = .79), Solution/Decision (r = .70), and Action Plan (r = .64). We also correlated DORA scores with SPSI-R:L scores; low correlations (Range = .00 to .33) suggest that the measures represent independent indicators of problem solving and decision making or that DORA and SPSI-R:L scores provide evidence of similar constructs differently.
Reliability
We documented interobserver agreement for DORA data in a series of preliminary analyses and pilot studies. We calculated the percentage of agreement between pairs of observers by comparing meeting foundation element scores and decision-making thoroughness scores at 20 meetings in a single-case research project (Todd et al., 2011). Reliability for meeting foundation scores averaged 94% (range 72% to 100%) and interobserver agreement for thoroughness scores averaged 88% (range 50% to 100%). We also documented agreement for problem type, data use, selected solutions, and action plan indicators across independent observers at 3 consecutive pilot test (Newton, Horner, et al., 2009) team meetings. The average agreement across observers was 85% for the team’s use of the foundational elements and ranged from 50% for the team’s identification of a problem to 92% for type of problem identified by the team, including agreement of 84% for the data use, 78% for solutions, and 85% for action plan elements. Average interobserver agreement for thoroughness was 85%.
We compared DORA and SPSI-R:L scores across intervention conditions as a final technical characteristic indicator. We predicted that observable behaviors assessed using DORA would be more likely to reflect changes in decision making following intervention than perceptions of attitudes assessed using SPSI-R: L. In this context, DORA scores would reflect “state” indicators susceptible to change and SPSI-R:L scores would reflect more stable indicators reflective of underlying decision-making “traits” (Newton, Horner, et al., 2009). As expected, SPSI-R:L scores varied less than 2 scaled scores across intervention conditions while differences in DORA scores were consistently large (Range = 22-40 percentage points).
Discussion
Federal regulations (e.g., Public Law 94-142 and its subsequent reauthorizations) and recommendations providing guidance to support the education of individuals with disabilities directed that evaluation and placement procedures be made by multidisciplinary teams. As Pfeiffer (1982) pointed out,
architects of [early legislation] undoubtedly believed that group decisions would provide safeguard against individual errors in judgment, while recognizing that only a group of specialists from different professions could deal effectively with the increasingly complex set of problems facing special education (p. 68)
and early research illustrated that the majority of team time was spent discussing instructional goals and objectives, team decisions were more consistent than those made by individuals, almost half the time in a meeting was spent discussing assessment information, and general classroom teachers often participated very little in team meetings (Bartels & Mortenson, 2005; Pfeiffer & Naglieri, 1983; Ysseldyke, Algozzine, & Mitchell, 1982). Put another way, while many believe team-based decision-making practices are important, few have documented their quality, and evidence on data-based problem solving is sparse.
In schools implementing positive behavior support, teams are expected to use data to improve academic and social outcomes for students. We were interested in studying these teams and developed the DORA to provide data on their problem-solving practices. We reasoned that we needed a measure that would be reflective of the processes that professionals believe should be taking place at these meetings and sensitive to changes in them. In reviewing the literature, we found consistent support for the processes professionals believe are core features of data-based problem solving. Most research to date has relied on perceptions and documentation of similarities and differences in special education decisions made by teams using different decision-making models compared to individuals or school psychologists use of psychometric data in classifying students with learning disabilities (cf. Burns, Scholin, Kosciolek, & Livingston, 2010), but there has been little focus on documenting and teaching core decision-making processes or on testing the effects of such professional development on teams and/or the decision made by them (cf. Bartels & Mortenson, 2005). We designed and field-tested DORA to meet this need.
Our preliminary psychometric data suggest that DORA focuses on processes that professionals believe define effective and efficient team decision making, it can be used with interobserver agreement that makes the scores trustworthy, and it appears to be much more sensitive to change than measures that focus primarily on perceptions of decision makers. In this context, it provides support for professionals conducting research focused on documenting and improving problem solving by teams. DORA also has value for practitioners using assessment information to develop effective interventions, including but not limited to providing a method and measure for documenting the extent to which critical features are evident at RtI and other data-based problem-solving team meetings.
Limitations
While the foundations and processes that we observed were consistent across school-based teams observed, we only documented similarities and differences for groups engaged in problem solving focused on social and academic problems within the context of SWPBS. The absence of a “gold standard” on which to evaluate concurrent validity inherent in many scale-development studies also restricted our work and support the need for additional research and data to more thoroughly establish the usefulness of DORA.
Implications for Future Research and the Improvement of Practice
Problem solving is at the core of effective psychoeducational assessment and the practice of school psychology (Burns et al., 2010; NASP, 2010a, 2010b; Tilly, 2008). As Hosp and Ardoin (2008, p. 69) point out, the reasons for this importance
. . . range from legal (Individuals with Disabilities Education Improvement Act [IDEIA] of 2004; No Child Left Behind [NCLB] Act of 2001) to professional and ethical (Council for Exceptional Children [CEC], 2003; Joint Committee on Standards in Educational and Psychological Testing, 1999; NASP, 2003, 2010a, 2010b) to logical and practical (Salvia, Ysseldyke, & Bolt, 2007; Tilly, 2008)
and the process is being applied to improve the academic and social behaviors of students in districts and schools across the country. In related research, we provided professional development related to and showed improvements in problem-solving practices of positive behavior support teams (Newton, Algozzine, Algozzine, Horner, & Todd, 2011; Todd et al., 2011). We used DORA to document initial and continuing levels of critical features of effective data-based decision making across positive behavior support teams. We believe our measurement tool has applicability beyond teams engaged in addressing social and academic problems in schools. The features demonstrated to be acceptable and successful for the teams focus on important consultative skills that enhance problem solving in any setting and for any educational purpose. Future research documenting decision-making practices across different types of teams will add to the emerging value of DORA in efforts to improve the process of using data to inform classroom instruction and support school psychologists’, teachers’, and other professionals’ efforts to meet the individual academic and social learning needs of all students.
Footnotes
Appendix
The opinions expressed herein are those of the authors and no official endorsement should be inferred.
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Preparation of this article was supported in part by Grant R324A070226 from the Institute of Education Sciences, U.S. Department of Education.