Abstract
The authors examined three instructional approaches for improving teachers’ curriculum-based measurement (CBM) graph comprehension, each differing in the extent to which reading the data, interpreting the data, and linking the data to instruction were emphasized. Participants were 164 elementary school teachers who were randomly assigned to one of three CBM instructional approaches or a control condition. Instruction was delivered via videos. Prior to and after receiving instruction, teachers completed a CBM graph-comprehension task. They also evaluated the instructional videos. Teachers in the three instructional groups improved more in CBM graph comprehension than control teachers. Improvements were seen primarily in interpreting and linking the data to instruction, two important but difficult aspects of CBM graph comprehension. Differences between the instructional groups were found for interpreting the data. Teachers evaluated the videos positively. Results indicate that teachers’ CBM graph comprehension can be improved via video instruction. Implications for teaching teachers to implement CBM are discussed.
Curriculum-based measurement (CBM) is a tool that enables teachers to closely monitor progress and evaluate the effectiveness of instructional programs for students with learning disabilities (Deno, 1985). CBM involves frequent (e.g., weekly) data collection on brief (e.g., 1–3 minutes) probes that sample student performance in an academic area. Scores from the probes are placed on a graph that depicts student progress. The elements of the graph include:
baseline data, representing the beginning level of performance for the student and peers;
a goal line, extending from the median baseline point to the end-of-year goal, representing the student’s expected rate of growth;
data points, representing the student’s weekly performance on CBM probes;
a slope line drawn through the data, representing the student’s actual rate of growth in response to the instruction;
vertical lines, indicating when a change in instruction has been made (see Figure 1 for a sample graph).
Sample curriculum-based measurement progress graph.
When implementing CBM, the teacher collects baseline data, sets the long-range goal, administers weekly probes to the student, and places the scores on the progress graph. After a set number of weeks, a slope line depicting student growth is drawn through the data. The teacher inspects the graph to determine whether the student is progressing at the expected rate, and thus whether the instructional program is effective. If the student is progressing at or above the expected rate, instruction continues as it is, and the goal may be raised. If the student is progressing below the expected rate—that is, if the slope line is below and less steep than the goal line (see Phase 1 in the sample graph)—the teacher changes the instruction to adapt to the needs of the student and continues to collect data. After a set number of weeks, a slope line again is drawn through the data, and the teacher inspects the graph to evaluate the effect of the instructional change. If the slope line is still below and/or less steep than the goal line, the teacher once again changes the instruction (see Phase 2 in the sample graph). This approach to CBM data-based decision making is referred the as the trend or slope-line approach (for a description of other approaches, see Deno, 2016; Hintze, Wells, Marcotte, & Solomon, 2018; Stecker, Fuchs, & Fuchs, 2005).
Use of CBM progress monitoring can be especially important for teachers who work with students with severe and persistent learning difficulties. These students may improve slowly and may not respond to instruction in the same way as other students. Through the recursive cycle of “data inspection – instructional decision – data inspection – instructional decision”, teachers can build powerful and effective instructional programs for students (e.g., see Stecker, Lembke, & Foegen, 2008). Furthermore, teachers can use the CBM graphs to communicate information about student progress to parents or colleagues.
Research has demonstrated that when teachers respond to CBM data with instructional and goal changes, student performance improves; however, research also has demonstrated that teachers often do not respond to the data, at least not without decision-making supports (Stecker et al., 2005). In the late 1980s and early 1990s, D. Fuchs, L. S. Fuchs, and colleagues investigated the effects of decision-making supports delivered via computer software on teachers’ CBM implementation and data-based decision making (for reviews, see Fuchs & Fuchs, 1989, 2002; Stecker et al., 2005). The supports assisted teachers in collecting, scoring, and graphing the data and prompted them to change instruction or raise the goal when necessary. In later stages of development, the supports provided diagnostic skills analysis and expert feedback to guide teachers’ decisions about what to change in their instruction and how to change it. The use of the computer supports led to improvements in teachers’ implementation of CBM decision rules, selection of appropriate instructional changes, and design of diverse educational programs, which in turn led to improvements in student performance (Fuchs & Fuchs, 1989, 2002; Stecker et al., 2005).
Although the computer software provided teachers with support, the teacher remained an important element in the decision-making process. For example, teachers were more accurate in timing instructional changes if they first formulated and entered instructional decisions and then received computer feedback than if they received computer feedback alone (Fuchs, Fuchs, & Hamlett, 1989a). Fuchs and Fuchs (1989) speculated that sole reliance on computer supports might serve to distance teachers from the data, reducing effective data-based decision making. Despite the important role of the teacher in CBM data-based decision making, researchers only recently have begun to examine teachers’ CBM data-based decision-making processes. Research in this area has focused thus far on the first step in the decision-making process: reading and interpreting the CBM progress graphs.
Teachers’ Comprehension of CBM Progress Graphs
The ability to read and interpret graphs is often referred to as graph comprehension (Friel, Curcio, & Bright, 2001). Friel et al. (2001) described three levels of graph comprehension: reading the data (extracting data from the graph), reading between the data (integrating and interpreting the graphed data), and reading beyond the data (evaluating and interpreting data within a given context). Applied to CBM graphs (van den Bosch, Espin, Chung, & Saab, 2017), these levels have been referred to as reading the data (describing the CBM data as they appear on the graph), interpreting the data (integrating and interpreting relationships between CBM graph elements, e.g., between the slope and the goal), and linking the data to instruction (evaluating and interpreting CBM data within the instructional context).
Two early studies of teachers’ CBM graph comprehension (Espin, Wayman, Deno, McMaster, & de Rooij, 2017; Wagner, Hammerschmidt-Snidarich, Espin, Seifert, & McMaster, 2017) employed a think-aloud method to examine teachers’ ability to describe CBM progress graphs. Results revealed that both inservice (Espin et al., 2017) and preservice (Wagner et al., 2017) teachers had difficulties with CBM graph comprehension, that years of experience implementing CBM was not related to CBM graph comprehension (Espin et al., 2017), and that preservice teachers comprehended CBM progress graphs less well than CBM experts (Wagner et al., 2017). These initial studies were limited in that they had relatively small sample sizes, focused primarily on the most basic level of graph comprehension (i.e., reading the data), and did not compare CBM graph comprehension of inservice teachers to CBM experts. Moreover, it was not clear from these studies whether difficulties with CBM graph comprehension were unique to teachers.
In a subsequent study, van den Bosch et al. (2017) employed the think-aloud method to compare teachers’ CBM graph comprehension to that of three types of “gold standard” experts: general graph-reading experts, education graph-reading experts, and CBM graph-reading experts. Teachers and experts completed think-alouds on two standard CBM graphs. Think-alouds were coded for accuracy, completeness, and sequential coherence and the number of data-to-data comparisons (comparing data across instructional phases), data-to-goal comparisons (comparing data to the goal or goal line), and data-to-instruction links (linking data to instruction).
Results revealed that teachers’ think-alouds were accurate but less complete and coherent than those of the CBM graph-reading experts; however, the teachers’ think-alouds were nearly as complete and coherent as those of the education graph-reading experts, and they were more complete and coherent than those of the general graph-reading experts. Furthermore, although teachers made fewer data-to-data comparisons, data-to-goal comparisons, and data-to-instruction links than the CBM graph-reading experts, they made nearly as many as the general and education graph-reading experts. These results illustrated that difficulties with CBM graph comprehension were not unique to teachers. Of concern, however, were (a) the small number of data-to-data comparisons, data-to-goal comparisons, and data-to-instruction links made by the teachers and (b) the fact that only a few teachers made any data-to-instruction links at all. “Data-to” comparisons and links are important because they reflect the teacher’s ability to interpret the CBM data and link it to instruction; these skills are the essence of data-based instructional decision making. Van den Bosch et al. (2017) suggested that teachers might need specific, directed instruction on interpreting CBM data and linking it to instruction.
Study Purpose: Improving Teachers’ Comprehension of CBM Progress Graphs
The purpose of this study was to examine the effects of CBM instruction on teachers’ CBM graph comprehension. We compared three different CBM instructional approach groups to a control group in which teachers received no CBM instruction. The three CBM instructional approaches – basic, interpretation, and interpretation + linking – differed in the extent to which they emphasized and provided interactive instruction and practice on interpreting CBM data and linking the data to instruction. CBM instruction was delivered via videos.
We hypothesized that CBM instruction would lead to greater improvements in teachers’ CBM graph comprehension than practice alone (control condition). We further hypothesized that the additional interactive instruction and practice provided in the interpretation and interpretation + linking conditions would lead to greater improvements in interpreting CBM data and linking the data to instruction than basic CBM instruction alone.
As a secondary issue, we examined the social validity of CBM progress monitoring and the CBM instructional videos. Social validity refers to a “consumer’s” acceptability of an intervention and is important in terms of the eventual design and implementation of an intervention (Foster & Mash, 1999; Schwartz & Baer, 1991). Examining social validity was important because (a) teachers’ attitudes toward CBM might be related to the effects of CBM use on student progress (for a study on teachers’ attitudes toward data-based decision making, see Keuning, van Geel, & Visscher, 2017), and (b) teachers’ opinions about the CBM instructional videos could be used to inform future development and use of CBM instructional videos. We had no specific hypotheses related to social validity; however, we did want to know if differences in length and required responses between the instructional conditions would affect teachers’ acceptability of CBM progress monitoring and/or of the CBM video instruction.
In sum, the following research questions were addressed in the study:
Research Question 1: What are the effects of CBM instruction on teachers’ CBM graph comprehension?
Research Question 2: Do the effects of CBM instruction on teachers’ CBM graph comprehension differ by instructional approach?
Research Question 2a: Are improvements in interpreting CBM data greater for teachers in the interpretation and interpretation + linking conditions than for teachers in the basic condition?
Research Question 2b: Are improvements in linking CBM data to instruction greater for teachers in the interpretation + linking condition than for teachers in the basic and interpretation conditions?
Research Question 3: What is the social validity of the three CBM instructional approaches?
The study employed a randomized-control, pretest-posttest design, with CBM instructional approach as the independent variable. Graph comprehension and social validity were the dependent variables. Analyses were conducted separately for each dependent variable.
Method
Participants
Participants were 164 Dutch elementary school teachers (146 women, 18 men; Mage = 37.87 years, SD = 11.97; range, 21–67) from 66 different schools. Teachers were recruited via flyers that were distributed through the researchers’ networks of schools and teachers. To participate, teachers had to have taught in Grades 3 to 6 in the five years preceding the study.
In the Netherlands, general education teachers typically are responsible for the instruction of students with dyslexia or students who struggle with reading. General education teachers organize what would, in the context of RTI, be considered Tier 1 and Tier 2 instruction for these students. Sometimes the students receive extra instruction from a specialist, but often the specialist is someone outside of the school system. In a limited number of cases, if problems are complex, students are sent to a specialized school. In our sample, most teachers (n = 158) were general education teachers. A small number (n = 6) worked in specialized schools. All teachers held a bachelor’s degree in education, and four also held master’s degrees in education, psychology, or languages. Teachers had on average 13.44 years (SD = 10.55; range, 0–42) of teaching experience. There were no significant differences between teachers in the four conditions in years of teaching experience, F(3, 160) = 0.35, p = .793. Participating teachers were not using or planning to use CBM in their actual practice: their participation was simply for the purpose of the study. For their participation, teachers received a small thank you gift (a bag of Dutch licorice).
Teachers were randomly assigned to either a control condition or one of the three CBM instructional conditions: basic, interpretation, or interpretation + linking. The initial sample consisted of 184 teachers. Inspection of the demographic data revealed that 4 teachers had been substitute teachers only and had never taught their own class. These 4 teachers were removed from the sample. In addition, 16 teachers had had previous training in CBM via a university course. Data from these 16 teachers were dropped from the analyses (see following section for details). In the final sample, the number of participants per condition was as follows: control = 44, basic = 38, interpretation = 42, and interpretation + linking = 40. Demographic data for the four groups of teachers are provided in Table 1.
Demographic Data for Teachers in the Control, Basic, Interpretation, and Interpretation + Linking Conditions.
Note. Demographic data for the 16 teachers that were dropped from the sample were: gender, 13 females (81.3%), 3 males (18.8%); age, M = 24.75, SD = 5.86, range, 22–45; school type, 16 general education (100%); highest degree, 12 bachelor’s (75%), 4 master’s (25%); teaching experience, M = 2.66, SD = 4.28, range, 0–17.
Teachers’ progress-monitoring experience
To provide a thorough description of the sample and examine group comparability on relevant factors, we collected data on teachers’ progress-monitoring experience and general graph-reading skills.
With regard to progress-monitoring experience, teachers were asked 10 yes/no or open-ended questions about their experience using various progress-monitoring systems, including CBM. All elementary schools in the Netherlands are required to monitor students’ academic progress using a learner-monitoring system. Most schools use one of three commercial products to monitor the students. In each system, students are administered nationally normed standardized tests once or twice a year. Individual- and class-level results are presented to teachers in the form of graphs and tables that display progress across the elementary school years in comparison to national normative data.
All teachers reported that their schools implemented one of the three commercial products for progress monitoring. Teachers were asked if they used the graphs from the systems and if so, how. Most teachers (n = 156/164) reported using the graphs to examine student progress, communicate student progress to parents or colleagues, place students in instructional groups, or decide which students needed additional instruction.
With regard to experience with CBM, few teachers reported being familiar with CBM prior to the study, probably because CBM is not in widespread use in the Netherlands. Sixteen of the 180 teachers in the original sample reported that they had learned about CBM via university coursework and that as part of this coursework, they had interpreted data from CBM graphs and/or used CBM to monitor student progress. Despite random assignment, these 16 teachers were not evenly distributed across conditions (n = 2, 5, 4, and 5 in the control, basic, interpretation, and interpretation + linking conditions, respectively). Demographic data for these 16 teachers (see Table 1) revealed that the teachers were younger, more highly educated, and had less teaching experience than other teachers in the sample. In addition, the pretest scores for these teachers (see Table 2, in the “Results” section) were in general higher than for the rest of the sample, especially with respect to interpreting and linking the data to instruction. Because these 16 teachers were overrepresented in the intervention conditions, including their scores in the analysis would inflate the mean scores for the intervention groups. It was thus decided to drop them from the analyses. (To examine the effect of dropping these teachers, the analysis was run a second time with the 16 teachers included. The overall pattern of results remained the same.)
Means and Standard Deviations (in parentheses) from the Pre- and Posttest CBM Graph-Description Task.
Note. ACC = accuracy; COM = completeness (out of 8); DD = data-to-data comparisons; DG = data-to-goal comparisons; DI = data to-instruction links; SEQ = sequential coherence; SPEC = specificity. Mean pretest scores for the 16 teachers that were dropped from the sample were: ACC, 97.71 (SD = 4.55); COM, 7.63 (SD = 0.81); SEQ, 59.97 (SD = 14.95); SPEC, 95.78 (SD = 7.73); DD, 2.31 (SD = 1.25); DG, 2.63 (SD = 1.89); DI, 4.88 (SD = 2.50).
Teachers’ general graph-reading skills
To examine general graph-reading skills, teachers completed two pretest-only measures: a graph-reading ability scale (self-report) and a graph-reading test. The graph-reading ability scale was a translated version of a subscale of the Graph Familiarity Questionnaire developed by Xi (2005). The scale included 12 items asking participants to rate their graph-reading ability on a 6-point Likert scale (1 = low and 6 = high). Mean scale scores were nearly identical across the four groups (control M = 4.71, SD = 0.84; basic M = 4.66, SD = 0.75; interpretation, M = 4.71, SD = 0.61; interpretation + linking, M = 4.74, SD = 0.90). A Kruskall-Wallis test revealed no significant between-group differences on the scores, χ2(3) = 0.95, p = .81.
The graph-reading test included seven multiple-choice items requiring teachers to read and interpret line graphs. These items were based on items from a graph skills test developed by Shah and Freedman (2011). Teachers received one point per item answered correctly. Mean test scores were comparable across groups (control M = 3.64, SD = 1.06; basic M = 3.32, SD = 1.12; interpretation M = 3.74, SD = 1.11; interpretation + linking M = 3.63, SD = 1.03). A one-way ANOVA revealed no significant between-group differences on the scores, F(3, 160) = 1.12, p = .34.
Independent Variable: CBM Instructional Approaches
CBM instruction was delivered via instructional videos developed by the research team. Videos were used to ensure comparability in instruction across conditions. To create the videos, scripts for each of the three versions were written and audiotaped, and graphs, visualizations, and animations were created using Excel and PowToon. The spoken text, animations, and visualizations were combined using Adobe Premiere Pro. In the final step, interactive instruction and practice tasks were added to the interpretation and interpretation + linking instructional videos, making these videos 10 to 20 minutes longer than the basic instructional video. Given that the goal of our research was to determine whether the additional interactive instruction and practice were necessary for improved graph comprehension, we allowed the length of the videos to vary across conditions.
The information in each video was presented by a narrator and illustrated via the story of a teacher, Mr. Kees, and his student, Sander, who had reading difficulties. The content of the videos was based on content from a university course on CBM, a book written for practitioners on how to implement CBM (The ABCs of CBM; Hosp, Hosp, & Howell, 2007), and training materials retrieved via the National Center on Progress Monitoring and the Research Institute on Progress Monitoring. Each video began with an introduction that provided background on CBM. The introduction was followed by four segments on CBM implementation: collecting data, graphing data, interpreting data, and linking data to instruction. Differences between the three video conditions were seen in the segments on interpreting data and linking data to instruction. These differences are described in the following sections.
Basic condition
In the basic condition, the segments on interpreting data and linking data to instruction consisted of explaining to and modeling for the teachers how to interpret the data and link it to instruction. The CBM instruction in this condition was not interactive, and teachers did not practice the skills. In the segment on interpreting data, teachers were shown how an online progress-monitoring system could provide recommendations for changing instruction or raising the goal and were told that the recommendations were based on answers to three data interpretations questions:
Is the student making progress?
Will the student reach his or her goal?
Does the instruction need to be changed?
Teachers were shown six sample CBM graphs, each with one phase of data and a slope line drawn through the data. For each graph, they were shown the computer-generated instructional recommendation and given an explanation of how the recommendation was generated.
In the segment on linking data to instruction, teachers were provided a description of five categories in which instructional changes could be made (activity, time, setting, material, and motivation) and given an example of a potential change within each category (devoting more attention to a specific skill for activity, providing longer or more instruction for time, providing 1:1 instruction for setting, using materials from a different level for material, and using material that is of interest to the student for motivation). The basic instructional video lasted 25 minutes.
Interpretation condition
In the interpretation condition, teachers were given the same instruction as in the basic condition but also were given interactive instruction and practice on interpreting the data. The additional interactive instruction and practice were designed to improve teachers’ ability to interpret the CBM data, that is, to make more data-to-goal and data-to-data comparisons. In the segment on interpreting data, for each of the six sample graphs used in the basic condition, teachers answered the three data interpretation questions by selecting yes or no and then typing an explanation for their answers in a text box on the screen. Teachers were then shown the recommendation that would be generated by the online progress-monitoring system, with an accompanying explanation for the recommendation (the same as in the basic condition). No additional feedback was provided to the teachers. This approach of having teachers answer data interpretation questions before receiving computer-generated instructional recommendations was similar to that used by Fuchs et al. (1989a). The segment on linking data to instruction was the same as in the basic condition. The interpretation instructional video lasted approximately 35 minutes, depending on how long it took the teachers to answer the data interpretation questions.
Interpretation + linking condition
In the interpretation + linking condition, teachers were given the same instruction as in the interpretation condition but also were given interactive instruction and practice on linking the data to instruction. The additional interactive instruction and practice were designed to improve teachers’ ability to make more data-to-instruction links when describing the graphs. In the segment on linking data to instruction, following the description of the five categories in which instructional changes could be made, Mr. Kees reflected on five potential changes for Sander — one per category. The teachers were then asked to (a) select one of the five changes for Mr. Kees to implement and (b) provide a rationale for their selection, which they typed into a text box. This process was repeated again at the next instructional phase. No additional feedback was provided to the teachers. The interpretation + linking instructional video lasted approximately 45 minutes, depending on how long it took the teachers to answer the data interpretation questions and select instructional changes.
Control condition
The control condition was designed to determine the effects of practice only on CBM graph comprehension. Thus, in the control condition, rather than watch a CBM instructional video, teachers completed two filler tasks: watching a short video clip and completing an opinion survey about assessment in schools. They also completed the demographic questionnaire at this point (see “Procedures” section). Completing the filler tasks and the demographic questionnaire took approximately 20 minutes, depending on how long it took the teachers to fill out the survey and the demographic questionnaire. After the study’s completion, control teachers were shown the basic instructional video.
Primary Dependent Variable: CBM Graph Comprehension
The primary dependent variable in the study was teachers’ CBM graph comprehension. At both pre- and posttest, graph comprehension was assessed via a CBM Graph-Description Task. In previous research, the CBM Graph-Description Task has been shown to differentiate graph comprehension of (a) preservice teachers from that of CBM experts (Wagner et al., 2017), (b) inservice teachers from CBM experts (van den Bosch et al., 2017), and (c) teachers with higher ratings versus lower ratings on understanding and interpreting CBM graphed data (Espin et al., 2017). As an example, in the study by van den Bosch et al. (2017), scores on the CBM Graph-Description Task for inservice teachers versus CBM experts were, respectively, 97.5% versus 100% for accuracy, 5.7 versus 8.3 for completeness, 51.7% versus 85% for sequential coherence, 1.7 versus 4.8 for data-to-data comparisons, 1.7 versus 4.1 for data-to-goal comparisons, and 1 versus 5 for data-to-instructional links (see following section for a detailed description of these variables).
In the present study, based on the recommendation by van den Bosch et al. (2017), we used a different prompt for the CBM Graph-Description Task than that used in previous research. Rather than ask participants to tell all they were seeing and thinking about the graph, we asked them to describe the graph as if they were describing it to a parent.
To complete the CBM Graph-Description task, teachers examined a CBM graph for 1 minute and then described the graph as if they were describing it to the student’s parent. The graph remained in view as the teachers described it. No time limits were placed on the graph descriptions. Teachers described two graphs, one at pretest and one at posttest. The graphs were researcher-made CBM graphs that depicted progress of a fictitious student in reading across one school year (see Figure 1 for a sample graph). One graph displayed the progress of “Anna” and the other displayed the progress of “Bob.” Data points on the graph reflected the student’s scores on maze-selection tasks.
The two graphs were parallel in format and graph patterns. Thus, although the data points differed across the two graphs, the goal lines and slope lines were constructed so that they would lead to the same data-based decisions across the two graphs. For example, for both graphs, in Phase 3 the slope line was below but parallel to the goal line, indicating a need to change instruction. The patterns displayed on the graphs were CBM graph patterns found in previous research to be somewhat difficult to interpret (Espin, Saab, Pat-El, Boender, & van der Veen, 2018). We included these patterns to ensure variation in scores. One graph was administered at pretest and the other at posttest. The order was counterbalanced across teachers. Prior to completing the pretest, teachers were shown a sample CBM graph and were provided a general description of the graph (see van den Bosch et al., 2017). In addition, teachers were shown an example of a maze task and were told how the maze task was used within CBM.
CBM Graph-Description Coding
Graph-description coding procedures were adapted from those used in previous research (Espin et al., 2017; van den Bosch et al., 2017; Wagner et al., 2017). Teachers’ CBM graph descriptions were audiotaped and transcribed, and each transcription was checked by a second coder. Graph descriptions were then parsed into idea units (i.e., statements that expressed one idea). Parsing for 10 graph descriptions was done by the first two authors together. After that, the first author parsed the rest of the graph descriptions into idea units. Once parsed, each idea unit was assigned a content code corresponding to one of eight graph elements: framing (statements describing the set-up of the graph such as titles, axes, and the legend), baseline (statements describing baseline data of the student/peers or procedures to obtain those data), goal setting (statements describing the goal or procedures to set the goal), instructional phases 1, 2, 3, and 4 (statements describing the data within a particular phase), and goal achievement (statements describing whether the student had achieved the goal). Statements that described the data across the four phases rather than within a phase were coded as general progress statements, and statements that were reflections on or evaluations of graph content were coded as reflective statements.
Following the assignment of content codes, the graph descriptions were further coded in two separate rounds of coding. In the first round, the descriptions were coded for accuracy, completeness, sequential coherence, and specificity. Accuracy was the percentage of statements correctly reflecting the data presented on the graph. Completeness was the number of graph elements mentioned (of the eight elements listed earlier). Sequential coherence was the percentage of statements that followed a logical and coherent sequence. To calculate sequential coherence, teachers’ descriptions were compared to an “ideal” sequence, that is, a sequence in which the eight graph elements were described in an order that reflected CBM implementation and that progressed from framing to baseline to goal setting to instructional phases 1, 2, 3, and 4 to goal achievement (for a detailed description of coding sequential coherence, see Espin et al., 2017).
Specificity was the percentage of statements that referred to progress within a specific instructional phase (coded as a phase 1, 2, 3, or 4 statement) rather than to progress across phases (general progress statement). Specificity was calculated by dividing the number of statements made about specific instructional phases by the total number of progress statements made. (Note that the denominator we used for specificity was different than that used in previous studies [Espin et al., 2017; Wagner et al., 2017], where the denominator was the total number of statements included in the graph description.)
In a second round of coding, data-to-data comparisons, data-to-goal comparisons, data-to-instruction links, and raising-the-goal comments were coded. Data-to-data comparisons were the number of times teachers compared student performance or progress data in one instructional phase to either the baseline phase or another instructional phase (e.g., “His scores in phase 1 are higher than his baseline scores” or “He shows more progress in phase 2 than in phase 1”). Data-to-goal comparisons were the number of times teachers compared student performance or progress data to the goal line or the goal (e.g., “Her scores are all below the goal line” or “At this rate of growth she will achieve her goal”). Data-to-instruction links were the number of times teachers linked student performance or progress data to the student’s reading instruction and included comments that referred to the effectiveness of instruction or the need for instructional changes (e.g., “He is making progress in phase 3, thus the change in instruction was effective” or “His slope line is flat, so instruction should be changed”). Finally, raising-the-goal comments were the number of times teachers stated that the student’s goal should be raised when progress was greater than expected (e.g., “Her slope is steeper than the goal, so the goal should be raised”).
Graph descriptions were coded by the first author and 10 master’s degree students in education and child studies. The students were trained by the first author across a number of sessions, each focusing on the various aspects of the coding procedures. Sessions lasted from 10 to 60 minutes, depending on which aspect was being addressed in the session. At the end of each session, the master’s degree students coded sample descriptions, and agreement with the first author was calculated. Students had to reach 80% agreement with the first author before they could begin coding each aspect. In only a few cases did students not reach 80% agreement on their first attempt. In these cases, the disagreements were discussed, the definitions and coding procedures were reviewed, and the student coded an additional sample. Agreement was then calculated for the new sample. In all cases, 80% or higher agreement was reached on the second sample. All data were double coded by the first author and one master’s degree student. Coding disagreements were discussed and resolved. If agreement could not be reached, the second author was consulted.
Intercoder agreement
Intercoder agreement was calculated for every third graph description prior to discussions between coders. For the content codes (framing, baseline, goal setting, etc.) and for accuracy, agreement was calculated by dividing the number of agreements by the number of agreements plus disagreements and multiplying by 100. Agreement for the content codes was 89.94% (range, 40%–100%; agreement for content codes was below 70% in only 2% of the cases). Because the content codes were used to calculate completeness, sequential coherence, and specificity, separate agreement percentages were not calculated for these variables. Agreement for accuracy was 97.03% (range, 76.92%–100%).
For data-to-data comparisons, data-to-goal comparisons, data-to-instruction links, and raising-the-goal comments, coders identified the number of occurrences of each variable in the graph descriptions. Agreement was calculated per graph description by dividing the smaller number of occurrences by the larger number of occurrences and multiplying by 100. Average agreement was then computed across graph descriptions. Agreement was 81.63% (range, 0%–100%) for data-to-data comparisons, 87.46% (range, 0%–100%) for data-to-goal comparisons, 80.90% (range, 0%–100%) for data-to-instruction links, and 98.36% (range, 0%–100%) for raising-the-goal comments. The 0% agreements were due to the low occurrence of these variables for some teachers. If a teacher made only one data-to-data comparison and this comparison was missed by one of the coders, agreement would be 0% for that graph description. Agreements of 0% occurred in only 5% of the cases.
Secondary Dependent Variable: Social Validity
The social validity of each CBM instructional approach was assessed via a self-developed 10-item scale. Teachers completed the scale after viewing the instructional video. Teachers first rated five statements about CBM (I understand what CBM is; I think CBM graphs are easy to read; I think CBM graphs would be helpful for instructional decision making; I think I am sufficiently trained to use CBM in my class; and I would like to use CBM in my class for individual students with reading problems) and four statements about the instructional video (I thought the CBM video instruction was clear/interesting/useful/informative). These nine items were on a 4-point Likert scale, ranging from strongly disagree to strongly agree. The tenth item then asked teachers to rate the video on a scale of 1 to 10, with 10 being the highest rating. No other specific anchors were provided for the scale. Because teachers in the control group were shown the basic instructional video after study completion, they too completed the social validity scale.
Procedures
Data for the study were collected on an individual basis at a place convenient for the teacher (school, home, or university) in a session lasting from 1.5 to 2 hours. Teachers in the three CBM instructional conditions completed the tasks in the same order: graph-reading ability scale, graph-reading test, pretest CBM Graph-Description Task, CBM instructional video, social validity scale, posttest CBM Graph-Description Task, and demographic questionnaire. Teachers in the control condition completed the tasks in nearly the same order, with the exception that instead of watching the CBM instructional video, teachers completed the two filler tasks about assessment in schools and the demographic questionnaire. After the study completion, control teachers viewed the basic instructional video and completed the social validity scale.
Data collectors were the first author and the 10 master’s degree students. Prior to data collection, the students were trained by the first author in a single session in which they practiced all data collection procedures. With the exception of the pre- and posttest CBM Graph-Description Task, all data were collected via computer. The CBM Graph-Description Task was administered by the data collector, who gave instructions and acted as the “parent” who listened to the graph description. Data collectors were present during the entire session to ensure that the teachers completed all tasks and collect fidelity data. All teachers completed all tasks except for one teacher, who did not complete the social validity scale. Fidelity of implementation for the CBM instructional videos was 100%; that is, all participating teachers watched the correct video, and all teachers completed all interactive practice tasks during the video. Data collectors were observed by the first author on their first data collection session and again at (approximately) the 15th session. All data collectors adhered to all data collection procedures.
Results
CBM Graph Comprehension: Descriptives
The first research question addressed the effects of CBM instruction on teachers’ CBM graph comprehension. We addressed this question by using profile analysis to examine group differences in pre-post changes on the various aspects of CBM graph comprehension. Prior to conducting the profile analysis, we inspected pre- and posttest mean scores for the entire sample on the various aspects of CBM graph comprehension (see Table 2, Column 1).
Inspection of pretest means revealed that accuracy and completeness scores were high at pretest. Pretest accuracy was nearly 98%, and completeness was 7 out of a possible 8 (see Table 2, Column 1), allowing little room for improvement on these variables. With regard to the remaining aspects, approximately half of teachers’ description statements were in a logical, coherent order at pretest, with approximately 82% of the statements being specific to phases. Teachers made on average one to two data-to-data comparisons, data-to-goal comparisons, and data-to-instruction links each at pretest. Posttest scores revealed an increase in scores for sequential coherence, specificity, data-to-goal comparisons, and data-to-instruction links.
Comments about raising the goal occurred rarely; thus, rather than compute a mean score for these statements, we counted the number of teachers who mentioned raising the goal. At pretest, no teacher mentioned raising the goal. At posttest, 30 teachers (18%) mentioned raising the goal: 0 in the control, 6 (16%) in the basic, 11 (26%) in the interpretation, and 13 (33%) in the interpretation + linking groups.
Due to the ceiling effects for accuracy and completeness at pretest and the low occurrence rates for raising the goal statements, these three variables were not included in the subsequent profile analysis.
Effectiveness of CBM Instructional Approaches: Profile Analysis
Profile analysis was used to examine the pretest-posttest changes across conditions. Profile analysis is a particular way of conducting multivariate analysis of variance (MANOVA) and often is described as a multivariate approach to repeated measures analyses (Tabachnick & Fidell, 2012). With profile analysis, the profiles of groups of participants who have been measured on several dependent variables at the same time are compared. The results of the analysis are presented visually in the form of profile plots that represent the profiles of different groups of participants for a set of dependent variables.
When comparing the profiles of different groups of participants, three questions are addressed:
Are the profiles of the different groups parallel?
Does one group, on average, have higher scores for the dependent variables than another group?
Are the scores for all of the dependent variables on average the same?
These three questions are referred to as parallelism, difference in levels, and flatness, and they are answered by examining the generalized linear model (GLM) results of the within-participants interaction effect, between-participants effect, and within-participants effect, respectively. If the profiles are not parallel, the question of flatness is irrelevant because nonparallel profiles are per definition not flat.
The goal of our profile analysis was to compare the CBM graph-comprehension profiles for teachers in the four conditions: control, basic, interpretation, and interpretation + linking. The aspects of CBM comprehension included in the analysis were sequential coherence, specificity, data-to-data comparisons, data-to-goal comparisons, and data-to-instruction links.
The first step in a profile analysis is to place all variables on the same scale to allow for group comparisons across variables (Tabachnick & Fidell, 2012). To place the variables on the same scale, we subtracted the pretest scores from the posttest scores for each variable and then converted the change score to a standardized score (z-score). The second step in profile analysis is to determine whether the assumptions of profile analysis are met. The groups had sufficiently equal sample sizes; the smallest group (n = 38) included (far) more cases than the number of dependent variables (5), justifying multivariate analysis, and there were no missing data. The assumption of multivariate normality was met for all dependent variables except specificity. The distribution of specificity was bimodal: For most teachers, pre- and posttest specificity scores were relatively similar, but for a few teachers, the difference between pre- and posttest scores was large, for example, 0% at pretest and 100% at posttest. Given that the sample size was sufficiently large (n > 30), specificity was kept in the analysis.
Inspection of boxplots revealed 13 outliers for data-to-goal comparisons and 12 outliers for specificity. We had no valid reason to drop the outliers from the sample, and for all cases, Cook’s distances were smaller than 1, indicating that the outliers had no substantial impact in determining the outcome of the analysis. Thus, the outliers were kept in the data set. The assumptions for linearity, homogeneity of regression, and multicollinearity and singularity were met. The assumption of homogeneity of variance-covariance matrices was not met; therefore, Games-Howell corrections were used to interpret the results.
After Steps 1 and 2 were completed, a GLM repeated-measures analysis with teachers’ standardized pre-post change scores for CBM graph comprehension was conducted. Results of the within-participants effect test for parallelism (i.e., the interaction between CBM graph comprehension and condition) revealed that the profiles of teachers in the four groups were significantly nonparallel, indicating that the profile of the dependent variables differed for at least one group (see Figure 2; F(12, 640) = 6.29, p < .001,
Profiles of the standardized curriculum-based measurement (CBM) graph-comprehension pre-post change scores from the four groups of teachers (control, basic, interpretation, and interpretation + linking). SEQ = sequential coherence; SPEC = specificity; DD = data-to-data comparisons; DG = data-to-goal comparisons; DI = data to-instruction links.
Results of the between-participants effect to test for the differences in levels revealed a main between-participants effect for condition, F(3, 160) = 6.85, p < .001,
The deviation from parallelism was evaluated by examining the profiles per group through simple-effects analyses; that is, we examined group differences for each dependent variable. The simple effects analyses revealed that there were no differences between groups in standardized pre-post change scores for sequential coherence, F(3, 160) = 1.07, p = .37; specificity, F(3, 160) = 0.49, p = .69; or data-to-data comparisons, F(3, 160) = 1.17, p = .32.
However, differences between groups were found for data-to-goal comparisons, F(3, 160) = 6.14, p = .001,
In sum, the results of the profile analysis revealed that the profiles of the CBM graph-comprehension standardized pre-post change scores were not parallel across the four groups. Overall, teachers in the three CBM instructional groups had significantly larger standardized pre-post change scores for CBM graph comprehension than teachers in the control group. Further, the mean standardized pre-post change score for teachers in each of the three instructional groups across the total set of dependent variables was positive, whereas for teachers in the control group, it was negative. More specifically, teachers who received CBM instruction improved more on data-to-goal comparisons and data-to-instruction links than teachers who received no CBM instruction. Moreover, for data-to-goal comparisons, significant differences were found between the instructional groups, with the teachers in the interpretation and the interpretation + linking groups improving more than teachers in the basic group.
Social Validity
Teachers’ ratings of CBM progress monitoring were positive, with an average rating across the four CBM-related items of 3.54 (SD = 0.38) out of 4. Mean ratings were similar across the four groups (basic M = 3.48, SD = 0.38; interpretation M = 3.47, SD = 0.40; interpretation + linking M = 3.58, SD = 0.35; control M = 3.61, SD = 0.37).
Teachers’ overall ratings of the CBM instructional videos (Item 10) were fairly positive, with an average rating across teachers of 7.96 (SD = 0.96) out of 10. Mean ratings were similar across the four groups (basic M = 7.82, SD = 1.18; interpretation M = 7.79, SD = 0.90; interpretation + linking M = 8.05, SD = 0.79; control M = 8.18, SD = 0.76). Teachers’ ratings of the specific characteristics of the videos (clear, interesting, useful, informative) also were positive, with an average rating across the four items of 3.52 (SD = 0.46) out of 4. Mean ratings were similar across the four groups (basic M = 3.43, SD = 0.48; interpretation M = 3.57, SD = 0.45; interpretation + linking M = 3.53, SD = 0.45; control M = 3.55, SD = 0.47).
Discussion
CBM progress graphs are meant to guide instructional decision making for students with learning difficulties; yet, if teachers do not respond to the data with instructional changes, the data are useless. Unfortunately, research has shown that teachers often do not respond to the data (Stecker et al., 2005), perhaps in part because they have difficulty comprehending the CBM graphs, especially with regard to interpreting the data and linking it to instruction (Espin et al., 2017; van den Bosch et al., 2017; Wagner et al., 2017).
The purpose of this study was to examine the effects of CBM instruction on teachers’ CBM graph comprehension, most specifically on their ability to interpret CBM data and link it to instruction. We compared three different CBM instructional approach groups to a control group. We also examined the social validity of each instructional approach.
Effects of CBM Instruction on CBM Graph Comprehension
On the whole, teachers became more coherent and specific in their CBM graph descriptions, and their descriptions included more data-to-goal comparisons and data-to instruction links at posttest than at pretest. Improvements were significantly greater for teachers in the three CBM instructional groups than for teachers in the control group, and these differences were due primarily to the improvements in data-to-goal comparisons and data-to-instruction links. That is, teachers in the three instructional groups were more likely to compare student performance or progress to the goal or goal line and describe the link between the student’s performance or progress and instruction than teachers in the control group.
The improvements seen in data-to-goal comparisons and data-to-instruction links are encouraging because it is these aspects of CBM graph comprehension that are the essence of CBM data-based decision making — and the aspects that are the most challenging for teachers (van den Bosch et al., 2017). To make CBM data-based decisions, teachers must compare student performance or progress to the goal to determine whether the student is progressing at the expected/desired rate (data-to-goal comparisons) and then link the information to instruction to decide whether there is a need to change instruction (data-to-instruction links). It is this latter aspect of data-based decision making that leads to improved instruction for students with learning difficulties.
It was somewhat disappointing that the CBM video instruction did not lead to improvements in other aspects of CBM graph comprehension. With regard to accuracy and completeness, the lack of improvement probably was related to a ceiling effect in scores. Pretest scores were high for all groups, including the control group, on these variables. Previous research has shown that teachers tend to describe CBM graphs accurately, with mean accuracy scores ranging from 85% to 97% (Espin et al., 2017; van den Bosch et al., 2017; Wagner et al., 2017). Completeness scores in the current study were higher than those found in previous research. Wagner et al. (2017) reported that preservice teachers mentioned only three of nine possible graph elements, and van den Bosch et al. (2017) reported that inservice teachers mentioned six of nine graph elements. The higher completeness scores in the present study (seven of eight graph elements) may be due to the fact that teachers were asked to describe the graphs as if they were describing them to a parent rather than to tell all they were seeing and thinking about a graph, as was done in the previous studies.
Although there was a ceiling effect for accuracy and completeness, this was not the case for the number of data-to-data comparisons. Here there was room for improvement, but teachers’ scores barely changed from pre- to posttest. Note that the average number of data-to-data comparisons made by the teachers (1.35 at pretest and 1.41 at posttest) was similar to that of the teachers in the van den Bosch et al. (2017) study (1.67). Perhaps comparing student data across instructional phases is a difficult skill to master, and/or perhaps it is a skill with which teachers are unfamiliar given that comparing data across instructional phases is fairly unique to CBM graphs. Regardless of the reason, it would seem that if teachers are to improve in making data-to-data comparisons, they need either more or different instruction than what was provided in the CBM instructional videos.
Pretest-posttest improvements were seen in sequential coherence and specificity, but these improvements were similar across the control and intervention groups, suggesting that practice alone was enough to improve in these areas. With regard to sequential coherence, there was room for greater improvements. The posttest sequential coherence score for teachers was only 65%. This score was greater than the score of 52% for teachers in the van den Bosch et al. (2017) study, where teachers also were new to CBM; however, it was not as high as the score of 71% for teachers from the Espin et al. (2017) study, where teachers had an average of 12 years of experience using CBM.
It is disappointing that there were not greater improvements in sequential coherence for the instructional groups. The ability to describe graphs in a logical and coherent manner is important for communicating with parents and sharing information in team meetings. Teachers may need more than a relatively short (25–45 minutes) instructional video — and experience using CBM to monitor student progress — to learn how to effectively communicate information from the CBM graphs to others.
Differences in Effects Across Instructional Approaches
Although it was encouraging that teachers in the instructional groups improved more than teachers in the control group on many aspects of CBM graph comprehension, we were most interested in whether the additional interactive instruction and practice provided in the interpretation and interpretation + linking conditions would lead to greater improvements in interpreting CBM data and linking the data to instruction than basic CBM instruction alone. Results of the profile analysis revealed that teachers in both the interpretation and interpretation + linking conditions improved more in data interpretation — more specifically, in making data-to-goal comparisons — than teachers in the basic condition. Thus, it would seem that the additional interactive instruction and practice provided for interpreting CBM data paid off. This was not the case, however, for linking the data to instruction. Teachers in the interpretation + linking condition did not improve more in linking the data to instruction than teachers in either the basic or interpretation only conditions. It is important to point out that in all three instructional groups, teachers made impressive gains in linking the data to instruction, with increases of three to four links in each condition. The number of posttest data-to-instruction links made by the teachers in the three instructional groups (n = 4.7–5.1) is especially impressive when one considers the fact that CBM experts in the van den Bosch et al. (2017) study made a similar number of data-to-instruction links (n = 5).
Raising-the-Goal Statements
Although the number of raising-the-goal statements was not included in the profile analysis, it is important to reflect briefly on the percentage of teachers who made such statements. At pretest, no teacher mentioned raising the goal. At posttest, teachers in all three instructional groups — but not the control group — mentioned raising the goal, with the largest percentage found for the interpretation and interpretation + linking groups. Fuchs et al. (1989b) found that setting ambitious goals was related to greater improvements in student achievement. Our results suggest that CBM video instruction, perhaps in particular additional instruction that is interactive and provides opportunities for practice (as was the case for the interpretation and interpretation + linking conditions), served to raise teachers’ awareness of the importance of raising the goal. Nonetheless, even in the interpretation and interpretation + linking conditions, only 26% and 33% of the teachers mentioned raising the goal at posttest. It is important in future research to examine teachers’ understanding of the importance of raising the goal.
Social Validity
Teachers’ positive evaluations of the CBM instructional videos and of CBM itself supported the social validity of all three CBM instructional approaches. Although it could be the case that the teachers’ evaluations were influenced by the fact that the data collectors were present when they filled out the social validity scale, the data collectors did not view the teachers’ responses as they filled in the scale.
Given the fact that educators’ positive attitudes toward data-based decision making are related to the effects of data-based decision making on student progress (Keuning et al., 2017), it is encouraging that participating teachers, who were not familiar with CBM prior to the study, developed a positive attitude about CBM via the instructional videos and were positive about the video instruction itself. The similarities in mean ratings across the groups suggest that the additional interactive instruction and practice in the interpretation and interpretation + linking conditions did not affect teachers’ attitudes toward CBM or toward the instructional videos.
These results fit well with results from Kennedy et al. (2016), who found that preservice teachers who received multimedia CBM instruction were more motivated than those who received the same instruction in an article format. Such results provide tentative support for the use of technology to provide CBM instruction to teachers. CBM video instruction can easily be incorporated into an online progress-monitoring system or offered as a standalone course in the context of e-learning. However, we must caution that we did not compare the effects of video and in-person training. Before fully recommending CBM video training, it will be important to compare the effects of video and in-person training on teachers’ CBM graph comprehension and eventual CBM implementation.
Study Limitations and Related Directions for Future Research
The major limitation to this study is that it did not examine the effects of CBM video instruction on teachers’ actual CBM implementation and the resulting student performance. These effects must be examined in future research to determine to what extent CBM video instruction is effective and to evaluate differential effects of instructional conditions.
In addition to this major limitation, there are four additional limitations to the study. The first concerns the “counterfactuals” (see Lemons, Fuchs, Gilbert, & Fuchs, 2014) used in the study design, that is, the control and basic comparison conditions. The control condition was designed to examine the extent to which practice alone would result in improved graph comprehension scores; thus, control teachers did not receive any CBM instruction. In future research, a “stronger” counterfactual might be to have teachers complete a CBM-related task, for example, reading an article that describes CBM (as was done in Kennedy et al., 2016).
The basic condition also served as a counterfactual. It was designed to reflect business as usual, that is, the type of CBM instruction that teachers might typically receive. In this condition, teachers were told about the importance of interpreting the data and linking it to instruction but were not provided with additional interactive instruction and opportunities for practice. What is not known is the extent to which the basic condition represented typical CBM instruction more broadly. For example, our basic instruction might have included more information on interpretation and linking of data to instruction than is typically provided in CBM instruction. It would be valuable to conduct a review of CBM instructional materials to determine to what extent typical CBM instruction focuses on data interpretation and linking data to instruction.
Second, the sample was somewhat limited. Although the sample was fairly large and diverse (164 teachers from 66 different schools in different regions in the Netherlands, including both rural and urban regions), it did not include teachers from the whole of the Netherlands, and it included teachers only from the Netherlands. Teachers’ data may have been negatively affected by the fact that Dutch teachers are unfamiliar with CBM, or positively affected by the fact that Dutch teachers are familiar with monitoring student progress via the learner monitoring systems implemented in the schools. In short, replication of the study with other samples of teachers, both within the Netherlands and from other countries, is in order.
Third, the three CBM instructional video conditions differed not only in content but also in length. It is not possible to know whether the difference found between the three conditions on data-to-goal comparisons was due merely to the teachers’ increased interaction with the content or the additional practice activities included in the interpretation and interpretation + linking conditions.
Fourth, we did not examine maintenance effects. In future research, it would be interesting to ask teachers to describe a CBM graph a month after watching the CBM instructional video to determine to what extent teachers maintain the information presented in the instructional video.
Conclusion and Implications
In conclusion, our results suggest that CBM video instruction can be used to improve teachers’ CBM graph comprehension, including their ability to make data-to-goal comparisons and link data to instruction. Given that both the interpretation and interpretation + linking approaches resulted in greater improvements in data-to-goal comparisons than the basic approach, we can recommend both approaches at this time. The interpretation approach is of shorter duration than the interpretation + linking approach; however, given the importance of linking the data to instruction, we do not yet want to eliminate the interpretation + linking approach.
Our recommendations are tentative because we did not examine the effects of the different instructional approaches on teachers’ actual CBM implementation and the resulting student performance. In addition, although teachers improved on several aspects of CBM graph comprehension after viewing the instructional videos, they did not improve in making data-to-data comparisons or improve as much as they could have on sequential coherence.
Before drawing firm conclusions about the effectiveness of CBM video instruction and the relative effectiveness of the various approaches to CBM instruction, it is essential that the effects of the CBM instructional approaches on teacher implementation and student performance be examined. Research on human decision making (e.g., see Gigerenzer, 2007; Kahneman & Klein, 2009; Simon, 1990) reveals that humans have difficulty relying on data to make decisions and that the use of data for decision making improves only via training and experience (for a description of this literature as it relates to CBM data-based decision making, see Espin et al., 2017). Perhaps a 25- to 45-minute instructional video is not enough for teachers to become skillful CBM data-based decision makers. The instructional video may provide a good beginning, but teachers may need continued, ongoing instruction and guidance to become proficient data-based decision makers.
Footnotes
Acknowledgements
We are grateful to the teachers who participated in present study and to the bachelor’s degree and master’s degree students who helped with developing the CBM instructional videos and collecting and coding the data for this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.