Protocol for a Systematic Review: Self-Management Interventions for Reducing Challenging Behaviors among School-age Students: A Systematic Review

The Problem, Condition or Issue

Most parents and teachers agree that students need to exhibit appropriate social behaviors in order to achieve academic goals; however, approximately 20% of students, or 3-4 students in the average classroom, repeatedly display challenging behaviors that interfere with normal academic and social development (Brauner & Stephens, 2006; Bushaw & Lopez, 2010; Satcher, 2004; Walker, Ramsey, & Gresham, 2004). Challenging student behaviors are defined as acts that (a) interfere with social and academic functioning; (b) harm a child, his or her peers, or adults; and (c) place a child at risk for developmental problems. Because research often distinguishes between subtypes of challenging behaviors, we specify three broad subtypes: (a) direct and indirect forms of aggression (e.g., hitting, name calling, spreading rumours; Dodge & Coie, 1987; Leff & Crick, 2010; Parke & Slaby, 1983); (b) overt and covert antisocial behaviors (e.g., stealing, bullying, lying, cheating); and (c) low intensity acts of insubordination (e.g., noncompliance, withdrawal, refusal to cooperate, impulsivity, inattention, off-task; Kaiser & Rasminsky, 2009).

Challenging student behaviors are harmful to everyone in schools—including students who exhibit the behaviors and their peers and teachers. Students who exhibit challenging behaviors are frequently removed from class, which interrupts instruction, exacerbates academic difficulties, and increases the likelihood of school failure and dropout (Gresham, Lane, & Lambros, 2000; Nelson, Benner, Lane, & Smith, 2004). Peers of disruptive students are adversely affected by the behaviors due to lost instructional opportunities (U.S. Department of Education [USDOE], 2006). Observational studies indicate challenging behaviors contribute to a loss of four hours of instruction per week in the average classroom or about 144 hours per student over the academic year (Walker et al., 2004). Lastly, teachers experience increased stress and burnout associated with managing challenging behaviors (Brouwers & Tomic, 2000; Clunies- Ross, Little, & Kienhuis, 2008; Grayson & Alvarez, 2008; Hastings & Bham, 2003; Joseph & Strain, 2003). A survey of highly-qualified teachers suggested that 53% of those who requested transfers and 44% of those who quit teaching cited challenging student behaviors as their primary reason for the decision (USDOE, 2005). Because challenging behaviors adversely impact everyone in schools, it is vital that school professionals assist students with challenging behaviors to learn adaptive social, emotional, and behavioral skills.

Research suggests school-based programs that promote competencies in social, emotional, and behavioral skills hinge on the development of five interrelated concepts: social-awareness, self-awareness, self-management, relationship, and problem solving (Bridgeland, Bruce, & Hariharan, 2013). Also called social, emotional, noncognitive, or soft skills, exposure to these skills increases the likelihood that students with challenging behaviors will experience better proximal school-related and distal life-course outcomes (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Heckman & Kautz, 2012; Wilson & Lipsey, 2007). For example, programs and practices promoting the development of the aforesaid skills are related to improvements in social functioning (ES = 0.69), attitudes toward school (ES = 0.24), behavioral problems (ES = 0.26), emotional stability (ES = 0.28), and academic performance (ES = 0.28; Durlak et al., 2011).

To facilitate the development of social, emotional, and behavioral skills, researchers and educators increasingly recognize the importance of autonomy support as an intervention mechanism (Algozzine, Browder, Karvonen, Test, & Wood, 2001; Field, Martin, Miller, Ward, & Wehmeyer, 1998; Lane, Menzies, Bruhn, & Crnobori, 2010). Autonomy refers to a sense of self-management (Deci & Cascio, 1972; Deci, Cascio, & Krusell, 1975; Deci & Ryan, 2011; Wigfield, Eccles, Roeser, & Schiefele, 2008; Wigfield, Eccles, Schiefele, Roeser, & Davis- Kean, 2007). Authority figures who endorse and enable the development of skills and opportunities required for self-management engage in autonomy support strategies (Deci & Ryan, 2011; Field et al., 1998). Strategies that integrate principles of autonomy support include—but are not be limited to—instruction in decision making, problem solving, goal setting, self-awareness, self-assessment, self-evaluation, self-management, and self-monitoring (Algozzine et al., 2001; Lane et al., 2010; Wehmeyer & Schwartz, 1997). Autonomy support strategies also facilitate improved student-teacher relations (Wentzel, Filisetti, & Looney, 2007). Improved student-teacher relations diminishes challenging behaviors and makes disciplining students who display those behaviors more effective (Hamre & Pianta, 2003).

In summary, promoting the development of competencies in social, emotional, and behavioral skills requires on-going and quality instruction in self-awareness, social awareness, self-management, relationship, and problem solving skills. Educators can nurture and cultivate the development of these valuable skills through autonomy support strategies that propagate an increased sense of self-management in students. Though many strategies impart social emotional skills and promote student autonomy separately, a self-management intervention combines social and emotional skills with autonomy support in a single approach.

Self-Management Interventions

The review will focus on the effectiveness of school-based self-management (SM) interventions—a widely-used cognitive behavioral intervention to address disruptive and challenging behaviors in school settings. Though SM interventions are referred by many names (e.g., self-control, effortful control, self-regulation), SM is defined as a set of strategies that students are trained in to assess, monitor, and evaluate their own behavioral performance (Briesch & Chafouleas, 2009; Cole, Marder, & McCann, 2000; Fantuzzo, Polite, Cook, & Quinn, 1988; Maggin, Briesch, & Chafouleas, 2013; Rothbart & Rueda, 2005; Shapiro & Cole, 1994; Shapiro, Durnan, Post, & Skibitsky-Levinson, 2002). More specifically, Fantuzzo and colleagues (1988) suggest a SM intervention includes one or a combination the following elements: 1.

self-selecting a target behavior

One challenge faced in the proposed review is that researchers and practitioners rarely implement the elements of SM in a consistent manner. The SM literature has been summarized by five prior reviews. The first review applied the above classification to all SM studies conducted between 1967 and 1988 and noted that studies included 9.6 out of the 11 elements listed above (88%)—with 60% of studies including all 11 elements (Fantuzzo et al., 1988). In a second follow-up review of all SM studies published between 1989 and 2008, Briesch and Chafouleas (2009) applied the same typology used by Fantuzzo and found that 7.6 out of the 11 elements listed above (60%)—with only 3% of the studies including all 11 elements. Upon closer examination, Briesch and Chafouleas noted 4 out of the 11 SM elements “consistently” appeared across the 30 SM studies (i.e., self-selecting a target behavior, self-defining the target behavior, self-observing the target behavior, and self-recording the observations) and a fifth step (i.e., self-evaluation) was noted in half of the studies reviewed (Briesch & Chafouleas, 2009, p. 111).

A third review examined the impact of SM on academic outcomes and proposed that a SM intervention consisted of five elements: i.e., goal setting, self-monitoring, self-evaluation, self-instruction, and strategy instruction (Mooney, Ryan, Uhing, Reid, & Epstein, 2005). A fourth review suggested SM was comprised of four processes: self-monitoring, self-evaluation, self-reinforcement, and self-instruction (Stage & Quiroz, 1997). The fifth and final review examined the impact of SM on behavioral outcomes by applying the What Works Clearinghouse (WWC) inclusion criteria to the studies collected in the 2009 Briesch & Chafouleas review (Maggin et al., 2013). The fifth review defined SM as including five steps: self-determining a performance goal, self-monitoring, self-recording, self-charting, self-evaluation, and self-reinforcement. To summarize, though researchers define and combine the elements of SM differently, we may observe studies in this review that include all 11 SM elements—though the likelihood of this occurring is low.

Adding to the variety of definitions and typologies that identify a SM intervention, very little research examines the impact of training students in SM. Studies do suggest that the presence of student training has an impact on the success of a SM intervention. For example, existing studies suggest that when students are provided with training in SM they evidence

improved academic (Greiner & Karoly, 1976; Harris, 1986) and behavioral performance outcomes (Harris, Friedlander, Saddler, Frizzelle, & Graham, 2005; Ninness, Fuerst, Rutherford, & Glenn, 1991). However, no studies directly compare training vs. no training or unpack the most effective SM training strategies that appear to enhance outcomes. Regardless, it is a widely-accepted best practice to train participants in the steps of a cognitive-behavioral intervention before they engage in the actual procedures of the intervention (Kazdin, 2003). As such, a novel opportunity for the current review is to capture characteristics of student training in SM to examine the impact of training on outcomes. To examine the impact of training students in SM, a summary of what is currently known about training students in the SM is offered before we describe SM procedures.

The SM student training

Considerations for training students in SM are provided by Cole et al. (2000), Shapiro et al. (2002), Strayhorn (2002b), and Lane et al. (2010)—however very little empirical research identifies which of these considerations are more important than others.

Some research suggests training is more effective when it (a) is sequenced, (b) uses active learning modalities such as modelling and rehearsal, (c) is focused on individual needs, and (d) explicitly defines the behavioral skills required to engage in SM (Durlak et al., 2011). Training is also enhanced when students have opportunities to practice the procedures of SM (Schunk & Zimmerman, 1998; Strayhorn, 2002a, 2002b), and when those practice sessions result in specific and formative feedback aimed at improving accuracy (Shute, 2008). Some research also indicates that feedback should also be supportive when affirming or correcting student SM efforts (Bandura, 1994; Dweck, 1975; Mueller & Dweck, 1998; Zimmerman, 1989). Praise for effort communicates that failure is a part of learning and effort matters more than achievement, effectively lowering the effect of appraisal on performance of a required task (Dweck, 2006). Such messages also engender the development of quality relations between students and teachers (Wentzel, 1991) and encourage children to practice and integrate SM skills (Lewis, 2000).

There are three possibilities when determining the focus of training for a SM intervention. The first possibility is to determine whether the student can perform a task. If the student cannot perform the task or achieve the desired outcome, then he or she will require direct instruction in the basic steps to perform that task. If the student has the ability to perform a task but requires assistance with doing the task fluently, smoothly, or with more confidence, a second possibility can be considered. In this instance it must be determined whether the student requires additional instructional supports, more practice, or both. Lastly, if the student has the capacity to perform the task fluently but refuses or is not motivated, then the student may need support or reinforcement to engage in the task. Either way, studies suggest that students may be trained in SM and that training can mitigate any of the three performance barriers listed here (Cole et al., 2000; Lane et al., 2010; Shapiro et al., 2002).

Given the number of cognitive and behavioral skills required to engage in SM, a number of

considerations raised by Cole and colleagues (2000) serve as a helpful guide to illustrate how student training in SM will likely vary across studies: 1.

What type of students will SM be used with?

The SM procedures

The SM procedures consist of students engaging in one or a combination of the 11 processes listed above that constitute a SM intervention. Although procedural aspects would certainly be taught during the training stage, the SM procedural stage directly refers to the cognitive and behavioral processes a student would be expected to engage in during the actual implementation of a SM intervention. The procedural stage may include any one or a combination of the 11 SM elements listed above (Hallahan & Sapona, 1983; Rutherford Jr, Quinn, & Mathur, 2004; Vaughn, Bos, & Schumm, 2011).

During the self-assessment phase, students may self-select, self-define, and self-determine reasonable performance goals to address a target behavior. Even if students are only self-monitoring on- or off-task behavior, they must first select and define a behavior. Ideally, the behavior will be one that occurs at a frequency or rate that allows it to be observed or detected at regular intervals. That is, if a behavior is low frequency then it is unlikely to register or be observed to a degree that self-monitoring will capture useful information about the behavior. Once a student has identified and defined a problem behavior, a goal may also be set to reduce the problem or increase the performance of a preferred replacement behavior. Using observable and measureable terms (i.e., frequency, duration, and/or severity of the behavior), a student may operationally define a goal using positive language (i.e., I will increase my work completion) or negative language (i.e., I will not tap my pencil on my desk). Though some researchers argue that to be considered a true SM intervention, students should directly participate in each of the 11 SM steps, many studies report a process whereby educators select, define, and set performance goals for students (Briesch & Chafouleas, 2009). Students or teachers may also identify reinforcements at this stage should the student achieve the goal. Once the student has selected, defined, and set a goal to address a behavior, the student is ready to self-monitor his or her performance.

During the self-monitoring phase, students first self-prompt or are externally prompted to self-observe. During the self-observation, a student reflects upon his or her performance and discriminates whether he or she displayed the target behavior during the interval. During the self-recording process, a student would physically record the observation on a schedule (see Tables 1 and 2) to indicate the presence or absence of the target behavior. Following the self-monitoring phase, a student then may evaluate his or her own progress.

During the self-evaluation of performance, students self-chart their performance by calculating percentages or creating graphic images of the data collected (DiGangi et al., 1991). Using the percentages, graphs, or charts, the student can self-appraise or compare his or her results to a predefined goal, prior observational data, using teacher or other third party observations, or combination of those standards (Thompson & Webber, 2010). Using the standards, students can determine whether their performance met the standard and—if relevant—self-administer selected reinforcements. The performance comparisons may also be used to develop new performance goals before the process is then iteratively repeated.

In summary, the review will examine SM interventions, a widely-used cognitive behavioral intervention that appears to be effective for academic and behavioral outcomes. Though implementing a SM intervention appears straightforward, there are many variations in practice surrounding training and implementation of a SM intervention. Regardless of these variations, we suggest a SM intervention is best defined as a set of strategies that train students to assess, monitor, and evaluate their own behavioral performance. As such, SM consists of two stages: a training stage and a procedural stage. Though some resources are available to suggest best practices and considerations for training students to engage in SM procedures, no manualized SM programs are available that sequence empirically supported elements of student training, which may improve outcomes. As such, it is expected that the type, quality, and degree of student training will vary greatly across studies of SM interventions. In addition, the SM procedural stage will also vary based upon many contextual and child-specific features.

How the Intervention Might Work

There are several important behavioral principals or mechanisms of change underlying a SM intervention. To begin, behavior change can occur by the very function of engaging in the self-monitoring aspect of a SM intervention (Nelson & Hayes, 1981). Also known as the reactivity principle, the simple act of collecting SM data regarding one's own behavioral functioning is thought to alter the behavior itself. For example, and observed in studies of self-regulated learning (Bandura, 2005; Cleary & Zimmerman, 2004), students who self-monitored their performance on a number of math problems were systematically introduced to a heightened awareness of the number of problems answered correctly. Change was hypothesized to occur as a direct result of internal reward mechanisms that influence behavior change. That is, the simple act of observing and recording one's own performance informs and influences reward centers, which alters motivation and behavior (Shapiro et al., 2002). The reactivity phenomenon has also been observed in a variety of other research areas. For example, weight loss was observed in studies where participants monitored daily caloric intake and types of foods eaten without engaging in dieting interventions (Boutelle & Kirschenbaum, 2012; Butryn, Phelan, Hill, & Wing, 2012). The reactivity phenomena has also been observed in studies of SM interventions with persons who have substance and alcohol abuse disorders (Bien, Miller, & Tonigan, 2006; Simpson, Kivlahan, Bush, & McFall, 2005). Although it may appear, on the surface, that no discernible extrinsic reinforcers are present during the SM procedure—the very act of reflecting on behavior, collecting data on behavior, and using that data to evaluate performance over time is a metacognitive activity that alters the targeted behaviors.

Another mechanism related to the theory of change underlying SM—one closely related to the concept of reactivity, self-management, self-awareness, and intrinsic motivation—is the concept of perceived autonomy (Deci, Cascio, & Krusell, 1975; Deci & Ryan, 2011; Wigfield, Eccles, Schiefele, Roeser, & Davis-Kean, 2007). In a SM strategy, students engage in an act of “perceived autonomy.” That is, students are encouraged to self-assess, self-monitor behaviors, and self-evaluate specific behaviors (Algozzine et al., 2001; Field et al., 1998; Wehmeyer & Schwartz, 1997). Because behavior change is often a “top down” activity that is prepared, planned, and applied to students by teachers, supporting student autonomy through the use of a SM intervention improves perceived ownership and motivation to engage in the intervention, which leads to an increased likelihood of positive outcomes (Lane et al., 2010). Indeed, a variety of studies have shown that when teachers engage in autonomy support strategies (e.g., choice making, goal setting, self-observation, instruction in self-control), participants perform tasks consistently better than tasks where autonomy is not supported (DeCharms, 1984). In short, SM is an autonomy support activity that provides students with choice and, as such, they experience increased levels of perceived autonomy.

Furthermore, because SM is an iterative process, students are provided with increased opportunities to practice skills. Opportunities to practice novel skills leads to an increased sense of self-efficacy or competency surrounding the completion of a required behavioral task (Eccles, Early, Fraser, Belansky, & McCarthy, 1997; Niemiec & Ryan, 2009). With increased competencies, students are more likely to adopt and integrate those external requirements into their repertoire of internalized skills and values (Gagné, 2003). Furthermore, autonomy support, relevant instruction, and increased opportunities to practice and develop competencies have been shown to improve relationships between students and teachers. That is, autonomy support is an important mediator shown to facilitate healthy and trusting student-teacher relationships (Connell et al., 1993; Cox & Williams, 2008; Hamre & Pianta, 2003; Wentzel, 1993; Wentzel, 2002; Wentzel et al., 2007). Quality relations between students with challenging behaviors and their teachers diminishes challenging behaviors (Wentzel et al., 2007) and makes disciplining students who do engage in challenging behaviors more effective (Hamre & Pianta, 2003).

Why it is Important to do the Review?

Five reviews currently examine the impact of SM on student behavioral or academic outcomes. Four of the reviews examine the impact of SM on behavioral performance and one review examines the impact of SM on academic performance. The reviews are summarized in Table 1 in Appendix A. The first review examined the effects of SM on behavioral outcomes for 817 students across 42 studies, 38 of which were single case designs (Fantuzzo & Polite, 1990). In the second review on the effects of SM for behavioral outcomes, Briesch and Chafouleas (2009) followed the format of the Fantuzzo review to examine the effects of SM for 106 students across 30 studies, 21 of which were single case designs. The fourth review by Maggin and colleagues is omitted from Table 1 because the review utilized the same group of studies collected during the 2009 Briesch and Chafouleas review (Maggin et al., 2013). The major difference between the Briesch and Chafouleas (2012) review and a Maggin et al. (2013) review was that the latter applied WWC standards as inclusion criteria for single case studies. A fifth review examined the effects of SM on academic performance for students with disruptive behaviors (Mooney, et al., 2005). In this review, Mooney and colleagues (2005) located 22 studies examining SM with a total of 78 students. Again, the majority of the studies included were single-case designs. The systematic reviews, taken together, strongly suggest that a SM intervention impacts desirable behavioral and academic outcomes. However, the studies have limitations that prompt the current proposed review.

The most noteworthy limitation of prior SM reviews has to do with the methods used to generate summary effect sizes. Although prior reviews did not benefit from emerging methods to generate standardized summary effects, the methods used in the studies likely overestimate the effects of SM. Though a great deal of debate surrounds the best approach for synthesizing findings from single case designs, all of the prior reviews combined single subject and group studies in one review, combined multiple baseline and intervention phases, and used the “no assumptions” approach for estimating summary effects (Kratochwill et al., 2002). The no assumptions effect size is estimated by subtracting the mean of the baseline from the intervention mean and dividing the result by the baseline standard deviation. Such summary estimates, when not properly accounted for, inflate effect size estimates, evidenced in part by summary effects of SM ranging anywhere from 4.19 to 30.25 (Briesch & Chafouleas, 2009; Fantuzzo & Polite, 1990; Mooney et al., 2005).

The Maggin, Briesch, and Chafouleas (2013) review relied on two complimentary yet limited approaches to estimate summary effects. The first statistic used in both studies was the percent of nonoverlapping data (PND)—a common metric developed for use in single subject studies (Scruggs, Mastropieri, & Casto, 1987). The drawback of PND is that the approach does not account for the autocorrelation inherent in single case studies. Furthermore, PND does not account for baseline trends that may explain improvements observed during the treatment phase. The second statistic used in both studies was a standard mean difference effect generated using ordinary least squares models with fixed effects. Such approaches do not account for the wide heterogeneity observed when (a) combining phases within single case studies, (b) combining effects across single case studies, or (c) combining single case and group designs in single summary effects. The Mooney et al. (2005) review used a standard mean difference to estimate summary effects. However, Mooney (2005) only included the average of the last three data points in each phase—a practice that has been shown to inflate summary effect sizes (Olive & Smith, 2005).

A second limitation of the prior reviews hinges on the search procedures used in each of the studies. That is, the prior studies relied upon (a) the same search terms and (b) the same two databases (i.e., PsycINFO and ERIC). In addition, prior reviews included no “grey” literature strategies to include effects of published and unpublished sources not commercially controlled. To demonstrate that the proposed review has prepared a comprehensive and broader list of search terms and databases, we conducted an initial search and compared our results with the initial results reported in two of the most recent SM reviews. Our preliminary search used the search terms listed below and focused on the following databases: Academic Search Premier, Dissertation Abstracts International, ERIC, MEDLINE, PsycINFO, Social Service Abstracts, Social Work Abstracts, and Sociological Abstracts. The search yielded a total of 5,830 studies (see Table 2 in Appendix A). By comparison, Fantuzzo and Polite (1990) initially located 987 results and Briesch and Chafouleas (2009) located 794 results. Though exclusion criteria will certainly eliminate a substantial number of the results we identified, we feel that our proposed search strategy (i.e., a broader range of terms and databases) will capture a pool of relevant studies not included by our predecessors. In addition, the preliminary results displayed in Table 2 in Appendix A did not exhaust the full list of proposed databases or include efforts to capture grey literature.

Lastly, prior reviews did not take full advantage of emerging meta-analytic techniques. That is, the reviews did not use random effects models to generate summary estimates, did not present visual funnel or forest plots, the reviews lumped single subject and group designs together in the analyses rather than analyse the studies separately, and the reviews did not report results from Q test or I ² to index the level of heterogeneity present in final estimates or suggest methods to address any observed between-study variance. Prior reviews also did not test moderation models to examine whether outcomes varied by important features of student training, student characteristics, or examine the impact of SM by subtypes of challenging behavior (i.e., direct and indirect forms of aggression, overt and covert antisocial, and common acts of insubordination). Though two prior reviews did attempt to conduct component analyses of SM and investigate how specific elements were related to effect size estimates (Briesch & Chaffouleas, 2009; Fantuzzo & Polite, 1990), those studies did not take advantage of models that may examine whether student participation in each of the independent SM components impacted outcomes. Because researchers routinely hypothesize that direct student involvement in each SM process would impact the success of the intervention, such analyses would make an important contribution to the current state of research underlying the effects of SM.

Criteria for including studies in the review

The review will include two analyses. The first will assess the effect of multiple group studies (i.e., randomized control [RCT] and quasi-experimental designs [QED]). The second analysis will examine the effects of studies using single subject designs (SSD). The following criteria will be applied to determine the inclusion of any study to estimate the effects of SM on classroom behavior.

Search strategy

All search procedures will be conducted by two persons. To retrieve eligible studies, several search strategies will be utilized in an attempt to identify and retain published and unpublished studies. Hand searches will be conducted with sources that are not yet electronically available (i.e., the most recent publications waiting to be indexed). Broadly speaking, we will search available electronic databases, research registries, conduct steps to capture gray literature, and review the reference lists of the three most recent reviews of SM strategies for studies meeting the inclusion criteria. More specifically, we will systematically search: 1)

Electronic databases a.

EBSCO Academic Search Premier

Description of Methods Used in Primary Research

Appendix B includes a brief summary of the prior reviews of SM and the methods of each study. Due to several factors (i.e., the nature of a SM intervention, classroom setting, student behaviors), the methods most often used in studies are single subject designs. Many of those studies, according to reviews, do employ withdrawal designs to strengthen internal validity. Some studies examine the effects of SM using within group designs, and a small proportion will use multiple non-equivalent QED. To date, only one RCT of a SM intervention exists. Outcomes are generally measured using repeated observations of behavior (i.e., teacher or third party observations) and standardized pre, post, and follow-up ratings of behavior change. As a result, behavior change data from SM studies is generally analyzed by comparing baseline and intervention percentages using t-tests and visual or graphical displays of baseline and intervention functioning.

Criteria for Determination of Independent Findings

We are interested in one primary outcome, challenging classroom behavior, and one secondary outcome, academic achievement. All codable effect sizes for these two outcomes will be extracted using an electronic version of a preformatted abstracting form (see Appendix B), and each outcome will be analysed separately. We anticipate that included studies will use multiple measures for each outcome, multiple reports of the same outcome measure, multiple follow-up time points, and possibly more than one counterfactual condition. These circumstances create statistical dependencies that violate assumptions of standard meta-analytic methods. In order to ensure independence of study-level effect sizes, we will include only one effect size estimate from each independent sample on each outcome construct (i.e. challenging behaviors, academic achievement).

For cases in which a study uses multiple measures (i.e., daily observation and a standardized instrument) of the same construct, we will code data for each measure and create a study level average across the measures. Although we are conceptualizing the outcome of challenging behavior as encompassing the subcategories of behaviors noted above (i.e. aggression, antisocial, and insubordinate behavior), we recognize our definition of challenging behaviors encompasses a range of behaviors that may cluster into conceptually distinct outcomes. Therefore, we plan to code the specific subtype of behavior being measured for all codable effect sizes. The strategy will allow an examination of the relation between SM for different challenging behavior subtypes and to conduct sensitivity analyses of the impact of aggregating the subtypes into one study level effect size as well.

In cases of multiple reports on the same outcome (i.e. parent and child report), we will code data for each report and take an average of all reports. Because averaging reports may result in masking different viewpoints of different reporters, we will examine their concordance beforehand by performing sensitivity analysis to see if results vary across reporters (Littell et al., 2008).

In cases where multiple points of follow-up are provided, we will code follow-up points to conduct a separate analysis for effect sizes comparing studies with similar points of follow-up. In cases where studies report multiple measures or multiple reports on the same outcome, the mean of the measures will be taken to estimate a study-level average across the measures of the same construct. However, and reflecting the exploratory nature of the impact of behavioral SM, we will collect information with regards to any academic outcomes reported to examine the effects of behavioral SM on academic performance. In the case of multiple counterfactual conditions, we will select the comparison condition that is most similar to those in the other included studies.

In single case designs with multiple observations during the baseline and intervention phases, we will extract averages and standard deviations observed from multiple observations during each phase of the design. To reliably capture the information, we propose to use a graphical program such as UnGraph (Hedges, Pustejovsky, & Shadish, 2012; Shadish et al., 2009) to extract numeric data from the graphics.

Once numeric data are extracted, we will rely on the statistical guidance provided by Hedges and colleagues (2012) to extract smoothed estimates of functioning for each baseline and treatment phase. To apply the standards offered by Hedges and colleagues, we will collect and code data from single case studies to reflect the number of cases in each study (in cases of multiple cases in one study), the number of phases of treatment alteration, the number of data points per phase, and the relevant outcome values. In studies where there are multiple follow-up observations, we will also extract and code data in a manner that will allow us to examine the effect sizes of SM on student behavior from a longitudinal perspective.

In cases where we examine that a particular study may be reported in multiple reports or in the more likely case where multiple experiments are conducted across several cases but reported in one study, care will be taken to ensure that the study findings are represented only once in the summary effects. If it is unclear whether reports and studies provide independent findings, the authors of the reports will be contacted to clarify the discrepancy.

Details of Study Coding Categories

All studies retained from the search criteria outlined above will be retrieved and data will be coded and extracted using the instrument in Appendix B. Section A of the coding in Appendix B will capture all bibliographic information needed to locate the study (i.e., report type, how study was located, the country and geographic location of origin, and the language of the study). Section A will also capture all information needed to screen and determine the relevance of a particular study. Relevant study characteristics that need to be captured include whether the study type meets selected WWC design standards, whether the study takes place in a school setting, whether the study outcomes examine challenging behavior, etc.

Sections B and C will capture information relevant to the study methods dependent upon study design. Multiple group (RCT and QED) designs will use Section B whereas single case studies (SSD) will use Section C. Both sections also collect information needed to establish the quality of the state of the research underlying SM. Relevant data will include method of assignment, degree of attrition, information needed to estimate effect sizes for continuous and dichotomous outcomes, information related to group equivalence, type and scale of outcome measures, reliability estimates, the analytic strategy used, and the assessment of fidelity. For studies that fail to provide adequate detail to estimate effect size, the review team will directly contact the study authors to gather the necessary and relevant information.

Section D will capture information regarding participants. The information included here will focus on demographic data of all study participants, including subtypes of challenging behavior (see definition provided above), age and grade, race/ethnicity, sex, and whether the participant received free and reduced lunch. Information regarding the type of setting will also be captured in the Section C.

Section E will capture details specific to the SM intervention itself. The information here will include the number of SM elements that students are exposed to, the name of the SM intervention, information surrounding whether teacher monitoring procedures accompanied student self-monitoring, and whether students compared their data with teacher data or other means, etc. Information also captured in this section will provide some understanding of the impact of training for outcomes, including length of training, the intervention agent providing the training, and features of training modalities used. Lastly, information surrounding the type of goal selected, the types of prompts, and the features of the recording device are captured along with information on whether reinforcements were used.

Lastly, Section F will capture details with regards to fidelity or the extent to which delivery of an intervention adheres to a study's SM protocol. To understand the relationship between fidelity and SM, we want to examine the types of fidelity that are recorded (structural or procedural) and how those measures are collected and reported as well as the levels of fidelity provided in studies.

To increase reliability of coding procedures and decisions, two research team members will independently code 100% of all selected studies. All coded studies will be cross-referenced. Care will be taken to estimate inter-rater reliabilities for the coding scheme by using Cohen's Kappa. Two Kappa measures will serve as a measure of (a) between and (b) within study inter-rater agreement. The two Kappa's will be averaged to communicate a total study interrater reliability estimate to be included in our final report. Any emerging discrepancies between the two coders will be submitted to the research team for a final decision using majority rule.

To code studies, we elect to replicate our coding document in Appendix B using Google Docs. Using Google Docs, data entered into an electronic form is directly linked to a spreadsheet (https://support.google.com/drive/answer/87809?hl=en). This strategy will allow reduce the likelihood of data entry errors. Google permits the spreadsheet data to then be saved as a Microsoft Excel spreadsheet, which can then be imported into conventional analytical programs to conduct descriptive and meta-analyses (i.e., analysis of heterogeneity, sensitivity analysis, analysis for publication bias, main study effects, and moderation models) using the extracted data. To conduct meta-analyses, we propose to use SPSS. In cases where some proposed analytical approaches may not be accessible in SPSS, the data can be transferred into Stata (StataCorp, 2005), a general purpose statistical program that has freely accessible macros written by experts in meta-analysis that can be downloaded to address issues related to publication bias (i.e., funnel plot, rank correlation test, Egger's test, trim and fill). In addition, SPSS macros are available from developers of methods for estimating a standardized mean difference statistic for single case designs that is similar to Cohen's d for between study effect size estimates (Hedges et al., 2012).

Statistical Procedures and Conventions

Once the data are imported into SPSS, we will conduct descriptive analyses on all variables of interest to provide information regarding •

the characteristics of study participants (e.g., subtypes of challenging behaviour, education status, gender, race, income level, grade, age),

Following descriptive analysis, the data will be examined for the impact of publication bias by examining the symmetry of a funnel plot where the relative effect size and sample size are plotted for each study. Should the funnel plot indicate the possibility of bias, Trim and Fill methods may be one option to adjust for publication bias (Cooper, Hedges, & Valentine, 2009). Stata macros are freely available to assist the authors to adjust for publication bias using the Trim and Fill method (Duval & Tweedie, 2000). Sensitivity analysis will also be conducted using forest plots to examine any biasing effects due to outliers. If necessary, additional sensitivity analysis will be conducted should other issues arise that may impede our confidence in the pooled estimates.

Next, we will estimate effect sizes for each individual study. For RCT and QED studies, we will use the natural log of the odds ratio effect size statistic for dichotomous outcomes and the standardized mean difference with Hedges' g correction for continuous outcomes. For SSD studies that will be included using the WWC criteria, we elect to estimate study level effect sizes using a standardized mean difference effect, which similar to a Cohen's d as described by Hedges and colleagues (2012). Indeed, prior comparative studies of available effect size options for SSD studies (i.e., PND, general linear approaches, and the percentage reduction measure) suggest that the standardized mean difference effect is the most consistent effect size metric for single case designs (Olive & Smith, 2005). That said, the authors acknowledge that there are other means for producing study level effect sizes for small datasets that are characteristic of SSD.

In group designs where groups of students, rather than students, have been assigned randomly, it will be necessary to correct for the clustering effects in the calculation of effects. If studies report cluster-adjusted data, the adjusted data will be utilized to conduct data analysis. If cluster adjustment has not been accounted for, corrections will be made using the intraclass correlation coefficient prior to calculating effect size (Littell, Corcoran, & Pillai, 2008).

Following the estimation of individual study level effects, we will then conduct separate meta-analyses to pool studies into summary effects for group (i.e., RCT and QED studies) and SSD studies. For RCT and QED studies, a weighted mean effect will be calculated by weighting each study level effect size by the inverse of its variance (Littell et al., 2008). For SSD studies, we propose to use the d-statistic as described by Hedges and colleagues (2012). Because methods for synthesizing the results across SSD are developing, further research and debate may arise which could provide a compelling case for using a different summary effect statistic for SSD. However, we elect to use the standardized mean difference effect size for single case designs because it is consistent with our study level approach and it is similar to the effect statistic we are using for between group experiments. Furthermore, the standardized mean difference effect size across SSD studies addresses 1) the number of time-points within cases; 2) the number of cases; 3) the autocorrelation arising from repeated measures; 4) the ratio of within case variability to total variability; 5) a Hedges' g correction for the small sample size; and 6) variation in the number of data points phase and repetitions of the AB phases (Hedges et al., 2012). Although methods of calculating effect size estimates for SSD are relatively new and in development, we have selected this technique at this point in time because it is fairly well-developed and we have access to the code and resources to apply the method. Lastly, random effects models are the most appropriate choice for the estimation of effects sizes as we anticipate significant between study heterogeneity when estimating the effects between RCT and QED studies and between the SSD studies.

Following the estimation of summary effects, we will conduct a test of homogeneity (Q-test) to compare the observed variance to what would be expected from sampling error. The I² statistic will also be used to describe the percentage of total variation across studies due to the heterogeneity rather than chance. We will also construct a forest plot displaying study-level mean effect sizes and 95% confidence intervals for the included studies to provide opportunity for visual analysis of the precision of the estimated effect sizes, detection of studies with extreme effects, and information regarding heterogeneity of studies.

Following the estimation of main intervention effects and sensitivity analysis, moderation models will examine whether subgroup differences exist. Provided that ample information exists, we will use moderation models to examine whether the effects of SM differ by student characteristics (e.g., age, gender, grade, race/ethnicity, regular or special education status, socioeconomic status, etc.). Moderation analyses will also be used to examine whether any study characteristics moderate the effects of SM strategies (e.g., design features, standardized versus observational studies, reliability estimates, measurement approaches, etc.). Provided that an ample number of studies and coded data exist, moderation models will then be used to examine whether the presence of student training and whether the features of student training moderate the relationship between SM and outcomes.

No modifications will be made to model missing data in any of the analytical steps outlined above. However, moderation models may be used to determine the differential effect sizes for studies that report characteristics to conduct moderation models and those that do not.

Treatment of Qualitative Research

Although the reviewers will note any qualitative data throughout the search process, we currently have no intention of conducting qualitative analysis as part of this review.