Considerations for Intensifying Word-Problem Interventions for Students With MD: A Qualitative Umbrella Review of Relevant Meta-Analyses

Search Results, Title, Abstract, and Article Screening

Figure 1 shows a PRISMA diagram of the study-selection process. The electronic search yielded 856 titles and abstracts. Two experienced screeners with expertise in conducting systematic reviews and meta-analyses independently checked these records, ensuring double screening for accuracy. At each stage, we calculated inter-rater reliability (IRR) as the number of agreements divided by the sum of disagreements and agreements multiplied by 100 and discussed discrepancies to reach a consensus. We used Endnote to identify and remove 371 duplicates. After de-duplication, we retained 485 unique abstracts and titles for screening using the inclusion and exclusion criteria (Table 1). We used Rayyan, a Web-based application, to screen the titles and abstracts obtained through Endnote. Rayyan uses machine learning algorithms to select studies to include in a systematic review (Ouzzani et al., 2016).

OPEN IN VIEWER

Figure 1.

PRISMA Diagram Showing the Search and Retrieval Process.

OPEN IN VIEWER

Table 1.

Inclusion and Exclusion Criteria for Selecting Meta-Analyses in the Systematic Umbrella Review.

Inclusion criteria

Exclusion criteria

We included meta-analyses based on the following criteria: 1. Intervention Focus. The review focused on interventions designed to address WPS outcomes among students with MD. 2. Design: We included meta-analyses that utilized either group-based (experimental or quasi-experimental) or single-subject design methodologies (e.g., multiple-baseline or multiple-probe across subjects). While single-subject research (SSR) has limitations in generalizability, it plays a crucial role in informing evidence-based practices with MTSS due to its suitability for studying individualized interventions for students with MD (Byiers et al., 2012; Horner et al., 2005). 3. MD criteria: The meta-analysis included a clear and explicit definition of MD used in the primary studies. 4. Moderator Analysis: The meta-analysis employed statistical techniques (e.g., subgroup or meta-regression analyses) to explore sources of heterogeneity in the reported intervention effects. 5. Publication type and Language: The meta-analysis was published in a peer-reviewed journal and was presented in English.

We excluded meta-analyses based on the following criteria: 1. Intervention Focus: The meta-analysis included WPS outcomes. However, its primary focus was not limited to interventions designed for WPS improvement. (e.g., Myers et al., 2023). 2. Outcome Specificity: The study addressed a comprehensive range of mathematics outcomes beyond WPS (e.g., Myers et al., 2021). 3. MD Criteria: The meta-analysis lacked precise details regarding the criteria for identifying students with MD (e.g., Peltier & Vannest, 2017). 4. Moderator Analysis: The meta-analysis presented only a qualitative synthesis of WPS outcomes without employing quantitative analysis techniques to identify moderators (e.g., Cook et al., 2020)

Note. WPS = word problem-solving; MD = mathematics difficulties; MTSS = multi-tiered systems of support.

We screened the 485 abstracts and titles using Rayyan and excluded 465, with an initial IRR of 85%. After discussing disagreements and reaching a consensus, we retrieved the full-text versions of the remaining 20 records and screened them using the inclusion and exclusion criteria. We excluded 13 articles, with an initial IRR of 87%. After discussion, we reached a 100% agreement and retained the remaining seven articles for inclusion in the study. Our manual searches of the reference lists of reviews meeting the inclusion criteria yielded two additional studies. We retained one of these studies in the final sample as it met our inclusion criteria. Our hand search did not produce any new articles. The IRR for this process was 100%.

Our systematic search yielded eight articles, three of which included only GDR (i.e., Kong et al., 2021; Lein et al., 2020; Myers et al., 2022), two focused solely on SSR (i.e., Lei et al., 2020; Shin et al., 2021), and three reported separate findings for both types of research (i.e., Xin & Jitendra, 1999; Zhang & Xin, 2012; Zheng et al., 2013). Our final sample comprised 11 meta-analyses (i.e., six GDR and five SSR reviews).

Study Quality Appraisal

Rigorous methodology is the foundation for trustworthy research; umbrella reviews that synthesize evidence from multiple meta-analyses are no exception. The quality of the underlying studies significantly influences the interpretation and reliability of findings presented in umbrella reviews (Faulkner et al., 2022). High-quality meta-analyses, employing robust methodologies that minimize bias and random error, ultimately provide a more solid foundation for the interpretations presented in this umbrella review. Conversely, lower-quality meta-analyses can introduce bias or methodological flaws that propagate to the umbrella review, potentially leading to misleading interpretations of the overall evidence (Fusar-Poli & Radua, 2018). Therefore, assessing and ensuring the methodological quality of meta-analyses included in umbrella reviews is crucial. By evaluating the quality of the included research, we enhance the credibility and trustworthiness of the findings, providing a more accurate and dependable synthesis of the evidence. To minimize bias and establish reliable findings in our umbrella review, we assessed the methodological rigor of the included meta-analyses, recognizing the importance of quality assessment (Fusar-Poli & Radua, 2018). While tools like the Assessment of Multiple Systematic Reviews (AMSTAR) exist, they are primarily designed for healthcare research (Kung et al., 2010). This highlights a critical gap in the social sciences where dedicated instruments for evaluating the quality of meta-analyses are lacking.

We adapted the revised AMSTAR (R-AMSTAR) scale (Kung et al., 2010) for our research. The R-AMSTAR checklist is well-suited for our study as it assesses 11 core methodological aspects of meta-analyses: (a) a priori design; (b) duplicate study selection and data extraction; (c) comprehensiveness of literature search; (d) inclusion of grey literature; (e) list of included and excluded studies; (f) characteristics of included studies; (g) quality assessment of included studies; (h) appropriate use of study quality in formulating conclusions; (i) methods for combining findings; (j) publication bias; and (k) conflict of interest (Kung et al., 2010). We made minor modifications to the tool, substituting healthcare-related examples (e.g., disease status) with information pertinent to our research (e.g., grade level). These changes were minimal, preserving the tool’s integrity. The tool’s flexible structure allows it to be applied across various fields, including social science research (e.g., Filiz, 2023).

We developed a comprehensive coding manual through a systematic process involving discussions among the research team to ensure consistent and reliable quality assessment. This manual provided detailed criteria and definitions for evaluating each aspect of the R-AMSTAR checklist. Using this established coding manual, the first and second authors independently assessed the quality of each meta-analysis using the R-AMSTAR checklist (see Supplemental Figures S1 and S2 for the checklist coding and manual, respectively). They coded each criterion within an item as “Yes” (met) or “No” (not met). Following the specific scoring guidelines for each item, they assigned an overall score (1–4), with higher scores indicating better quality. Finally, using the final score (sum of all item scores, with a maximum of 44), they ranked the quality of each meta-analysis as insufficient (0–11), low (12–22), medium (23–33), or high (34–44) based on Youngs’ (2017) criteria. We calculated the IRR to ensure consistency, obtaining 82% agreement and resolving discrepancies to achieve 100% agreement. We retained all studies in the review but reported the quality scores alongside the findings to allow readers to assess the potential influence of methodological limitations on the result.

Overlap Analysis

To ensure the validity and reliability of our umbrella review, we evaluated the extent of overlap across the included meta-analyses, following established guidelines (Hennessey et al., 2019). Overlap occurs when multiple meta-analyses incorporate the same primary studies, potentially introducing bias if not adequately addressed (Faulkner et al., 2022; Hennessy & Johnson, 2020). We assessed overlap using two complementary approaches: visual inspections of citation matrices and interpretation of the Corrected Covered Area (CCA) index (Pieper et al., 2014). We evaluated overlap across the SSR and GDR meta-analyses separately. For the visual checks, we created a table with each primary study citation in a row and each meta-analysis in a separate column, marking the included primary studies with checkmarks (Pieper et al., 2014). We then examined the patterns and relationships in the matrix.

For the second approach, we calculated and interpreted the CCA using data from the citation matrices. The CCA was calculated using the following formula: CCA = (k − r) / (r [c − r]). In this formula, k is the total number of primary studies across the meta-analyses (i.e., all checkmarks), r is the number of rows (i.e., the number of unique primary studies), and c is the number of columns (i.e., the number of meta-analyses). We used criteria given by Pieper et al. (2014) to assess the degree of overlap: slight (0%–5%), moderate (6%–10%), high (11%–15%), or very high (>15%). Based on these categories, we interpreted the extent of overlap to understand its potential impact on our findings and to ensure that any conclusions drawn account for the possible bias introduced by overlapping studies. Recognizing that high overlap can inflate the precision of meta-analytic estimates, we accounted for this overlap in our review to mitigate potential bias (Hennessy & Johnson, 2020; Pieper et al., 2014).

Coding Procedures

To ensure consistency in the coding process, we employed a systematic approach for developing coding tools and utilized a double-screening method to extract information from the included meta-analyses. The first author initially created a web-based coding survey and a comprehensive coding manual. Subsequently, a co-author then reviewed the documents and provided feedback for further refinement. The first and second authors collaboratively discussed the feedback and incorporated the suggested modifications to create draft versions of the coding survey and manual. We conducted a pilot study to assess the draft coding tools’ effectiveness. The first and second authors independently coded a single study using the draft coding manual and survey. They then compared and discussed their coding results, meticulously evaluating any discrepancies to identify areas for improvement in the coding tools. Based on the pilot study’s findings, we made final revisions to the coding manual and survey.

Using the finalized coding tools, the first and second authors independently coded various aspects of each study, including descriptive information (e.g., search year and number of studies) and results of the moderator analyses of intervention and participant characteristics. Notably, we did not extract data on methodological characteristics, such as the nature of treatment conditions, as they were not of interest in this study. The IRR for this process was 92%. After discussing inconsistencies and applying the established coding definitions, we reached 100% agreement. The features we coded for each study are presented in Table 2.

OPEN IN VIEWER

Table 2.

Coding Summary of Extracted Characteristics and Outcomes from Meta-Analyses.

Feature	Description
Study Identification	We recorded the full title, authors, and publication year of the meta-analysis.
Design of Included Research	We recorded the research design used in the primary studies included in the meta-analysis, distinguishing between reviews synthesizing group design research (i.e., experimental and quasi-experimental designs) and those synthesizing single-subject research (SSR) methodologies (e.g., multiple-baseline, multiple-probe across subjects).
Grade Level	We coded the range or categories of grade levels targeted in the review (e.g., K–12, K–6, Grades 1–5).
Search Years	We recorded the years covered in the literature search for primary studies included in the meta-analysis.
MD Criteria	We recorded specific details or operationalization of the criteria used to define and identify students with MD.
Addressed Dependency in Effects	We coded if the study included methods to address effect size (ES) dependency, such as Robust Variance Estimation (RVE). (Yes/No)
Addressed Outliers	We recorded whether the methods included specific statistical treatments for outliers (e.g., winsorization or sensitivity analyses) of ESs. (Yes/No)
Number of Studies	We noted the total number of primary studies included in the meta-analysis.
Number of Effect Sizes	We noted the total number of effects extracted from the primary studies included in the review.
Pooled Effect Size	We extracted the mean ESs reported in the studies, specifying the measures used (e.g., Cohen’s d, Hedges’ g, and Tau-U). When authors provided estimates with and without outliers, we prioritized those excluding outliers to avoid bias.
Moderator Analysis Results	Through detailed analyses of the moderator sections in each meta-analysis, we extracted information on potential moderators representing intervention (e.g., intervention model) and participant characteristics (e.g., academic risk area): • Categories examined: The specific categorization the authors used for the moderator (e.g., academic risk area: MD vs. MD & RD) • ESs: Reported ES for each moderator category and the specific measure used. • Heterogeneity index: Statistical measure of the variability in ESs across studies between and within each moderator category, such as I2 (I-squared) or τ2 (Tau-squared). • 95% confidence intervals (CI): Range of plausible values for the estimated ES in each moderator category. • p-value: Statistical significance level for the observed heterogeneity within each moderator category.

Note. MD = mathematics difficulties; RD = reading difficulties.

Connecting Moderators of WPS Outcomes to TII Dimensions

We thoroughly reviewed the literature to understand how intervention and participant characteristics, examined as moderators of WPS outcomes across the meta-analyses in our umbrella review, correlate with relevant TII dimensions. We aimed to connect these potential moderators to the TII framework, focusing on six dimensions, excluding intensity. These dimensions pinpoint factors crucial for intensifying instruction for students with MD. Unlike the others, the intensity aspect concentrates solely on the overall intervention effectiveness (mean effect), which does not align with our goal of understanding how to intensify instruction for these students. The links were not mutually exclusive, allowing for a comprehensive understanding of multiple influences on outcomes. The first and second authors collaboratively drafted the initial linkage, which was reviewed and refined through feedback from two additional authors. This iterative process involved multiple rounds of review and revision until we reached a consensus on all the links between the dimensions and moderators.

Table 3 serves as a structured overview of the findings, presenting the potential theoretical associations between intervention dimensions and various participant and intervention characteristics examined as moderators across the meta-analyses. The first column lists the dimensions (e.g., Dosage) and potential moderators indicative of that dimension. The second column provides a detailed explanation of each dimension’s application in the context of intervention intensity, breaking down the components and explaining their significance in the overall intervention process. For instance, under “Dosage,” the second column distinguishes “Dosage Across the Intervention” from “Dosage Within the Intervention,” highlighting aspects such as the number of sessions and opportunities for student engagement. The third column offers explanations, grounded in the literature, of how each moderator may be linked to the dimensions.

OPEN IN VIEWER

Table 3.

Connections Between the Dimensions of the Taxonomy of Intervention Intensity and Intervention and Participant Characteristics Examined as Moderators.

Dimension & associated moderators	Description of dimension	Explanations
Dosage (a) Intervention Frequency (b) Number of Treatment Sessions (c) Intervention Duration (d) Group size	In intensifying instruction, two areas of instructional dosage need consideration: (a) Dosage across the intervention, which refers to the number of minutes per session, total number of sessions, and frequency of instruction, and (b) Dosage within session addresses factors related to the opportunities for students to actively engage with the intervention and receive corrective feedback. It focuses on the number of opportunities for students to respond and receive feedback during each session.	Dosage Across the Intervention Three possible study characteristics include intervention frequency, number of treatment sessions, and duration of interventions (Powell, Benz, et al., 2022). An increase in at least one of these characteristics would increase student exposure to explicit instruction, increase the number of opportunities to learn and practice, and subsequently improve student performance. Dosage Within the Intervention Reducing instructional group size empowers teachers to personalize instruction and maximize instructional dosage, fostering greater student engagement and opportunities to respond (D. Fuchs et al., 2014). Smaller groups enable frequent corrective feedback, progress monitoring, and individualization, ensuring instruction supports each student’s needs (D. Fuchs et al., 2014).
Alignment (a) Participant Math Difficulty (MD) Status (b) Academic Risk Area (c) English Learner (EL) Status (d) Subject Area Focus (e) Culturally Responsive Practices (f) Content Domain (g) Word Problem Type/Task (h) Broad Math Achievement (i) CCSS-Math Content Standards (j) CCSS-Math Practice Standards (k) Student-Directed Learning	Two essential areas of alignment are critical for word problem instruction: (a) Student alignment: Consider students’ academic characteristics (such as MD status, academic risk area, and EL status), cultural background, and behavioral characteristics when selecting word problem interventions, and (b) Curriculum alignment: Select interventions that align with the specific math content standards and practice standards addressed by the word problems being taught. These two areas of alignment can interact and influence each other. For example, an EL student with MD and low student-directed learning might need an intervention combining CRP strategies and explicit instruction on self-regulation strategies to manage their behavior and learning. The selected intervention should also be linked to the specific math content and practice standards covered in the word problems they must solve.	Student Alignment The success of WPS interventions hinges on understanding and addressing individual student characteristics. Two key factors influencing intervention outcomes are participants’ MD status (e.g., LD vs. At-risk) and academic risk area (i.e., MD only vs. MD and RD). Research consistently demonstrates that students with LD generally require more intensive instruction than their at-risk peers (Compton et al., 2012; L. S. Fuchs et al., 2002). This increased intensity arises from the specific learning challenges associated with LD, which often require more explicit and long-term support. Furthermore, studies have shown that students experiencing comorbidity of MD and RD perform significantly lower in WPS tasks than those with MD alone (e.g., Myers et al., 2022). This suggests that RD adds additional complexity to the learning process, necessitating tailored intervention approaches that simultaneously address MD and RD (Powell et al., 2020). Therefore, carefully considering student MD status, the presence of comorbid RD, and individual academic risk areas is crucial for ensuring appropriate instructional alignment and maximizing the effectiveness of WPS interventions (Myers et al., 2022). Additionally, the language demands of word problems can further complicate the situation for English learners (ELs) with MD. This population often faces more significant WPS challenges than their peers without language difficulties (Lei et al., 2020). They must simultaneously decipher the mathematical concepts and the linguistic intricacies of the problem itself (King & Powell, 2023). Consequently, ELs with MD necessitate targeted and individualized instruction that directly addresses their multifaceted learning needs (Lei et al., 2020). This necessitates incorporating culturally responsive practices (CRP) into their word problem instruction. CRP strategies (e.g., utilizing culturally relevant contexts and promoting the use of students’ native languages) may help ELs with MD better connect with the material and overcome the additional linguistic barriers they face (Kong et al., 2023). Addressing ELs’ linguistic and cultural needs with MD is crucial for enhancing their WPS skills (Lei et al., 2020). Student-directed learning (SDL), where students actively manage their learning process, is crucial in enhancing word problem interventions for students with MD, particularly those with LD. Compared to their typically developing peers, MD/LD students are more likely to display internalizing (e.g., anxiety) and externalizing (e.g., aggression) behaviors that negatively impact their learning (Jordan et al., 2020). SDL empowers students to become more self-aware and develop coping mechanisms for managing these behaviors. Students with high levels of SDL can effectively manage their learning and behavior independently (Fitzpatrick & Knowlton, 2009). However, students exhibiting lower levels of SDL may significantly benefit from WPS programs that incorporate explicit instruction on self-regulation strategies. These strategies equip students with tools like self-monitoring (observing and recording one’s behavior and emotions) and self-talk (positive self-affirmations and self-talk) to manage their behavior effectively (Feeney, 2022), fostering a more conducive learning environment.
		Alignment with Curriculum Furthermore, students with MD struggle to perform calculations involving number operations and fractions (Aunio et al., 2015; Shin & Bryant, 2017); therefore, the content domain targeted by an intervention should be paramount in instructional alignment.The subject area focus of the intervention is crucial for linking word problem instruction with the curriculum because it ensures the skills being taught are relevant and applicable to the specific content area (Lei et al., 2020). For example, a WPS intervention targeting both reading comprehension and mathematics may help students make connections between literacy and numeracy, enhancing their understanding of word problems in math contexts. Alternatively, interventions focusing solely on one area can provide more targeted support for that skill. Notably, students with MD and RD might significantly benefit from interventions addressing both areas (Myers et al., 2022). This integrated approach can facilitate connections between literacy and numeracy, enhancing their overall understanding and application within word problem contexts (Arsenault & Powell, 2022a).Students must also solve different word problems, such as those requiring one or multiple. A key consideration is how the type of word problem addresses students’ academic skill deficits (Powell, Berry, et al., 2022).The Broad Math Achievement score serves as a screening measure of students’ achievement on standardized achievement tests, such as the Woodcock-Johnson Psycho-Educational Battery Tests of Achievement. These tests assess foundational skills, including operations with whole numbers and applied problems. Many students with MD struggle with these foundational skills, which are essential for understanding and solving word problems (Powell et al., 2020). Broad Math scores are valuable as they provide a quantifiable measure of these foundational skills. By considering students’ current achievement levels, educators can pinpoint areas of weakness and customize WPS instruction to bridge these gaps.Connecting WPS instruction for students with MD with rigorous mathematics standards is also vital (Shin et al., 2020). These students are expected to demonstrate proficiency on assessments built on rigorous general education math content standards, such as the Common Core State Standards-Mathematics (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). These assessments may include word problems related to CCSSM’s math content standards (i.e., math topic) and practice standards (i.e., pedagogical approach). Hence, in planning WPS instruction for students with MD, practitioners are encouraged to select interventions that emphasize building a solid foundation in the specific math content standards and the practice standards outlined in the CCSSM (Shin et al., 2021).
Attention to Transfer (a) Group Size(b) Intervention Model	The intervention program should support students in (a) applying mathematical skills or concepts learned in one context to a different context and (b) realizing connections between the concepts they have learned and related skills.	Manipulating instructional group size to create small learning groups can be a powerful strategy for enhancing the transfer of learning (Pai et al., 2015). It allows for personalized instruction, immediate feedback, peer collaboration, and continuous adjustment of instruction based on student needs, all of which contribute to a deeper, more flexible understanding that students can apply in new contexts (Pai et al., 2015).Furthermore, intensive WPS intervention programs for students with MD should prioritize explicit instruction on the transfer of learning (Powell, Berry, et al., 2022). Simply acquiring WPS knowledge and skills is insufficient; students require the tools and strategies to apply them across contexts and tasks. Intervention models that explicitly guide students through this process are crucial for maximizing the impact of WPS programs and fostering long-term WPS success (Powell & Fuchs, 2018)
Comprehensiveness (a) Intervention Model	The number of explicit instruction principles within the intervention is essential for providing clear and concise explanations, modeling efficient solution strategies, providing guided and independent practice, and incorporating strategies to help students monitor their understanding and adjust as needed.	Practitioners’ choice of intervention model may significantly impact the comprehensiveness of WPS instruction for students with MD. SBI models are a multi-component program offering a holistic approach to WPS instruction. This program goes beyond simply teaching problem-solving steps; it explicitly guides students to identify the underlying structure (schema) of each problem, represent it accurately, formulate a plan for solving it, execute the plan, and check the solution’s appropriateness (Powell et al., 2020). Additionally, SBI incorporates strategy instruction (e.g., the use of cognitive and meta-cognitive strategies), where students regulate and monitor their learning and use attack strategies, mnemonics, or general heuristics to effectively remember and implement the solution steps (Witzel et al., 2022). In contrast, other WPS models, like strategy instruction and direct instruction, while valuable, are less comprehensive (Myers et al., 2022). While they, like SBI, prioritize assisting students with sequencing the steps of solving problems, they lack the explicit focus on the fundamental problem structure that distinguishes SBI (Lein et al., 2020). This difference can significantly impact the development of students’ deep understanding and flexible WPS skills.
Behavioral Support (a) Intervention Setting(b) Student-Directed Learning(c) Group Size	Including (a) self-regulation and executive function components and (b) behavioral principles to support productive behavior in word problem intervention may be essential, particularly for students with MD who display behavioral challenges. Specifically, these components may help students with MD to regulate their attention, manage their emotions, and stay on task during word problem instruction.	The setting in which WPS interventions take place, and the group size may interact to play a critical role in the effectiveness of behavioral support. Large general education classrooms, while convenient, often limit the ability of practitioners to provide individualized behavioral support to students with MD (Powell, Berry, et al., 2022). This can hinder their ability to focus, manage distractions, and ultimately learn effectively.Smaller instructional groups in separate settings offer a significantly more supportive environment for enhancing behavioral support in WPS interventions (Powell, Berry, et al., 2022). These settings provide several benefits for students with MD, such as increased teacher attention, reduced distractions, and enhanced self-regulation training (Powell, Benz, et al., 2022).Student-directed learning is also crucial in tailoring behavioral support and WPS intervention strategies for students with MD. As noted, many of these students display behavioral challenges that may inhibit their learning (Jordan et al., 2020). While SDL empowers students with practical self-regulation skills, those requiring additional support may benefit from explicit instruction in self-regulation strategies like self-monitoring and self-talk (Feeney, 2022). When WPS instruction matches students’ individual behavioral needs, educators can create a more supportive and conducive learning environment for all students (Powell, Berry, et al., 2022).
Individualization (a) Group Size(b) Intervention Setting	A key consideration for word problem instructional programs for students with MD is a data-driven process for individualizing intervention, in which the educator systematically adjusts the intervention over time in response to the student’s progress-monitoring data.	Smaller instructional groups are more effective for facilitating instruction as they allow more concentration on students’ needs (Powell, Berry, et al., 2022).The intervention setting should also be considered given that teachers typically provide such instruction outside the general education classroom (Vaughn et al., 2012).

Note. MD = mathematics difficulties, LD - learning disabilities; RD = reading difficulties; WPS = word problem-solving; CCSS-Math = Common Core State Standards�Mathematics.

Data Analysis

We used a two-step process to answer our research question. In the first step, we conducted a detailed qualitative synthesis of each meta-analysis. Instead of focusing on characteristics correlated with the TII framework, our approach examined all intervention and participant characteristics reported as potentially influencing intervention effectiveness. This broader approach ensured we captured the full range of potential moderators identified across the studies, aligning with Booth’s (2016) recommendations for transparency. It also enhances the research’s credibility by providing a more complete picture of the factors considered in the original moderator analyses. Specifically, we summarized the results of these analyses to understand how these characteristics affected the outcomes. Our summaries included the ES, their corresponding confidence intervals (CIs), and measures of heterogeneity, such as I² and tau-squared (τ²).

We used established criteria to evaluate the strength of the ESs and the level of heterogeneity among the studies included in the meta-analysis. We interpreted the magnitude of the reported effects using benchmarks specific to educational interventions in mathematics, established by Bloom et al. (2008). These benchmarks focus on annual gains in student achievement in mathematics, providing a more meaningful context for interpreting the intervention effects than generic benchmarks. For example, an ES of 0.3 might be considered a moderate ES based on the benchmarks set by Bloom et al., indicating an annual gain equivalent to 3 months of additional learning. Higher values of the I² statistic and τ² indicate more significant heterogeneity or inconsistency among the included studies. Common interpretations suggest that I² values above 50% or τ² values exceeding 0.1 indicate substantial heterogeneity (Higgins & Thompson, 2002).

The second step focused on a cross-study analysis of moderators correlated with the TII framework (details in Table 3). We aimed to identify intervention and participant characteristics within this framework that consistently exhibited statistically significant moderating impacts on the effectiveness of WPS interventions for students with MD. Using a qualitative approach, we examined moderators demonstrating consistency in the direction and magnitude of their effects across the studies. This analysis sought to identify patterns and trends in how these TII-linked moderators impact intervention effectiveness. Ultimately, we aimed to inform efforts to intensify WPS instruction for students with MD by determining characteristics related to the relevant TII dimensions that consistently moderate intervention effectiveness.

Descriptive Summary of Study Information and Quality Appraisal Results

We present descriptive information and the quality appraisal results for the GDR and SSR reviews in Tables 4 and 5, respectively. A more detailed version of the quality appraisal results is available in the Supplemental Material (See Table S1). Across the GDR reviews (n = 6), researchers synthesized 85 unique primary studies, while the SSR reviews (n = 5) comprised 41 studies. The mean ESs from these reviews were positive, ranging from moderate to large. These findings suggest that students with MD who participated in WPS interventions experienced meaningful improvements in their problem-solving skills. A moderate ES, for example, translates to approximately 3 months of additional learning compared to students who did not receive such interventions (Bloom et al., 2008). Notably, while the authors of the included meta-analyses reported measures of heterogeneity, the majority relied on the Q statistic, which has limitations (Higgins & Thompson, 2002). More robust measures of heterogeneity, such as I² and τ², were not consistently reported. This inconsistency limits our ability to definitively analyze the extent of variation (heterogeneity) across the studies’ findings.

OPEN IN VIEWER

Table 4.

Study Design Summary and Quality Appraisal Results for the Meta-Analyses of Group Design Research (GDR).

Feature	Xin & Jitendra (1999)	Zhang & Xin (2012)	Zheng et al. (2013)	Lein et al. (2020)	Kong et al. (2021)	Myers et al. (2022)
Study Design
Grade Level	K to Post-secondary	K to 12	K to 12	K to 12	K to 6	Grades 1 to 5
Search Years	1960–1996	1996–2009	1986–2009	1989–2019	1990–2019	1975–2021
MD Criteria	Diagnosis of a mild disability, such as LD & ADHD, or at risk for math failure	Diagnosis of LD	Scores below the 25th percentile (standard score of 90) on a norm-referenced math test	Diagnosis of LD or scores below the 35th percentile on a standardized math measure	Diagnosis of LD or at risk for LD based on scores below a designated cutoff on a math assessment	Diagnosis of LD or at risk for LD based on scores below a designated cutoff on a math assessment
Addressed Dependency in Effects	No	No	No	Yes a	No	Yes b
Addressed Outliers	Yes	No	Yes	Yes	No	Yes
Number of Studies	14	29	7	31	18	52
Number of Effect Sizes	14	29	7	34	113	231
Pooled Effect Size	d = 0.89	d = 1.58 c	g = 0.45 (MD and RD) c g = 0.95 (MD only) c	g = 0.56	g = 1.08	g = 0.81
Quality Appraisal (R-AMSTAR)-Results
Overall Score and Quality Rating	33 (High)	31 (Medium)	31 (Medium)	35 (High)	27 (Medium)	40 (High)
Score by Each Item
1. A priori design	3	3	3	3	3	3
2. Duplicate study selection and data extraction	4	4	4	4	4	4
3. Comprehensive literature search	4	4	4	4	3	4
4. Inclusion of gray literature	3	3	4	3	3	3
5. List of included and excluded studies	3	3	3	4	1	4
6. Characteristics of included studies	4	4	3	4	3	4
7. Quality assessment of included studies	1	1	1	1	1	4
8. Use of quality in formulating conclusions	1	1	1	1	1	4
9. Methods for combining findings	4	4	4	4	4	4
10. Assessment of publication bias	3	1	1	4	1	3
11. Conflict of interest	3	3	3	3	3	3

Note. LD - learning disabilities; ADHD = attention-deficit/hyperactivity disability; MD = math difficulties; RD = reading disfficulties; WPS = word problem-solving; R-AMSTAR = Revised Assessment of Multiple Systematic Reviews.

a

The authors selected the treatment that best represented the word problem-solving instruction. ^b Used robust variance estimation (RVE). ^c Lacked assessment of publication bias.

OPEN IN VIEWER

Table 5.

Study Design Summary and Quality Appraisal Results for the Meta-Analyses of Single-Subject Research (SSR).

Feature	Xin & Jitendra (1999)	Zhang & Xin (2012)	Zheng et al. (2013)	Lei et al. (2020)	Shin et al. (2021)
Study Design
Grade Level	K to Post Secondary	K to 12	K to 12	K to 12	K to 6
Search Years	1960–1996	1996–2009	1986–2009	Not Reported	1975–2020
MD Criteria	Diagnosis of a mild disability, such as LD & ADHD, or at-risk for math failure	Diagnosis of LD	Scores below the 25th percentile (standard score of 90) on a norm-referenced math test	Diagnosis of LD or low math scores and considered as English learners (ELs)	Diagnosis of LD
Addressed Dependency in Effects	No	No	No	No	Yes a
Addressed Outliers	Yes	No	Yes	Yes	No
Number of Studies	12	10	8	10	20
Number of Effects	15	13	27	72	39
Pooled Effect Size	PND = 89%	PND = 95% b	d = 0.90 b	TAU = 0.81	BC-SMD = 4.52 c
Quality Appraisal (R-AMSTAR)-Results
Overall Score and Quality Rating	33 (High)	31 (Medium)	31 (Medium)	36 (High)	37 (High)
Score by Each Item
1. A priori design	3	3	3	3	3
2. Duplicate study selection and data extraction	4	4	4	4	4
3. Comprehensive literature search	4	4	4	4	4
4. Inclusion of gray literature	3	3	4	3	2
5. List of included and excluded studies	3	3	3	1	3
6. Characteristics of included studies	4	4	3	4	4
7. Quality assessment of included studies	1	1	1	4	4
8. Use of study quality in formulating conclusions	1	1	1	2	2
9. Methods for combining findings	4	4	4	4	4
10. Assessment of publication bias	3	1	1	4	4
11. Conflict of interest	3	3	3	3	3

Note. LD = learning disabilities; ADHD = attention-deficit/hyperactivity disability; R-AMSTAR = Revised Assessment of Multiple Systematic Reviews; PND = percentage of nonoverlapping data. BC-SMD = between-case standardized mean difference.

a

Multi-level models with robust variance estimation (RVE) were used. ^b Lacked assessment of publication bias. ^c Results showed potential publication bias.

Results of the quality appraisal revealed the included meta-analyses exhibited medium to high methodological quality, with a mean score of 33.75, a median of 34.0, a range of 27–40, and a standard deviation of 3.83. None of the studies were considered insufficient or low. More recent studies (e.g., Lei et al., 2020; Myers et al., 2022; Shin et al., 2021) tended to have higher quality ratings than older ones, suggesting a possible improvement in the methodological quality of meta-analyses over time. Strengths across the meta-analyses included comprehensive literature searches and appropriate outcome syntheses. However, areas for improvement were often related to detailed reporting of study characteristics and publication bias assessment. The distribution of quality ratings (seven medium and six high) bolsters our confidence in the synthesized results.

Overlap in Primary Studies Across the Included Meta-Analyses

The overlap analysis revealed substantial overlap in the included research (Cohen’s Kappa: 15% for GDR and 11% for SSR), suggesting the reviews relied heavily on the same primary studies. Visual inspections of the citation matrices (see Figures 2 and 3) confirmed this overlap. This high degree of overlap in primary studies validates our choice of a qualitative synthesis approach for assessing the quality of the meta-analyses and generating informative qualitative summaries (Gates et al., 2020). However, due to this high degree of overlap, careful interpretation of the findings from our qualitative synthesis remains crucial.

OPEN IN VIEWER

Figure 2.

Citation Matrix of the Meta-Analyses of Group Design Research (GDR).

OPEN IN VIEWER

Figure 3.

Citation Matrix of Meta-Analyses of Single Subject Research (SSR).

Qualitative Summaries of Included Meta-Analyses

We present a qualitative summary of each meta-analysis chronologically for easy reference. For reviews reporting results with and without outliers, we focused on the results excluding outliers to minimize bias and obtain more reliable ESs. These qualitative summaries are complemented by the detailed information in Tables 4–6. These tables provide a comprehensive overview of the included meta-analyses, detailing study design characteristics, quality assessment, and the results of moderator analyses. We first summarize the GDR reviews, followed by the SSR reviews.

OPEN IN VIEWER

Table 6.

Summary and Results of Comparative Analysis of Moderator Effects Across Meta-Analyses.

Result	Group design research (GDR)						Single-subject research (SSR)
	Xin & Jitendra (1999)	Zhang & Xin (2012)	Zheng et al. (2013)	Lein et al. (2020)	Kong et al. (2021)	Myers et al. (2022)	Xin & Jitendra (1999)	Zhang & Xin (2012)	Zheng et al. (2013)	Lei et al. (2020)	Shin et al. (2021)
Intervention Characteristics (n = 16)
CCSS-M Content Standards a											Y
CCSS-M Practice Standards a											Y
Content Domain a		N		N		N		N		Y
Culturally Responsive Practices a										N
Dependent Measure Type b		Y		Y	Y	Y		N
Fidelity b						N
Group Size a	Y			N	Y	Y				N
Instructional Setting a	N	Y				N	N	N
Intervention Duration a				N	Y	N				N
Intervention Frequency a						Y
Interventionist b	Y			Y	Y	Y	N			Y
Intervention Model a	Y	Y		Y		N	Y	N
Number of Treatment Sessions a	Y				Y	N	Y
Student-Directed Learning a	N
Subject Area Focus a										Y
Word Problem Task/Type a	Y	N				N	N	N
Participant Characteristics (n = 8)
Academic Risk Area a			Y			Y			Y
Broad Math Achievement a			N						N
LD Diagnostic Approachb		N						N
English Learner (EL) Status a					Y					N
Gender b										N
Grade Level b	Y		N	Y	Y	N	N		N	Y
IQ b	Y		N				N		N
Participant MD Status a	Y	N		N	Y	N	N	N		Y

Note. Y = Significant moderator effect reported (p < .05); N = non-significant moderator effect reported (p > .05); CCSS-M = Common Core State Standards for Mathematics; LD = learning disabilities; MD = mathematics difficulties. Variables in bold emerged as consistent moderators with uniform results.

a

Connections were established between these variables and at least one dimension of the Taxonomy of Intervention Intensity (TII). ^b No connections were established between these variables and any of the TII dimensions

Findings of Umbrella Analysis: Consistent Moderators of WPS Interventions

Table 6 summarizes the moderating variables examined across the reviews and their findings. Among 24 unique variables investigated across the reviews, 16 were intervention-related, and eight were participant-related. Findings showed 17 variables (13 intervention and 4 participant characteristics) directly mapped onto at least one TII dimension, highlighting their potential for intensifying and tailoring WPS interventions for students with MD. The remaining seven (gender, dependent measure type, grade level, LD diagnostic approach, IQ scores, fidelity, and interventionist) fell outside the scope of the TII dimensions. Notably, the type of dependent measure produced consistent results, with several studies reporting higher effects for researcher-based measures than for standardized assessments.

Our analysis revealed that most of the 17 potential moderators correlated with the TII framework did not significantly influence WPS outcomes. Factors like word problem type and content focus lacked consistent effects across studies. In addition, moderators with limited coverage, like EL status and broad math achievement (examined by only one or two studies), made it difficult to draw definitive conclusions about their moderating role. Inconsistencies also emerged for other moderators, such as participant MD status, where some studies showed significant effects while others did not.

We identified four variables that consistently moderated WPS outcomes: intervention model, academic risk area, number of treatment sessions, and group size. Notably, the academic risk area represents the sole participant characteristic among these moderators. The remaining three variables—intervention model, number of treatment sessions, and group size—pertain to intervention characteristics. The ESs associated with these moderators were generally substantial and positive, suggesting these factors hold substantial educational significance for improving the effectiveness of WPS interventions for students with MD (Bloom et al., 2008). We summarize the results of each of these moderators in the preceding sections.

Consistent Moderators of WPS Intervention Outcomes

Our analysis revealed that four intervention and participant characteristics related to various dimensions of the TII framework consistently moderated the outcomes of WPS interventions for students with MD. These variables included the intervention model, which was connected with the TII components of comprehensiveness and attention to transfer. The academic risk area (i.e., MD vs. MD and RD) was linked to the alignment dimension. The number of treatment sessions served as an indicator of dosage, while group size was connected with multiple TII components, including dosage, attention to transfer, behavioral support, and individualization. These findings suggest that the efficacy of WPS interventions consistently varied across the meta-analyses based on these four variables. Therefore, they are likely essential considerations for intensifying WPS instruction for students with MD. We can consistently expect changes in intervention outcomes when these variables are manipulated.

Limitations

We note several limitations in our study. First, the small sample size of meta-analytic studies (n = 11) used in our analysis may limit the conclusiveness of its findings. This lack of research underscores the need for more primary research on EBPs promoting WPS outcomes among MD students. Second, we did not empirically synthesize findings across the meta-analyses, preventing us from making clear inferences about the exact mean magnitude of ESs across studies. Variations in the statistical and clinical designs across the studies examined precluded our ability to empirically summarize the literature via a meta-analysis of the meta-analyses (McKenzie & Brennan, 2022). Third, we only included peer-reviewed meta-analyses, excluding unpublished reviews such as dissertations, which may introduce publication bias. Fourth, our use of a modified tool (R-AMSTAR) for assessing methodological quality, while a common practice in umbrella reviews, may introduce limitations. Social science research can differ from healthcare research in its approach to evidence and inquiry. Ideally, a tool specifically designed for evaluating the quality of social science meta-analyses might capture these nuances more effectively. A fifth limitation of our study is the assessment of heterogeneity. Most meta-analyses we reviewed reported the Q statistic primarily, with few providing I² or τ² values, restricting our ability to infer the magnitude and distribution of heterogeneity. The Q statistic indicates significant heterogeneity but lacks detail on its extent or nature. Metrics such as I² and τ² are more appropriate for a substantive discussion on heterogeneity (Higgins & Thompson, 2002). The absence of these metrics restricted our comprehensive assessment and comparison of heterogeneity across studies. Finally, we note potential biases arising from overlapping primary studies in the meta-analyses we synthesized deserve consideration. While such overlap can introduce biases and threaten the validity of findings, it is often inevitable and can even have advantages (Hennessy & Johnson, 2020). In this case, the overlap suggests our meta-analyses drew from a consistent pool of conceptually similar studies, potentially leading to more reliable evidence (Hennessy & Johnson, 2020). While potential biases remain, our findings offer valuable insights into crucial moderators for intensifying WPS interventions, providing essential guidance for future research and practice.

Implications for Practice

Our study has five notable implications for using the TII to tailor WPS interventions for students with MD within MTSS. First, practitioners may find it beneficial to adopt SBI as an effective instructional strategy for WPS interventions due to its association with critical dimensions of intensive instruction, including comprehensiveness, behavioral support, and attention to transfer. SBI’s multi-step intentional approach to solving word problems offers a more comprehensive framework than other models, such as strategy instruction (Jitendra et al., 2021). It encourages students to use metacognitive strategies, such as self-instruction and self-monitoring, which are crucial for promoting behavioral regulation and fostering the transferability of students’ WPS skills to various problem types (Powell & Fuchs, 2018). Second, our findings demonstrate consistently larger ESs for researcher-developed WPS measures than standardized assessments, suggesting potential implications for data-collection practices in progress monitoring and intervention evaluation and intensification for students with MD. We urge practitioners to exercise caution when selecting measures for progress monitoring. Researcher-made assessments may inflate ESs, while standardized ones may underestimate them (Scammacca et al., 2007). Both measures may improve practitioners’ understanding of student WPS progress and inform targeted instructional adjustments.

Third, we deduced that students with both MD and RD exhibited lower WPS outcomes than their peers with MD only. Therefore, it is crucial to anticipate additional WPS challenges among this student group and plan proactively to address these challenges. We recommend that practitioners use pretreatment performance data to identify students with MD who demonstrate language difficulty and RD and embed language and reading supports within WPS interventions targeting these students (Arsenault & Powell, 2022b). Also, we urge practitioners to incorporate strategies, such as paraphrasing and explicit instruction in comprehension and vocabulary, to help students with RD overcome reading challenges, ultimately advancing their WPS abilities (Powell et al., 2019). Fourth, our results for the number of intervention sessions suggest that practitioners should consider extending WPS interventions for students with MD. This extended instruction allows for more in-depth teaching, personalized practice, and targeted feedback tailored to students’ needs (Powell, Benz, et al., 2022). Finally, our null findings for important intervention characteristics, including setting, word problem type or task, and content domain, suggest that the WPS interventions we analyzed may be effectively implemented in various locations, including general or special education settings, across diverse content domains, and using multiple word problem types, without compromising intervention effectiveness. These findings highlight the potential flexibility and adaptability of WPS interventions for students with MD, enabling teachers to tailor instruction to meet individual student needs.

Implications for Research

The results of our umbrella review highlight several critical areas for future research on WPS interventions for students with MD. First, the inconsistencies in our results suggest the need for a more comprehensive approach to identifying gaps in the existing research. Evidence mapping, a visual tool for identifying areas where research is lacking, could be instrumental in exploring these gaps and advancing the field (Saran & White, 2018). Second, the limited sample sizes in previous meta-analyses necessitate more research. This research should include primary studies that involve groups of participants (e.g., GRD designs) and those focused on a single participant over time (SSR designs) on WPS interventions for students with MD. Building a richer research base will enable researchers to conduct more robust meta-analyses and ultimately identify factors influencing these interventions’ outcomes.

Third, future primary research incorporating designs that directly manipulate features of WPS interventions (e.g., number of sessions) may be necessary to understand how these features impact student outcomes. However, researchers must prioritize student well-being and adhere to ethical guidelines when designing interventions. Fourth, future research should consistently report suitable metrics to enable more robust evaluations of variability in intervention effects. As noted, indices such as I² and τ² provide more accurate assessments of the variation in results between studies, which facilitates better comparisons across studies. Fifth, developing and validating a tool for more precise and relevant evaluations of methodological quality in social science meta-analyses is crucial. This instrument would enhance the reliability of findings and provide more accurate guidance for educational interventions. Finally, given the substantial variation observed in intervention effects, future research should explore how multiple variables interact and influence WPS outcomes. Meta-analytic designs using machine learning algorithms can be beneficial in identifying potential synergistic effects among variables that traditional methods might miss (Williams et al., 2022).