PROTOCOL: The relationship between homework time and academic performance among K-12 students: A systematic review

Types of studies

We will include treatment-control group design or a comparison group design studies, to adequately address the effect of differing homework time on the academic performance of K-12 students.

Randomized controlled trials (RCTs) that aim to explore the effect of homework time on academic performance by comparing the test score differences between different groups before and after the intervention will be included. In addition, non-RCTs such as cohort studies (NRCTs), controlled before and after studies, and interrupted time series studies also will be included if they explicitly take homework as intervention, and report the time spent on homework and the mean and standard deviation of academic achievement.

Relevant correlational studies without mean and standard deviation of academic achievement will be excluded. Other study designs such as case studies, narrative reviews, and nonprimary studies such as editorials, book reviews, commentaries, and letters to the editor, will also be excluded. Qualitative evidence is also beyond the inclusion criteria in the present systematic review.

Types of participants

This review will include studies of K-12 school students. We excluded children with disabilities, since the context and effects may be different for that population group. Students from special education schools are excluded. If the primary study includes mixed samples (e.g., special education and nonspecial education students), we will use the sub sample excluding special needs students if reported.

Types of interventions

In this review, we will explore the relationship between homework time and academic performance by comparing the academic scores with different amounts of time spent on homework. The eligible intervention studies must be clear that the intervention is homework assigned to students to complete during nonschool hours regularly by schoolteachers which aims to improve academic achievement. This does not mean that the intervention must consist of academic activities, but rather that the explicit expectation must be that the homework, regardless of the nature of the homework content, will result in improved academic performance. Furthermore, we will only include school-based interventions, that is, homework allocated by other people such as parents or teachers from extracurricular schools, study clubs, and extracurricular activities such as sports and participation in clubs are excluded. Homework related to psychotherapy will also be excluded.

The comparison condition is different time spend on the homework, and we plan to divide the comparisons into several groups, such as 0–15, 16–30, 31–45, 46–60, 61–90, 90–120 min, and more than 120 min.

Types of outcome measures

The objective of the review is to explore the impact of homework on students' academic outcomes. We will extract the homework time and academic performance provided in the primary study. The homework time is the exact time or a time frame reported by students or parents. Academic performance will be measured by the teacher, exam results and/or by the research team using any valid standardized test and reported as test scores.

As valid standardized tests, we will consider norm-referenced tests (e.g., Gates-MacGinitie Reading Tests and Star Math), state-wide tests (e.g., Iowa Test of Basic Skills), national tests (e.g., National Assessment of Educational Progress). If it is not clear from the description of outcome measures in the studies, we will use electronic sources to determine whether a test is standardized or not.

Primary outcomes

The primary outcome is academic performance (test score and standard deviation), and studies that have measured academic performance (and homework time) will be included.

Secondary outcomes

Academic motivation and quality of homework will be included as secondary outcomes.

Electronic searches

The following databases will be searched from inception to present:

Social Sciences Citation Index (Web of Science)

Below, the search strategy for Web of Science is provided:

#1 TI = homework OR AB = homework

#2 TI = home-work OR AB = home-work

#3 #1 OR #2

#4 TS = K-12 OR TS = preschool student* OR TS = pre-school student* OR TS = Kindergarten

student* OR TS = middle school student* OR TS = high school student* OR TS = senior school

student* OR TS = primary school student* OR TS = pupil OR TS = schoolchild OR TS = junior

high school student* OR TS = school-age

#5 TS = achievement OR TS = performance OR TS = grade OR TS = score OR TS = academic

achievement* OR TS = GPA OR TS = academic performance

#6 #3 AND #4 AND #5

Searching other resources

We will consult the following sources of gray literature, and search the websites of organizations devoted to the education research, to identify relevant unpublished studies and reports. The following gray literature resources will be searched with the keyword “homework”:

What Works Clearinghouse (https://ies.ed.gov/ncee/wwc/)

We will also search the Google Scholar with the keyword “homework,” and we will stop scan if there are 5 consecutive pages with no relevant studies.

The following international journals will be hand searched for relevant studies with the keyword “homework”:

American Educational Research Journal

Additionally, the primary studies included in the previous systematic reviews on the relationship between homework and academic performance will be scanned, and the reference lists will also be searched. Furthermore, the studies of experts in the research of homework (such as Cooper Harris, Trautwein Ulric, and Xu jianzhong) will be searched systematically to check our search strategy, and they will be contacted to help identify other relevant studies if possible.

Selection of studies

The selection of studies will be performed independently by the first two reviewers (Guo LP and Jieyun Li) in Rayyan (https://rayyan.qcri.org/). All titles and abstracts of the records identified after retrieval will be screened, the potentially relevant references will be located with full-text, and the primary studies that meet our criteria will be included for further analysis. Studies that meet the selection criteria and have the outcomes of interest measured, but do not report these outcome data, will be included and described in the results section of the review. Any discrepancies between the two reviewers will be resolved by consensus with another reviewer involved (Kehu Yang). The whole process of study screening will be based on the PRISMA guidelines (Moher et al., 2009).

Data extraction and management

Information extraction and coding will consist of two parts. The first is general information, including the information of primary study (publication, the year of publication, and the year of data collection), sample characteristics (e.g., sample size, gender, grade level, region, family economic status, parental education level), methodological characteristics (e.g., sampling method, measures of homework time, and the measure of academic performance), and the intervention characteristics (e.g., subject, mode of homework and the type of homework). The other is the effect size, including the homework time and the test score (The details are shown in Supporting Information Annex 1). This process will be conducted independently by two authors (Zheng Xu and Xing Xin), and disagreements between coders will be resolved by discussions with another author (Xiuxia Li).

If a study contains multiple interventions (e.g., different homework modalities such as online vs. book-based), the reviewers will only extract data that are eligible for this review.

In case of controlled before and after studies, mean or median change from baseline scores will be extracted or computed by the reviewers if all necessary data are available. If change scores are not available or cannot be computed, post-intervention values will be extracted by the reviewers.

Assessment of risk of bias in included studies

For RCTs, the Cochrane bias risk tool will be used to assess the quality of the method and potential defects (Higgins & Green, 2011). For nonrandomized studies (including cohort studies, controlled before and after studies, and interrupted time series studies), the risk of bias in nonrandomized studies of interventions (ROBINS-I) will be used to check the quality of the individual study (Sterne et al., 2016). In addition, the grade will be used to rate the overall quality of the evidence included in this review, ranging from high, moderate, low, and very low, based on the assessment to study design, imprecision, inconsistency, indirectness, and publication bias (Atkins et al., 2004; Schünemann et al., 2013). The risk of bias assessment will be conducted by the two authors (Zheng Xu and Xing Xin), and any disaccord will be solved by discussion with another author (Xiuxia Li).

Dealing with missing data

If there are any missing data, we will contact the author at least twice to obtain more information if the correspondence address is available. If these data are unavailable, we will only analyze the available data, and the studies with missing data will be described in the Results section. Besides, the potential impact of missing data on comprehensive estimates will be considered in the Discussion section.

Assessment of heterogeneity

Forest plots will be inspected to visually investigate overlaps in the confidence intervals (CIs) of the results of the individual studies. The χ ² test will be performed, and the Q statistics, I ² and τ ² index will be adopted to evaluate heterogeneity across studies. For Q statistics, a p value of .05 will be used as a threshold for statistical significance. The I ² index refers to the truly observed variation ratio (Borenstein et al., 2009), and 25%, 50%, and 75% of the I ² indicate low, medium, and high heterogeneity (Higgins & Thompson, 2002). And the parameter τ ² is the between-studies variance (the variance of the effect size parameters across the population of studies), that is, the variance of true effect sizes (across an infinite number of studies).

Assessment of reporting biases

Reporting bias, also called publication bias, refers to the potential of the studies with statistically significant findings to be accepted for publication, whilst those with statistically nonsignificant findings would hardly be accepted for publication. These studies would have a higher probability of being left in the “file drawer,” according to the so-called “file drawer problem.” If 10 or more studies were identified, visual funnel plots and Egger's test of funnel plot symmetry were performed to evaluate potential publication bias (Rosenthal, 1979). If there is evidence of funnel plot asymmetry from a test, we will attempt to conduct the comparisons between the effect with that after trim and fill. The possible reasons for this (e.g., nonreporting biases, poor methodological quality leading to spuriously inflated effects in smaller studies, true heterogeneity, artefactual, and chance) will also be considered (Page et al., 2019).

Data synthesis

We will include all intervention studies that meet our inclusion criteria and extract the mean (M) and SD of test scores. We will take the standard mean difference (SMD) and CI as our effect size index. For subgroups, we plan to divide the outcomes into several groups by time spent on the homework, such as 0–15, 16–30, 31–45, 46–60, 61–90, 90–120 min, and more than 120 min, and then compare the difference of SMD between groups to explore the role of homework time on academic achievement.

If two or more studies are identified that have investigated the effect of homework time on academic performance with sufficiently available data, a random-effects meta-analysis will be performed to estimate the comprehensive effect due to the expected heterogeneity between groups using the Review Manager 5 software. The pooled estimates will be presented in forest plots. If quantitative synthesis is not suitable, narrative synthesis will be adopted.

Planned moderators

If statistically significant heterogeneity is detected, subgroup analyses with stratification analysis will be conducted to explore the source of heterogeneity based on the available data.

Gender. Previous studies showed that girls more frequently reported managing their homework than boys (Mau & Lynn, 2000; Xu, 2007), and zero-time homework students are most often male (Hagborg, 1991). Therefore, it is worth identifying the influence of gender on homework time and academic performance.

Sensitivity analysis

In addition to the implicit sensitivity analysis in both the analysis of heterogeneity and subgroup analysis, the “One-leave-out” method is adopted for sensitivity analysis to check for outliers that potentially influence the overall results, and test the robustness of the meta-analysis.