Protocol for a systematic review: 21st century adaptive teaching and individualized learning operationalized as specific blends of student-centered instructional events: a systematic review and meta-analysis

Abstract

Background

The problem, condition or issue

Conceptual understanding of individualized learning and adaptive teaching varies broadly, encompassing a multitude of instructional strategies, approaches, and activities. It stretches from accounts as narrow and specific as scaffolding adaptive feedback in computer-based instruction (e.g., Atkinson, Renkl & Merrill, 2003) to more general conceptions, such as cooperative and collaborative learning (e.g., Johnson & Johnson, 2002). It also includes educational concepts derived from elements of constructivism, such as discovery learning, inquiry-based learning, experiential learning, problem-based learning and other forms of student-centered education. These instructional forms, which have been broadly described from the 1960s (e.g., Summerhill), were recently criticized by Kirshner, Sweller & Clark (2006) with a response published by Tobias & Duffy (2009).

The learning sciences have further contributed to the distinction between social constructivism and individual constructivism (i.e., instructional system designs) providing a theoretical grounding for teacher vs. learner-based strategies (Kolodner, 2004). Current and developing applications, informed by pedagogical principles espoused by case-based learning (e.g., Kolodner et al., 2008), exemplify the transformation of learning environments which apply Bruner's concept of not just discovery for the student, but co-discovery on the part of the teacher.

Examples of various attempts to make teaching and learning more adaptive can be found in both the early and current research literature. They include, though not limited to, mastery learning (e.g., Bloom, 1968), Personalized Systems of Instruction or PSI (e.g., Keller, 1968; Gifford & Vicks, 1982; Davies, 1981), assorted forms of peer instruction (e.g., Mazur, 1997), various reciprocal reading/writing activities (e.g., Huang & Yang, 2015; MacArthur, Schwartz & Graham, 1991), adaptive hypermedia (Brusilovsky, 2001), accommodation for individual learning styles (e.g., Özyurt & Özyurt, 2015) and more recent Intelligent Tutoring Systems or ITS (e.g., Huang & Shiu, 2012; VanLehn, 2011). To some extent, findings of primary research on these and related instructional practices have been summarized in two rather sparse collections of meta-analyses separated in time by almost three decades.

In the late 1970s and early 1980s, several relevant meta-analyses were published. First, Lysakowski and Walberg (1982), Gusky and Gates (1986), Slavin (1986) and Kulik, Kulik and Bangert-Drowns (1990) each performed successive meta-analyses (Slavin's was a best evidence synthesis) on the efficacy of mastery learning. The studies produced equivocal findings. Also, Kulik, Kulik & Cohen (1979) reviewed 75 individual comparative studies of Keller's PSI (a spin-off of mastery learning) college teaching method. In comparison to conventional instruction the PSI was demonstrated to have a positive effect on student achievement and course perception (mean effect size of nearly 0.70 for both). Aiello and Wolfle (1980) summarized research on individualized instruction in science compared with traditional lectures and found that individualized instruction was more effective. Horak's (1981) study of self-paced modular instruction of elementary and secondary school math produced a wide variety of both positive and negative effect sizes. Bangert and Kulik (1982) looked at the effectiveness of the Individualized Systems of Instruction (ISI) in secondary school students. They broadened the list of outcomes to account not only for student achievement (e.g., final exam), but also critical thinking, attitudes toward subject matter, and student self-concept. For all outcome types the findings were unsettled. For example, for the achievement data only eight out of 49 studies demonstrated statistically significant results in favour of ISI (four studies favoured more conventional teaching methods and the rest were inconclusive). Finally in 1984, Kulik attempted a wider research synthesis (encompassing over 500 individual studies) of effectiveness of programmed instruction and ISI, paying special attention to the moderator variables of study dates and grade levels. Among the most promising findings, the author indicated that more recent studies showed higher effects than the earlier and that college-level students benefited significantly from using ISI compared with elementary and secondary school students. In summary, these meta-analyses produced inconclusive results. Moreover, they are rather outdated – practically none of the above-mentioned instructional methods exists now in its original form (e.g., Eyre, 2008 was able to identify fewer then 50 studies of PSI for the period between 1990 and 2006 in the PsycInfo database). All this suggests the need for a more refined (both methodologically and substantively) update of systematic reviews in the field, especially taking into account how much the methodology of meta-analysis itself has evolved since then.

Several meta-analyses addressed the topic of individualized instruction, though in very specific narrowly focused forms. Cole (2014) examined the effectiveness of cooperative, collaborative, and peer tutoring for English language learners. A low-to-moderate average effect size of

\bar{g}

= 0.49 was found in favour of peer tutoring over individualized or teacher-centred comparison instructional conditions. The effect size tended to be relatively small in middle school students, but higher at elementary and high school levels. More in line with the already mentioned earlier meta-analyses of various forms of computer-assisted instruction, Ma, Adesope, Nesbit and Liu (2014) meta-analysed studies of Intelligent Tutoring Systems (ITS) in a variety of subject matters, from reading and math to law and medical education. The list of moderator variables included the type of both experimental and comparison treatments, as well as outcome type, student academic level, discipline studies, etc. The highest achievement effects of using ITS were found in comparison with non-ITS computer-based instruction (

\bar{g}

= 0.57) and teacher-centred, large-group instruction (

\bar{g}

= 0.42), whereas in comparison with human tutoring it was even negative (

\bar{g}

= -0.11), though not statistically significant. ITS-based practices were similarly effective when used either alone or in combination with various forms of teacher-led instruction in many subject domains.

In summary, research evidence concerning the effects of adaptive teaching and individualized learning remain relatively inconclusive, while there is obviously a need for better understanding how K-12 formal education may be more successful in addressing students’ personal needs and interests, accounting for their diverse abilities with the main goal of advancing their learning. The proposed systematic review will not only rely on the most rigorous and comprehensive methodology of meta-analysis, but also should be conceptually sound (i.e., thoroughly exploring educational practices to find consistent links among a multitude of individual pedagogical approaches) and timely (i.e., account for the most recent developments in education).

The intervention

The main research question of the proposed meta-analysis is:

Can more Student-Centred (SC) (i.e., more adaptive and individualized) approaches to K-12 instruction be distinguished from more Teacher-Centred (TC) approaches in terms of their effect on student achievements and what substantive and demographic factors moderate these effects?

For better understanding and more successful practical application, educational practices subsumed under this generic pedagogical idea of adaptive teaching and individualized learning deserve a valid conceptual working model, both inclusive enough to account for various forms of personalized/individualized instruction and sufficiently sensitive to fluctuations due not only to the influence of numerous moderator variables, but also to nuanced qualities of particular instructional approaches themselves. SC instructional strategies could, in our view, serve such an overarching conceptual framework with adequate explanatory power, but only if operationalized properly to avoid an oversimplified dichotomy of inductive vs. deductive education (constructivism vs. direct instruction).

Indeed, we are less interested in deciding between these two extremes and more interested in understanding the circumstances or combination of circumstances that optimize teaching and learning. Gresalfi and Lester (2009) for mathematics teaching, and Klahr (2009) for science teaching argue that the goal of instruction should be to achieve curricular and process objectives by choosing the most appropriate method based on student age, ability, prior knowledge, level of content, etc. In this regard, we would like to avoid the conceptual error of falsely dichotomizing pedagogical environments as either TC or SC since neither instructional practice likely exists in its pure form. As Gersten et al. (2008) observed in their systematic review of mathematics teaching practices: “[We] found no examples of studies in which students were teaching themselves or each other without any teacher guidance; nor did the Task Group find studies in which teachers conveyed … content directly to students without any attention to their understanding or response. The fact that these terms, in practice, are neither clearly nor uniformly defined, nor are they true opposites, complicates the challenge of providing a review and synthesis of the literature …” (p. 12). Since SC pedagogical practices try to emphasize guidance over direct instruction, the question becomes how much and what kind of guidance is offered to students and who takes responsibility for the design and implementation of various components of learning experience to make them truly adaptive/individualized and, hence, more effective.

How the intervention might work

To define the key quality of instruction as “adaptive” and “individualized,” for the purposes of the proposed systematic review, we suggest deconstructing teaching and learning according to the events associated with them (e.g., setting objectives, implementing instructional methods, assessing learning). Accordingly, a more SC (more adaptive) classroom is one in which students play a more central role in the conduct of the instructional events. If these events can be isolated in reports of primary classroom research, they can be rated individually on a TC to SC continuum. Each event could then be: 1) examined separately to determine their individual strengths; 2) examined in clusters as combinations of events; or 3) collapsed into a multi-dimensional composite that would yield a “greater-than to lesser-than” distinction between two different instructional settings. This approach avoids problems associated with either subjectively defining instructional conditions as SC vs. TC or vaguely labeling them, such as PSI, mastery learning, etc. It also has the advantage of allowing us to examine instructional events in isolation and in various combinations in the search for optimal instructional practices.

Why it is important to do the review

Most of the significant effects from the meta-analyses described in the first section of this protocol on the topic cluster around

\bar{d}

0.40SD, but the data also reflect a wide range of effects, depending on the whole spectrum of moderator variables. In other words, the picture painted by these meta-analyses remains in large part as inconclusive as it was several decades ago in the 1980s. Of special concern to us is the fact that both earlier and recent meta-analyses are rather limited in scope and focus of interest, addressing very specific instructional practices and technological tools. There were no serious attempts to find and conceptualize pedagogical commonalities among the interventions in question that would allow treating them within the same class of phenomena broadly depicted as individualized learning and adaptive teaching. Thus, the need for a review that would be broad in scope, summarize research evidence up to date, and have a conceptually sound foundation is pressing.

OBJECTIVES

The main objective of the proposed review is to summarize research data on the effectiveness (in terms of learning achievement outcomes) of adaptive and individualized instructional interventions operationally defined here as more SC pedagogical approaches. The overall weighted average effect size will be an indication of that. Additionally, and no less important, the review aims to better understand under what circumstances (e.g., with what populations of learners, for what subject matters) the effects of adaptive and individualized instruction rich their highest potential, and what conditions may depress them. To explore the latter, a set of substantive and demographic study features will be coded and subjected to moderator variable analyses.

The review outcomes will inform education practitioners and research community of the best instructional practices, preconditions for their successful implementation and potential pitfalls, as well as of directions for further empirical research in the area.

METHODOLOGY

Criteria for including and excluding studies

Types of study designs

The review will include studies that are experimental (i.e., RCT) or high-quality quasi-experimental (i.e., statistically verified group equivalence or adjustment) in design that address adequate to the research question group comparisons, contain legitimate measures of academic achievement (i.e. teacher-made, standardized), and report sufficient statistical information for effect size extraction.

Types of participants

Students in K-12 formal educational settings (approximate age 5-18), i.e., eventually leading to a certificate, diploma, degree, or promotion to a higher level. Educational interventions may take place either in the classroom (F2F), via distance education (DE), or as a blended (various combinations of F2F and DE) intervention.

Types of interventions

As depicted earlier, the intervention in question (an experimental condition) will be considered any combination of instructional events that is higher in SC qualities than a comparison (control) condition. Student participation in decisions about (control over) selection of study materials and learning activities, pacing of instruction evaluation of performance of peers (including self-evaluation), as well as any for of adapting learning for students’ individual needs, interests, backgrounds, etc., constitute, in our view, such SC qualities of instruction. Two independent reviewers will code instructional conditions featured in a given primary study (on a scale from 1 to 5, where “1” reflects instructional events being entirely under control of the teacher/program and “5” depicts maximum possible involvement of students) to reflect to what extent each group possesses these qualities. The following dimensions will be in the focus of coding: •

Selection/design of instructional activities;

•

Choice/development of study materials;

•

Formulation of learning objectives;

•

Determination of pacing of instruction (i.e., flexibility in time and learning trajectories);

•

Performance evaluation;

•

Distribution of roles between students and teacher (authority, resource, mentor, guide, facilitator, co-investigator, etc.);

•

Adaptation of materials, activities, and other aspects of instruction (e.g., learning group formation) to individual students.

The highest total rating across these dimensions will determine an Intervention condition to be compared on achievement outcomes to a Comparison condition, lowest in total rating That is differential total across all dimensions, as some of them may be higher in ratings for one group and some – for another. At the same time, keeping individual positive and negative differentials will allow identifying consistent clusters of instructional events that are more/less likely to work to the advantage of students’ learning.

Types of outcome measures

All types of objective measures of academic achievements are to be considered. Their psychometric features (e.g., standardized, non-standardized teacher/researcher-made assessment tools) and type of representativeness (e.g., cumulative final examinations or averages of several performance tasks covering various components of the course/unit content) will be documented and used in subsequent moderator variable analyses. Self-assessments are to be excluded, as well as attitudinal and behavioural measures. Data of their prevalence in the reviewed primary literature will be collected to inform further reviews in the area with a potential focus on those types of outcomes.

Duration of follow-up

To maximize coverage of primary research, fully compatible in terms of outcome measures, only immediate post-test (that is assessment administered at the end of treatment implementation) results will be considered. Various forms of delayed post-tests will be documented and their time lags categorized to inform further reviews.

Types of settings

K-12 formal educational settings (approximate age 5-18), in educational programs eventually leading to a certificate, diploma, degree, or advancement to the next academic level/grade are of interest to the current meta-analysis. Other settings (i.e., home schooling, auxiliary programs, summer camps, etc.) are to be excluded.

In summary, the overall set of inclusion/exclusion criteria for the proposed meta-analysis contains the following conditions. To be retained for the analyses any primary research study should: •

Be publicly available (or archived) and encompass studies from 2000 to the present;

•

Feature at least two groups of different instructional strategies/practices that can be compared according to the research question as SC and TC instruction;

•

Include course content and outcome measures that are compatible in the groups that form these comparisons;

•

Contain sufficient descriptions of major instructional events in both instructional conditions;

•

Satisfy requirements of either experimental or high-quality quasi-experimental design;

•

Be conducted in formal K-12 educational settings eventually leading to a certificate, diploma, degree, or promotion to a higher level;

•

Contain legitimate measures of academic achievement (i.e., teacher-made, standardized); and

•

Contain sufficient statistical information for effect size extraction.

Search strategy

Addressing studies from 2000 onward seems to strike a reasonable balance of covering various approaches to individualized/adaptive learning prominent throughout several decades (studies published in early 2000s would still reflect most interesting pedagogies of 1990s), while primarily focusing on those that retain relevance in most recent educational practices. In order to retrieve a broad base of studies to review, we will begin by having an experienced Information Specialist search across an array of bibliographic databases, both in the subject area and in related disciplines. The following databases will be targeted:

ABI/Inform Global (ProQuest)

Academic Search Complete (EBSCO)

ERIC (EBSCO)

PsycINFO (EBSCO)

CBCA Education (ProQuest)

Australian Education Index

British Education Index

Education Source (EBSCO)

Web of Knowledge

Scopus

Engineering Village

Francis (EBSCO)

Medline

ProQuest Dissertations & Theses Global

ProQuest Education Database

Linguistics and Language Behavior Abstracts (ProQuest)

Database searching will be supplemented by searches of the Web using Google and Bing to locate additional articles, but also grey literature (research reports, conference papers, theses and research published outside the conventional journals). We will also search the OpenGrey.eu and the Learn Tech Lib online collections for grey literature, and will consult the Campbell guide (Hammerstrøm, Wade, & Jørgensen, 2010) for other useful online resources.

Finally, the reference lists of identified literature reviews will be ‘branched’ for additional relevant studies using a citation search approach. The most recent issues of the top journals (based on inclusion rate) will be searched manually toward the end of the search process to catch any recent publications that match our screening criteria. When possible, we will contact noted experts in the field to ensure we have all their relevant research.

Although the search strategy will be tailored to the features of the various databases, i.e. making use of database-specific controlled vocabulary and search filters, the following is representative of what the overall search statement would look like.

(“student cent*” OR “learner cent*” OR “learner control” OR constructivi* OR “individualized instruction” OR “discovery learning” OR “active learning” OR scaffold* OR “experiential learning” OR “teacher guid*” OR “self-direct*” OR “problem based learning” OR inquiry OR “humanistic education” OR “democratic education” OR “progressive education” OR “adaptive learning” OR “adaptive education” OR “adaptive class*” OR “adaptive teach*” OR differentiation)

AND

(“creative teaching” OR “instructional innovation” OR “instructional effectiveness” OR “teaching methods” OR “program effectiveness” OR “program evaluation”)

AND

(compar* OR contrast* OR “control group” OR experiment* OR “matched group*” OR quasiexperiment* OR posttest OR “post test” OR “comparative case study”)

Description of methods used in primary research

True experimental and quasi-experimental studies are to be included as far as they feature two educational interventions covering the same content (required knowledge acquisition and/or skill development) as assessed on compatible outcome measures, where one group (experimental) is higher in student-centred qualities (as described earlier) compared to the other (control) group. Reporting quantitative data sufficient for an effect size extraction is a necessary condition for study inclusion.

Criteria for determination of independent findings

There are several major threats to the independence of findings. These are: 1) repeated use of data coming from the same participants; 2) reporting multiple outcomes of the same type; and 3) aggregating outcomes of different types representing the same sample of participants (does not apply to the proposed review, as it is limited to learning achievement outcomes only). The means that we will use for ensuring data independence are presented in the Within Study Synthesis sub-section below.

Details of study coding categories

In addition to coding dimensions of SC pedagogical qualities that would determine proper comparisons for effect size extraction, the following groups of study coding categories will be used in the proposed review. First, study methodological quality will be assessed for features such as design type, fidelity of treatment implementation, attrition, and the unit of assignment/analysis (Cooper, Hedges, & Valentine, 2009). Within the same category, we will code for outcome source and psychometric quality of the assessment tools, as well as for the precision of procedures used for effect size extraction and for equivalence of instructor and study materials. Jointly, these methodological study features, used in moderator variable analyses will inform us of any potential threat to all types of study validity (Cooper et al., 2009).

Substantive study features will further clarify description of SC pedagogical qualities by specifying theoretical models underlying instructional practices under review, treatment duration, instructor's experience, provision of professional development for teachers and training for students, whenever is required by specific instructional intervention. Demographic study features will encompass learners’ age, educational background and ability level, as well as, subject matter studied. All these study features will be subsequently analysed as moderators for their potential impact on treatment effects.

All coding activities (i.e., abstract screening, full-text review, study features coding, as well as effect size extraction) will be carried out by two reviewers working independently and discussing and resolving disagreements, when necessary eliciting a third opinion of the project P.I. Reliability rates (initial judgement), i.e., Pearson's r and Cohen's kappa – for continuous and ordinal data, respectively, will be calculated and reported.

Statistical procedures and conventions

Effect Size Computation: Our review aims to include and summarize quantifiable achievement outcomes of primary empirical studies of high methodological quality that compare effectiveness of more SC (i.e., more adaptive and individualized) versus more TC (i.e., more conventional undifferentiated) instructional interventions. The following are the primary metrics and procedures that will be used for the effect sizes extraction and subsequent analyses.

Standardized Mean Difference Statistic (d-index)

For studies that report descriptive statistics for continuous measures of student achievement outcomes, the post-intervention mean of the control group will be subtracted from the post-intervention mean of the intervention group and the resulting difference will be divided by the pooled standard deviation of both groups (Cohen's d).

d = {\bar{X}}_{T r e a t m e n t} - {\bar{X}}_{C o n t r o l} / S D_{p o o l e d}

Outcome data are likely to be reported in a variety of formats. For studies that report inferential statistics such as t, F, or p-values only, the appropriate conversion formula will be applied to calculate the d-index as the effect size estimate (Lipsey & Wilson, 2001; Hedges, Shymansky, & Woodworth, 1989; Hedges & Olkin, 1985).

Sample size correction

To introduce proper corrections for the small sample size bias, all d-indices will be converted into the unbiased Hedge's g statistics using the following formula:

g ≅ (1 - \frac{3}{4 N - 9}) d

To test for statistical significance of calculated effect sizes we will use standard error of g, and the 95% confidence intervals. In all calculations, to aggregate effect sizes across studies and for moderator variable analyses, we will use Comprehensive Meta Analysis 3.0 (Borenstein, Hedges, Higgins, & Rothstein, 2005) software package.

Synthesis of Effect Sizes

Within Study Synthesis

This review will focus on learning achievement outcomes. There is a possibility, however, of finding several measures representing the same outcome type in the same study based on the same sample of participants. When it happens we will either decide in favour of the most representative measure – typically, cumulative final exams or post-tests or when no single outcome fully reflects learning performance throughout the unit of instruction – we will average effects deriving from different (complementary or equally representative) achievement measures. Also, the same sets of participants are never to be used to calculate multiple effect sizes of the same type. Whenever the same group of participants is used repeatedly (for example, when the same control group compared to two different treatment groups, each unique in its SC qualities – for the purposes of retaining as much of explanatory capacity of the review), its sample size will be reduced proportionally in order to avoid any overrepresentation (i.e., disproportionally high weighs of the respective effects) of the same participants in the final data set.

Across Study Synthesis

Independent effect sizes will be aggregated across studies using the random effects model (Borenstein, Hedges, Higgins, & Rothstein, 2010), as neither of the assumptions (conceptually grounded uniformity of interventions and access to the entire population of relevant studies) for applying the fixed effect model is met. Reflecting on the nature of our review as a random (though comprehensive) sampling of various populations of empirical studies in education, the random effects model provides a more accurate measure of the treatment effectiveness. The results obtained from a random effects model analysis will represent the overall effect of a diverse collection of SC instructional intervention on student learning across age groups, subject matters, etc.

Fixed effect model will be used to assess heterogeneity of the distribution of effect sizes in which the Q- statistic and I ² to determine the collective extent to which studies deviate from the fixed effect average for the collection. Also the I ², a statistic derived from Q, indicates the proportion of true heterogeneity (i.e., variability exceeding what would be expected based on the sampling error estimate) associated with each distribution of effect sizes. If the observed heterogeneity is above sampling error, coded study features, as potential sources of systematic variation, will be further explored through moderator variable analysis under the mixed effects model.

Sensitivity Analysis

It is necessary to assess potential bias that may be associated with out-of-range individual calculated effect sizes and may potentially distort the overall interpretation of the findings. Sensitivity analysis (Hedges & Olkin, 1985) is intended to determine whether the removal of a certain effect size increases the fit of the remaining effect sizes in a homogeneous distribution while not substantially affecting the interpretation of the recalculated mean effect size. Various approaches to identifying potential outliers will be used, including visual examination of data organized into a forest plots and also performing “one study removed” CMA routine. Identified outliers will be examined with the potential to remove them from the final dataset. Potential sources of bias, such as study design, type of treatment, publication source, missing data, sample size, or attrition, will be carefully examined through the corresponding moderator variable analyses.

Publication Bias

There is widely recognized concern that relying on published studies alone may substantially distort (misrepresent) the overall intervention effect. To assess potential publication bias in this review, we will visually inspect the resulting funnel plot and run the Duval & Tweedie's (2000) trim and fill routine in CMA (Borenstein et al., 2005). Also, the classical Fail-Safe N test will help determine the number of null effect studies needed to raise the p-value associated with the average effect above a specified level of α. Orwin's (1983) Fail Safe N will also be used to determine how many studies missing studies that, when added to the analysis, will bring the combined Hedges’ g below a specified threshold.

Incomplete Reporting of Study Data

For studies that do not report complete outcome data (that is, in exceptional cases when missing information from otherwise perfectly suitable studies is minimal – e.g., sample size), the first author of the study will be contacted to retrieve missing information. If needed data are unavailable, a process of data imputation may be conducted, where appropriate. In such cases, a sensitivity analysis needs to be conducted to assess the impact of the imputed data on the overall analysis and synthesis of the results.

Treatment of qualitative research

No qualitative research will be reviewed within the framework of this project.

Footnotes

Appendix A

Appendix B

References

Aiello

N. C.

Wolfle

L. M.

(1980, April). A meta-analysis of individualized instruction in science . Paper presented at the annual meeting of the American Educational Research Association, Boston, MA. Retrieved from http://files.eric.ed.gov/fulltext/ED190404.pdf

Atkinson

R. K.

Renkl

Merrill

M. M.

(2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95, 774–783. http://dx.doi.org/10.1037/0022-0663.95.4.774

Bangert

R. L.

Kulik

J. A.

(1982, March). Individualized Systems of Instruction: A Meta-Analysis of Findings in Secondary Schools . Paper presented at the annual meeting of the American Educational Research Association, New York, NY. Retrieved from http://eric.ed.gov/?id=ED220358

Bloom

B. S.

(1968). Learning for mastery. Evaluation Comment, 1(2), 1–12.

Borenstein

Hedges

Higgins

Rothstein

(2005). Comprehensive meta-analysis version 2. Englewood, NJ: Biostat.

Borenstein

Hedges

L. V.

Higgins

J. P. T.

Rothstein

H. R.

(2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. http://dx.doi.org/10.1002/jrsm.12

Brusilovsky

(2001). Adaptive hypermedia. User Modeling and User-Adapted Interaction, 11, 87–110.

Cole

M. W.

(2014). Speaking to read: Meta-analysis of peer-mediated learning for English language learners. Journal of Literacy Research, 46(3), 358–382. http://dx.doi.org/10.1177/1086296X14552179

Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.

10.

Davies

C. S.

(1981). Teaching introductory Chemistry: Generality of the PSI approach. Journal of Chemical Education, 58(9), 686–689. http://dx.doi.org/10.1021/ed058p686

11.

Duvall

Tweedie

(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. http://dx.doi.org/10.1111/j.0006-341X.2000.00455.x

12.

Eyre

H. L.

(2008). Keller's Personalized System of Instruction: Was it a fleeting fancy or is there a revival on the horizon? The Behavior Analyst Today, 8(3), 317–324. http://dx.doi.org/10.1037/h0100623

13.

Gersten

Chard

Jayanthi

Baker

Morphy

Flojo

(2008). Mathematics instruction for students with learning disabilities or difficulty learning mathematics: A synthesis of the intervention research. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

14.

Gifford

V. D.

Vicks

(1982). A comparison of the personalized system of instruction and a conventional biology course on the achievement of junior college freshmen. Journal of Research in Science Teaching, 19(8), 659–664. http://dx.doi.org/10.1002/tea.3660190805

15.

Gresalfi

M. S.

Lester

(2009). What's worth knowing in mathematics? In Tobias

Duffy

T. M.

(Eds.), Constructivist Theory Applied to Instruction: Success or Failure? (pp. 264–290). New York, NY: Routledge.

16.

Guskey

T. R.

Gates

S. L.

(1985, April). A synthesis of research on group-based mastery learning programs . Paper presented at the annual meeting of the American Educational Research Association, Chicago.(ERIC Document Reproduction Service No. ED 262 088)

17.

Hammerstrøm

Wade

Jørgensen

A.-M. K.

(2010). Searching for studies: A guide to information retrieval for Campbell Systematic Reviews. Supplement 1. Retrieved from Campbell Collaboration website: http://www.campbellcollaboration.org/resources/research/Methods_Policy_Briefs.php

18.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

19.

Hedges

L. V.

Shymansky

J. A.

Woodworth

(1989). A practical guide to modern methods of meta-analysis. Washington, DC: National Science Teachers Association.

20.

Horak

V. M.

(1981). A meta-analysis of research findings on individualized instruction in mathematics. Journal of Educational Research, 74(4), 249–253. http://dx.doi.org/10.1080/00220671.1981.10885318

21.

Huang

Shiu

(2012). A user-centric adaptive learning system for e-learning 2.0. Journal of Educational Technology & Society, 15(3), 214–225.

22.

Huang

C.-T.

Yang

S. C.

(2015). Effects of online reciprocal teaching on reading strategies, comprehension, self-efficacy, and motivation. Journal of Educational Computing Research, 52(3), 381–407. http://dx.doi.org/10.1177/0735633115571924

23.

Johnson

A. W.

Johnson

(2002). Cooperative learning methods: A meta-analysis. Journal of Research in Education, 12(1), 5–14.

24.

Keller

F. S.

(1968). “Good-bye, teacher …”. Journal of Applied Behavior Analysis, 1, 79–89. http://dx.doi.org/10.1901/jaba.1968.1-79

25.

Kirschner

P. A.

Sweller

Clark

R. E.

(2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41, 75–86. http://dx.doi.org/10.1207/s15326985ep4102_1

26.

Klahr

(2009). “To every thing there is a season, and a time to every purpose under the heavens”: What about Direct Instruction? In S. Tobias & T. M. Duffy (Eds.), Constructivist Theory Applied to Instruction: Success or Failure? (pp. 291–310). New York, NY: Routledge.

27.

Kolodner

J. L.

(2004). The learning sciences: Past, present, and future. Educational Technology, 44(3), 37–42.

28.

Kolodner

J. L.

Starr

Edelson

Hug

Kanter

Krajcik

, … Zahm

(2008, June). Implementing what we know about learning in a middle-school curriculum for widespread dissemination: The Project-Based Inquiry Science (PBIS) story . Paper presented at the eighth International Conference of the Learning Science (ICLS), Utrecht, The Netherlands. Retrieved from https://llk.media.mit.edu/courses/readings/Kolodner-ICLS.pdf

29.

Kulik

J. A.

(1984, April). The fourth revolution in teaching: Meta-analyses . Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Retrieved from http://eric.ed.gov/?id=ED244617

30.

Kulik

C-L.

Kulik

J.A.

Bangert-Drowns

R.L.

(1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60, 265–299. http://dx.doi.org/10.3102/00346543060002265

31.

Kulik

J. A.

Kulik

C.-L. C.

Cohen

P. A.

(1979). A meta-analysis of outcome studies of Keller's Personalized System of Instruction. American Psychologist, 34(4), 307–318. http://dx.doi.org/10.1037/0003-066X.34.4.307

32.

Lipsey

M. W.

Wilson

(2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.

33.

Lysakowski

Walberg

(1982). Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal, 19, 559–578.

34.

Adesope

O. O.

Nesbit

J. C.

Liu

(2014). Intelligent tutoring systems and learning outcomes: A Meta-analysis. Journal of Educational Psychology, 106(4), 901–918. http://dx.doi.org/10.1037/a0037123

35.

MacArthur

C. A.

Schwartz

S. S.

Graham

(1991). Effects of a reciprocal peer revision strategy in special education classrooms. Learning Disabilities Research and Practice, 6(4), 201–210.

36.

Mazur

(1997). Peer instruction: Getting students to think in class. In Redish

E. F.

Rigden

J. S.

(Eds.), The Changing Role of Physics Departments in Modern Universities, Part Two: Sample Classes (pp. 981–988). Woodbury NY: American Institute of Physics. Retrieved from http://mazur.harvard.edu/sentFiles/Mazur_274537.pdf

37.

Orwin

R.G.

(1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159. http://dx.doi.org/10.3102/10769986008002157

38.

Özyurt, Ö., & Özyurt, H. (2015). Learning style based individualized adaptive e-learning environments: Content analysis of the articles published from 2005 to 2014. Computers in Human Behavior, 52, 349–358.

39.

Slavin

R.E.

(1986). Best-evidence synthesis: An alternative to meta-analytic traditional reviews. Educational Researcher, 15(9), 5–11. http://dx.doi.org/10.3102/0013189X015009005

40.

Tobias

Duffy

T. M.

(Eds.). (2009). Constructivist instruction: Success or failure. New York, NY: Routledge.

41.

VanLehn

(2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46, 197–221. http://dx.doi.org/10.1080/00461520.2011.611369