Strategies for Teaching Students to Think Critically

Contested Terrain: Defining Critical Thinking

Researchers have offered many definitions of CT (Ennis, 1962, 1987; Facione, 1990a, 1990b; Kurfiss, 1988; Lipman, 1991; Paul & Binker, 1990; Scriven & Paul, 1996; Siegel, 1988).

One leading definition of CT, developed by a Delphi consensus panel of 46 experts, including leading scholars such as Ennis, Facione, and Paul, and organized by the American Philosophical Association (APA), recognizes this complexity by offering a broad definition:

We understand critical thinking to be purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based. . . . The ideal critical thinker is habitually inquisitive, well-informed, trustful of reason, open-minded, flexible, fair-minded in evaluation, honest in facing personal biases, prudent in making judgments, willing to reconsider, clear about issues, orderly in complex matters, diligent in seeking relevant information, reasonable in the selection of criteria, focused in inquiry, and persistent in seeking results which are as precise as the subject and the circumstances of inquiry permit. (Facione, 1990b, p. 2)

For the purposes of this review, we decided that the APA’s report (issued by Facione, 1990b) is clear and inclusive enough to serve as a framework for deciding which empirical studies have adequately captured elements of CT. The authors describe six CT skills (including 16 “subskills”) and 19 CT dispositions as listed in Appendix A.

Within Appendix A, the skills/dispositions divide is worth a small additional discussion. The Delphi panel maintained that it was possible to possess the cognitive skills necessary to carry out CT but lack the affective dispositions—the general habits and attitudes—to exercise these skills. Consider the example of a person who possessed the cognitive skills associated with CT but who lacked the disposition to learn about or discuss social issues—it would be difficult to call this individual an effective critical thinker. Hence, the panel held that the development of both skills and dispositions was critical in the education of well-rounded critical thinkers (Facione, 1990b).

There are a number of potential objections to the APA’s consensus definition, one of which is that it is insufficiently broad (Alston, 2001). For example, Barbara Thayer-Bacon (2000) argues that good CT (she uses the term constructive thinking) broadens the definition to include more than just traditional rational skills and dispositions. Pushing beyond the traditional tools of reason, Thayer-Bacon argues convincingly that intuition, emotion, and imagination also play important roles in effective constructive conversations. She further argues that constructive thinking is more of a social practice than a skill or disposition inculcated in individuals. For Thayer-Bacon, good constructive thinking is analogous to a quilting bee in which one has many different tools at one’s disposal along with multiple participants who are diverse in their backgrounds and orientations. An even more profound objection to the APA definition is that it is not only insufficiently broad but also fundamentally inadequate. Two versions of this argument are available in a special issue of Studies in Philosophy of Education—one by Gert Biesta and Geert Stams (2001) and another by James Marshall (2001).

Biesta and Stams (2001) note that a specific conception of “criticality” always underlies any CT that is undertaken. Two of the approaches to criticality that they discuss are “critical dogmatism” and Derridean deconstruction. Critical dogmatism conceives of the activity of critique as “the application of a criterion to evaluate a particular state of affairs” (Biesta & Stams, 2001, p. 60). Given that the APA definition suggests critical thinkers should apply a variety of criteria in order to assess propositions, one can locate the definition of CT used in this article firmly within this camp. Biesta and Stams point out that a significant problem for the position of critical dogmatism is that it rests on an unjustifiable belief in the validity of its own method; when one tries to look back behind the criteria that are postulated, there is no further justification of them available that is not circular or that does not imply an infinite regress. Therefore, thinkers who hold this position reflectively actually stop at this point and make a deliberate choice to hold to their criteria “dogmatically” since no further justification is possible.

Biesta and Stams’s (2001) central objection to critical dogmatism is its “totalizing” tendency—its pretence to be a foundational point from which all critique can proceed and with which all critique must align. By contrast, their preferred framework of Derridean deconstruction abandons all pretentions of trying to establish a foundation of this type and, in fact, attacks critical dogmatism on this basis, thus serving as a critique of the standard approach to critique. Instead of the traditional approach, deconstruction places a heavy emphasis on what is made invisible and excluded:

Deconstruction thus tries to open up the system in the name of that which cannot be thought of in terms of the system. . . . Deconstruction is an affirmation of what is wholly other, of what is unforeseeable from the present . . . It is from this concern for what is totally other, a concern to which Derrida sometimes refers as justice, that deconstruction derives its right to be critical, its right to deconstruct. (Biesta & Stams, 2001, p. 68)

It is difficult to say exactly how an approach like this would translate into CT teaching practice, but at any rate, a deconstructionist approach would imply a style of CT that would depart dramatically from most of the studies synthesized in this meta-analysis.

Marshall’s (2001) account ventures less far afield in its critique of standard approaches to CT but is nonetheless more caustic. Like Biesta and Stams (2001), Marshall (2001) is dubious of the APA definition of CT cited above but for a different set of reasons. Marshall thinks that traditional conceptions of CT view it as a neutral set of conceptual tools that “lack the force of ‘critical’ to be found in, e.g., the Frankfurt school” (p. 75). Like the critical theorists of this group, Marshall is interested in a more robust style of critique that will enable students to question foundational aspects of the social systems they inhabit.

In the course of providing an articulation of this style of critique, Marshall (2001) traces the history of the concept of the self in Western philosophy from Descartes to Foucault. It is the Foucaultian conception that Marshall finds to be particularly promising. For Foucault, Marshall (2001) maintains, the fundamental question for the self is “how one practices one’s freedom” (p. 83). In this regard, Foucault (1984/1998) emphasizes the importance of a kind of self-care, which involves “an exercise of the self on the self by which one attempts to develop and transform oneself” (p. 282). This kind of reflective, critical practice, which also includes care-for-others by extension, allows for the development of positive “practices of freedom” that go beyond simply the negative but important acts of liberating people from oppressive social systems (e.g., oppressive sexuality; Foucault, 1984/1998, p. 283).

These counternarratives to the APA definition of CT are important, and there is no intellectually honest way to wholly reconcile our analysis with them. It is, for example, virtually impossible that the kinds of studies captured by this meta-analysis (or, indeed, any other meta-analysis of CT) are going to correspond to the kind of criticality that Biesta and Stams (2001) and Marshall (2001) would hope to see. They would likely object to the studies’ generally short duration, the standardized assessments of CT, and the conventional and narrow definition of CT that is endorsed in most of them. Furthermore, all of the authors, Thayer-Bacon (2000) included, would like to see an approach that puts a greater emphasis on working with the student as opposed to intervening on him or her, and all of them have at least some sympathy for a position that would teach students to question social systems.

Some of the questions raised by these counternarratives will be further addressed in the “Outstanding Questions” section toward the end of the article, but for now, let us give three responses to these challenges to the APA definition of CT that we have endorsed. First, we are, in fact, sympathetic to some of these objections. We appreciate the inclusiveness of Thayer-Bacon’s (2000) conception of constructive thinking, and some of us feel Marshall (2001) may well be right in saying that a deeper and less interventionist approach to the promotion of criticality is desirable. However, as meta-analysts, this question is outside the competence of our method; our task is not to forge new pathways in conceptualizing CT but rather to describe and pool existing findings. The specificity of the APA definition, with its enumeration of specific skills and dispositions, serves well to synthesize the kinds of interventions that are typically used in CT research.

Second, although this is a question that would ultimately need to be resolved empirically, we believe that even if we adopted a broader definition of CT that attempted to account for some of these counternarratives, it would have made little to no difference in terms of the results of the meta-analysis. This is due to the fact that the kinds of new approaches to CT in which Thayer-Bacon (2000), Biesta and Stams (2001), and Marshall (2001) are interested are unlikely to be investigated using the types of quantitative approach that can be synthesized by a meta-analysis. We recognize that this raises legitimate questions about the limitations of the method we have chosen, which we address briefly in the final section.

Third, although there is a sense in which these authors, particularly Biesta and Stams (2001) and Marshall (2001), are working in an alternate paradigm of criticality vis-à-vis the APA definition, the paradigms are not totally incommensurable. There is some potential common ground between even the narrowest effort to promote the development of CT skills and the kinds of efforts that a scholar who is sympathetic to critical theory (e.g., Marshall) might hope to promote. Let us take Foucault’s (1978/1997) article—“What Is Critique?”—as an illustration in this case. Foucault (1978/1997) locates the beginnings of the “critical attitude,” which he defines as “the art of not being governed quite so much,” in the early efforts to establish the consistency and truth of scripture (p. 45). He also identifies the efforts to question and move beyond justifications rooted merely in authority as an important precursor to the modern critical movement.

Arguably, a similar commonality can be found between the kinds of relatively modest efforts to teach CT skills and dispositions that are synthesized in this meta-analysis and the more ambitious research programs detailed by scholars like Thayer-Bacon, Biesta, and Marshall. A short intervention that endeavours to enhance students’ CT skills is not what these philosophers would prefer to see, but it may nonetheless enhance students’ openness to “not be governed quite so much,” which could be useful in getting students ready to think about the more expansive conceptions of criticality that these scholars wish to investigate further. In other words, the mere fact that we have opted for the APA definition of CT skills and dispositions does not mean that the analysis we offer has nothing to offer for those who would endorse another definition. The kinds of very general techniques—dialogue, posing authentic problems, mentorship—that our meta-analysis indicates are valuable in the development of CT skills and dispositions will likely also have some value for those who have alternate conceptions of CT.

Are CT Skills Generic or Context-Specific?

Settling on a working definition of CT is a good first step, but there are other important questions that remain outstanding even if one is able to convene on the APA definition. For example, in his pioneering research on the development of CT, Glaser (1941a, 1941b) highlighted a difficulty that has remained outstanding ever since: If CT has aspects that are appropriately regarded as skills-based, do these skills represent generic traits and skills or are they context-bound? Glaser (1941b) also found the changes in CT to be somewhat general in character:

The aspect of critical thinking which appears most susceptible to general improvement is the attitude of being disposed to consider in a thoughtful way the problems and subjects that come within the range of one’s experience. An attitude of wanting evidence for beliefs is most subject to general transfer. Development of skills in applying the methods of logical inquiry and reasoning, however, appear to be specifically related to, and in fact limited by, the acquisition of pertinent knowledge and facts concerning the problem of subject matter towards which the thinking is to be directed. (p. 175).

Some 70 years after Glaser, there is little consensus about whether CT is a set of generic skills that apply across subject domains (engineering, arts, science) or depends on the subject domain and context in which it is taught (Ennis, 1989). Prevailing psychological views tend to favor the generic traits approach; learning to think critically is understood as gaining mastery over a significant series of discrete skills or mental operations and dispositions that can be generalized across a variety of contexts. These skills include concepts such as interpreting, predicting, analyzing, and evaluating. The primary appeal of the meta-cognitive skills discourse on CT involves its presumptive transfer between contexts. According to Woolfolk (1998), higher order thinking such as CT requires consciously applying abstract knowledge, heuristics, or procedures learned in one context to some novel circumstance or situation.

Many philosophers like Ennis (1989), Siegel (1988), Govier (1985), and Paul (1985) also view CT skills as rather general (applicable across subject domains) and conjecture (a) that general CT skills might apply to more than one subject area, despite the fact that CT always involves thinking about some specific context; and (b) that the existence of general skills does not imply the nonexistence of context-specific knowledge. The generalist view supported by Siegel (1988) contends that the ability to identify informal errors of reasoning (e.g., post hoc and hasty generalization fallacies) is easily transferable between different contexts.

Alternatively, the specifist view (perhaps best represented in the position of McPeck, 1981) argues against general CT skills or capacities due to the fact that thinking is always tied to a subject domain. McPeck also argues against the existence of general skill on the grounds that (a) all thinking is thinking about something; (b) general CT ability is not possible because knowledge of a subject is necessary for CT; and (c) CT varies greatly from field to field. McPeck (1981) remarks,

[I]t makes no sense to talk about critical thinking as a distinct subject and therefore cannot profitably be taught as such. To the extent that critical thinking is not about a specific subject X, it is both conceptually and practically empty.” (p. 5)

Strategies for Teaching CT Skills and Dispositions: What Has Previous Research Found?

The generic versus content-specific debate has significant practical implications for education—if CT is generic, then it could be fruitfully taught in specialized courses that focus on CT skills (Royalty, 1995; Sá, Stanovich, & West, 1999), and if it is dependent on subject matter, then it might best be learned by tackling concrete problems in specific disciplines (Halliday, 2000; Smith, 2002). But the generic versus content-specific debate is not the only outstanding debate about how CT skills can be taught. Generally speaking, there are a number of ways in which CT skills are usually taught in university and K-12 contexts. In previous analyses of current approaches (e.g., Abrami et al., 2008), Ennis’s (1989) CT typology of four courses, generic, infusion, immersion, and mixed, was used for classifying and describing various instructional interventions. The typology breaks down as follows: in generic courses, CT skills and dispositions are the course objective, with no specific subject matter content. In contrast, content is important in both the infusion and immersion approaches. CT is an explicit objective in the infusion course but not in the immersion course. In the mixed approach, CT is taught as an independent track within a specific subject content course.

According to Ennis (1989), the general approach attempts to teach CT skills and dispositions separately from the presentation of the content of existing subject matter offerings, with the purpose of teaching CT. Examples of the general approach usually do involve some content but do not require that there be content. The infusion of CT requires deep, thoughtful, and well-understood subject matter instruction in which students are encouraged to think critically in the subject. Importantly, in addition, general principles of CT skills and dispositions are made explicit. In the immersion approach, subject matter instruction is thought-provoking, and students do get immersed in the subject. However, in contrast to the infusion approach, general CT principles are not made explicit. The mixed approach consists of a combination of the general approach with either the infusion or immersion approach. Under it, students are involved in subject-specific CT instruction, but there is also a separate thread or course aimed at teaching general principles of CT.

This mix of approaches to developing CT skills and dispositions is mirrored in the empirical literature on CT; Norris (1985) testified to this when he remarked,

[T]here is little, if any, evidence on the long-term impact of instruction in critical thinking, despite the fact that the vision of such impact is central to the justification of critical thinking instruction . . . If diagnosis and remediation of specific flaws in reasoning are goals of critical thinking instruction, then more fine-grained information on the effects of particular teaching strategies will have to be sought. (pp. 44–45)

Norris’s (1985) concerns remain valid today, and they can be coupled with the fact that most reviews have either been inconclusive in determining whether particular teaching strategies seem to enhance CT skills and dispositions or restricted their focus of attention to very specific issues. Some examples: Adams (1999) summarized studies of the impact of nursing education programs on the CT skills of professional nursing students; Allen, Berkowitz, Hunt, and Louden (1997, 1999) studied the impact of various methods of improving public communication skills on CT; Assessment and Learning Research Synthesis Group (2003) at University of London reviewed in great detail the impact on students and teachers of the use of ICT (information and communication technologies) for assessment of creative and CT skills; Bangert-Drowns and Bankert (1990) reported some effects of explicit instruction for CT on measures of CT; Follert and Colbert (1983) analyzed research on debate training and CT improvements; Follman (1987, 2003) documented the difficulty of teaching CT, especially to adults; McMillan (1987) considered the effects of instructional methods, courses, programs, and general college experiences on changes in college students’ CT; and Pithers and Soden (2000) reviewed the research literature on teaching CT skills (methods and conceptions of teaching likely to inhibit or enhance CT). Otherwise, although not a comprehensive review, Abrami et al. (2008) found 117 studies based on 20,698 participants, which yielded 161 effect sizes with an average effect size of g+ = +0.34 and a standard deviation of 0.61, suggesting a modest but robust effect favoring the instructional viability hypothesis that is at the heart of the current review. Finally, no review has examined comprehensively the impact that CT instructional interventions might have on subject matter achievement.

In sum, despite a number of significant efforts to collate and review the results of previous empirical research on CT, the question of effective teaching strategies for CT remains outstanding. Thus, in undertaking the current review of research on instructional interventions affecting CT, several basic questions will be addressed: (a) Can CT skills and dispositions be taught? (b) What are some promising strategies for teaching students to think critically? (c) Which students benefit from CT instruction? (d) Are there curricular areas for which CT instruction works best?

Method

Data and Effect Size Extraction

The final collections (i.e., generic CT skills, content-specific CT skills and CT dispositions) contained a combination of standardized and nonstandardized measures. The majority of the studies in each collection reported only one outcome measure. However, in some cases two measures of the same type were found. When several measures representing the same outcome type were reported in a given study, reviewers always chose a standardized measure over a locally produced measure (e.g., teacher-made). When two standardized (e.g., WGCTA and CCTST) or two locally produced measures were encountered, effect sizes were first calculated for each test separately and then the effect sizes were averaged (Scammacca, Roberts, & Stuebing, 2013). For example, in the collection of generic outcomes, 20 composites out of 341 effect sizes were created in this way. Also, whenever the same group of participants was used repeatedly (i.e., the same control group compared to two different treatment groups), the group sample size was reduced proportionally in order to avoid its overrepresentation in the final data set.

Effect size is a standardized metric expressing the difference in two group means (usually a control and a treatment). Cohen’s (1988) d (Equation 1) is the biased estimator of effect size.

d = X ¯ e − X ¯ c S D Pooled .

There are also two modifications of this basic equation: one for studies reporting pretest data for both experimental and control groups and another for a single-group pre–posttest design. In other cases (e.g., t tests, F tests, p levels), effect size is estimated using conversion formulas provided by Glass, McGaw, and Smith (1981) and Hedges, Shymansky, and Woodworth (1989).

To correct for bias in small samples, d was converted to the unbiased estimator g (Hedges & Olkin, 1985), as follows:

g ≅ ( 1 − 3 4 N − 9 ) d .

The standard error of g was calculated using Equation (3).

S E g = 1 n e + 1 n c + g 2 2 N ( 1 − 3 4 N − 9 ) .

Comprehensive Meta-Analysis (Version 2.2.064; Borenstein, Hedges, Higgins, & Rothstein, 2005), a dedicated statistical software package, was used for all primary, moderator, publication bias, and sensitivity analyses.

Results

The results are divided into four sections of unequal size. The first and largest section is devoted to an examination of generic CT outcomes. A second section briefly presents the evidence on the effects of CT instruction on achievement outcomes. The third section presents the results for content-specific outcome measures. The final section is devoted to the effects of CT instruction on CT dispositions.

Agreement rates at various stages of the review were as follows:

Generic CT Outcomes

Publication Bias

Several measures were taken to examine potential bias due to publication source. First, a funnel plot of the effect sizes by standard error (i.e., sample size) was examined to determine if the collection was relatively balanced on either side of the average random effect size. This examination yielded a reasonably symmetrical funnel with no obvious biases. Second, a classic fail-safe analysis and Orwin’s adaptation yielded the following results: (a) 434,819 studies are needed to bring the observed p value to an alpha above .05 and (b) 1,833 null-effect studies would be required to reduce the average effect in this meta-analysis to a “trivial” average effect size value of 0.10. Both of these analyses were conducted within the analysis module of Comprehensive Meta-Analysis, and they indicate the robustness of the collection.

Date of Publication

The formal development of standardized instruments to measure CT skills began in the 1940s along with the first experimental studies of CT instruction. The examination of the instructional viability and nature of CT teaching methods continues to this day. The frequency of empirical included studies increases in the decades from the late 1930s to 2009, with the greatest frequency in the past two decades (i.e., 1990–1999, 26.6%; 2000–2009, 44.6%) for a combination of more than 70%. Research on the topic has increased steadily over time, which is likely indicative of the importance that educators and researchers have come to attach to the teaching of generic CT skills.

Characteristics of the Complete Collection

Table 1A shows the summary statistics for the collection of 867 effect sizes included in the original collection. This average effect size is on the lower boundary of a moderate average effect size according to Cohen’s (1988) categories of effect size magnitude. This positive average effect size is suggestive of the potential to teach generic CT skills.

OPEN IN VIEWER

Table 1

Analysis of generic CT skills: (A) overall mean effect size for the fixed and random models; (B) mixed-effects moderator analysis of categories of research design

		Effect size		95% Interval
Overall outcome	k	g+	SE	Lower	Upper	Test of null, z
A
Fixed effects	867	0.33	0.01	0.31	0.34	47.42*
Heterogeneity	Q-Total = 5,275.62, df(Q) = 866, p < .001, I2 = 83.58
Random effects	867	0.39	0.02	0.35	0.42	21.39*
B
Preexperiments	363	0.39	0.03	0.34	0.45	13.00*
Quasi experiments	361	0.33	0.03	0.28	0.38	11.00*
True experiments	128	0.30	0.04	0.21	0.38	7.50*
Q-Between = 5.25, df = 2, p = .07

Note. CT = critical thinking; df = degrees of freedom.

*

p < .001.

Analysis of Outliers, Research Design, and Quality of Outcome Type

Because of the magnitude of this collection, it was prudent to examine it first in terms of the potentially biasing effects of extreme outliers, research design, and the quality and nature of the outcome measures. Fifteen outliers, whose magnitude was g = +2.5 or higher were identified and removed. The average random effect size (k = 852) dropped to g+ = 0.35, p < .001.

Table 1B shows the breakdown of studies categorized by research design. Although the average mixed effect for research design approached significance (p = .07), Bonferroni-adjusted post hoc analysis revealed that preexperiments were significantly higher than both true experiments and quasi experiments, and that these categories were not significantly different from each other (z = 0.82, p = .21). The decision was made to use only true and quasi experiments in further analyses. Also, studies that were long, over a semester, in treatment duration or when the treatment particularities could not be confidently identified.

This left 425 effect sizes with a random effects average of g+ = 0.35, z = 14.17, p < .00. Under the fixed-effects model, heterogeneity was significant (Q_T = 2,059.43, degrees of freedom [df] = 424, p < .001) and I², a descriptive measure of between-study heterogeneity exceeding sampling error was 79.41%.

To further determine if these initial results stood up to scrutiny when the larger collection of evidence was reduced to the set of better evidence, nonstandardized outcome measures (i.e., those that are of indeterminate reliability and validity) were removed. After removing 84 effect sizes derived from nonstandardized measures, the collection was further reduced to 341 effect sizes. Removing nonstandardized measures also reduced the random-effects average effect size, but only minimally (Table 2A). The collection, however, remained heterogeneous (Q_T = 1,225.89, df = 340, p = .001) and I² remained high at 72.27%.

OPEN IN VIEWER

Table 2

Analysis of generic CT skills with standardized measures only: (A) overall mean effect size for the fixed and random models; (B) mixed-effects moderator analysis of categories of research design

		Effect size		95% Interval
Overall outcome	k	g+	SE	Lower	Upper	Test of null, z
A
Fixed effects	341	0.25	0.01	0.23	0.28	21.09*
Heterogeneity	Q-Total = 1,225.89, df(Q) = 340, p = .001, I2 = 72.27
Random effects	341	0.30	0.02	0.25	0.34	12.06*
B
Quasi experiments	243	0.29	0.03	0.23	0.34	9.67*
True experiments	98	0.32	0.05	0.23	0.41	6.40*
Q-Between = 0.31, df = 1, p = .58

Note. CT = critical thinking; df = degrees of freedom.

*

p < .001.

After removing nonstandardized outcomes, high-quality quasi-experimental designs and true experiments were compared and were found to be not significantly different (Table 2B), suggesting that these two categories of effect sizes could be combined. The final reduced sample, on which all other comparisons are based, was k = 341. This number represents 39.33% of the effect sizes originally included in the study.

A frequency histogram of this reduced collection is shown in Figure 1. This distribution is slightly leptokurtic, but essentially normal, with an un-weighted mean of 0.33 and a standard deviation of 0.55. Of the 341 effect sizes, 261 (76.5%) were zero or above, and 80 (23.5 %) were below zero.

OPEN IN VIEWER

Figure 1.

Distribution of unweighted effect sizes for generic critical thinking skills (k = 341, X ¯ = 0.33, SD = 0.55).

Even with the dramatic reduction in the number of effect sizes retained for analysis, the average effect size and heterogeneity statistics did not change dramatically. In the remaining analyses only the subset of effect sizes considered to represent the best and most methodologically sound evidence on instructional interventions was included.

Course-Level Moderator Variables

This section begins the exploration of demographic factors, instructional variables, and other characteristics that may relate to the effect sizes. As we will outline below, variations in educational level, subject matter, and treatment duration did not generate significant differences in outcome. As a result, there are likely no confounds with these variables that could compromise subsequent substantive analyses.

Educational level

Table 3A shows the 341 effects broken down by educational level. There are no significant differences among the levels, suggesting that a skills-based approach to CT improvement can achieve some level of success at all grade levels. Furthermore, with the exception of the last category (i.e., graduate and adult students) the average effect size of CT instruction is significantly greater than zero.

OPEN IN VIEWER

Table 3

Mixed moderator variable analysis of course demographic variables: (A) educational level; (B) subject matter; (C) duration of treatment/control

		Effect size		95% Interval
Educational levels (age, years)	k	g+	SE	Lower	Upper	Test of null, z
A
Elementary school (6–10)	49	0.37	0.08	0.22	0.52	4.63*
Middle school (11–15)	78	0.37	0.06	0.26	0.48	6.17*
High school (16–18)	71	0.25	0.05	0.15	0.35	6.25*
Undergraduate	126	0.26	0.04	0.19	0.33	6.50*
Graduate and adult students	17	0.21	0.13	−0.05	0.46	1.62
Q-Between = 4.83, df = 4, p = .30
B (k = 269; “Missing” and “Other” categories removed)
Health/medical education	29	0.20	0.09	0.02	0.37	2.22*
STEM subjects a	73	0.31	0.05	0.21	0.41	6.20*
Non-STEM subjects b	167	0.29	0.04	0.22	0.36	7.25*
Q-Between =1.25, df = 2, p = .53
C (k = 338) a
Short (1 hour to 2 days)	13	0.66	0.26	0.15	1.17	2.54*
Medium (2 days to < one semester)	99	0.33	0.05	0.24	0.43	6.60*
One semester	130	0.27	0.03	0.21	0.34	9.00*
>One semester	96	0.23	0.04	0.15	0.32	5.75*
Q-Between = 4.54, df = 3, p = .21

Note. df = degrees of freedom.

a

STEM subjects are science, technology, engineering, and mathematics.

b

Non-STEM subjects are all other subject areas, except for health/medical education.

*

p < .05

Subject matter

No significant differences among different broad types of subject matter were observed as shown in Table 3B. In fact, the confidence intervals for STEM and non-STEM overlap almost perfectly, with a g+ of 0.31 for the former and 0.29 for the latter. Furthermore, the average effect size of CT instruction was significantly greater than zero for all subject matters.

Duration of instruction in treatment/control

Duration of the intervention did not significantly affect CT effect sizes, suggesting that the effects of CT instruction do not vary whether instruction is brief or not. More importantly, the average effect size for CT instruction was significantly greater than zero for each category of treatment duration (Table 3C).

The analyses of educational level, subject matter, and treatment duration point toward the generalizability of CT instructional effects, notwithstanding the fact that there remains significant variability among the effect sizes.

Instructional Moderator Variables

Having dealt with these basic study features, we now turn to an analysis of instructional variables related to generic CT outcomes based on the above-described two taxonomies (Ennis, our three-category scheme) and resultant coding schemes.

Instructional variables based on Ennis’s (1989) classification

All four of the methods in Table 4, coded according to Ennis’s (1989) classification, produced significantly positive average effect sizes, but the categories did not differ from one another.

OPEN IN VIEWER

Table 4

Mixed-effects moderator analysis of Ennis’s (1989) classification of instruction (k = 341)

		Effect size		95% Interval
Overall outcome	k	g+	SE	Lower	Upper	Test of null, z
Direct instruction	44	0.26	0.06	0.14	0.38	4.33*
Infusion	152	0.29	0.04	0.22	0.36	5.80*
Immersion	61	0.23	0.05	0.13	0.32	4.60*
Mixed	84	0.38	0.06	0.26	0.51	6.33*
Q-Between = 4.10, degrees of freedom = 3, p = .25

Instructional variables based on three-category scheme

The Three-Category Scheme includes coding for Dialogue, Authentic Instruction, and Mentoring. The analyses of each instructional category are based on a subset of the data because not all types of instruction were present in each study. To further refine the analyses, findings where there is no difference in type of instruction between the experimental and control groups (no difference in instruction) are contrasted with findings where the experimental group had higher amounts of the type of instruction compared to the control condition (instruction favoring the experimental group).

Two of the three categories, Dialogue and Authentic or Anchored Instruction, produced significantly larger positive effect sizes for instruction favoring the experimental group. This did not hold true for the Mentoring category, perhaps due to the smaller number of effect sizes for the latter. However, as Table 5 details, all three categories of instruction were significantly different from zero in terms of instruction favoring the experimental group.

OPEN IN VIEWER

Table 5

Mixed-effects moderator analysis of categories of the three-category classification: (a) dialogue; (b) authentic or anchored instruction; (c) mentoring, coaching, or tutoring

		Effect size		95% Interval
Categories	k	g+	SE	Lower	Upper	Test of null, z
A (k = 201)
No difference	87	0.19	0.04	0.11	0.27	4.75*
experimental > control	114	0.32	0.05	0.23	0.42	6.40*
Q-Between = 4.53, df = 1, p = .03
B (k = 214)
No difference	118	0.22	0.04	0.14	0.29	5.50*
Experimental > Cont.	96	0.34	0.05	0.24	0.44	6.80*
Q-Between = 4.07, df = 1, p = .04
C (k = 266)
No difference	238	0.26	0.03	0.20	0.31	8.67*
Experimental > control	28	0.39	0.08	0.23	0.55	4.88*
Q-Between = 2.58, df = 1, p = .11

Note. df = degrees of freedom.

*

p < .05.

Single and combined effects of instructional interventions

The following analysis (Table 6) describes certain combinations of instructional strategies (e.g., authentic instruction plus dialogue) when these strategies exceeded those in the control condition. Other combinations were excluded because of low cell frequency. Therefore, this result constitutes a refinement of the result reported in Table 5. This analysis represents the unique contributions of authentic instruction and dialogue as well as both the outcome of the dual combination of authentic instruction and dialogue and the triple combination of authentic instruction, dialogue, and mentoring (A + D + M). As an example, the 19 effect sizes represented in the A + D + M row of Table 6 are the only studies in which all three dimensions were higher in the experimental groups than in the control groups. A comparison between “authentic instruction + dialogue + mentoring” and “authentic instruction + dialogue” revealed a significant difference (z = 1.98, p = .024). A combination of all three strategies produced the highest effect size.

OPEN IN VIEWER

Table 6

Mixed-effects moderator analysis of category combinations based on the three-category classification

		Effect size		95% Interval
Categories	k	g+	SE	Lower	Upper	Test of null, z
Authentic Instruction (A)	22	0.25	0.10	0.05	0.46	2.50*
Dialogue (D)	43	0.23	0.08	0.07	0.39	2.87*
A + D	45	0.32	0.08	0.17	0.47	4.00*
A + D + Mentoring	19	0.57	0.10	0.38	0.77	5.70*
Q-Between = 8.19, degrees of freedom = 3, p = .04

*

p < .05.

Instructional subcategories

For each subcategory, studies were identified where the experimental group exceeded the control group on the quality or quantity of each specific feature. When the experimental group equaled or exceeded the control group for at least five comparisons (i.e., k ≥ 5), an analysis was conducted to determine whether a subcategory explained effect sizes significantly (p < .05). The following Dialogue subcategories were significant and favored the experimental group: teacher poses questions (g+ = 0.38, k = 19, p < .001), whole-class discussions led by the teacher (g+ = 0.42, k = 16, p < .001), and small group discussions led by the teacher (g+ = 0.41, k = 14, p < .001). Two subcategories of Authentic and Anchored Instruction significantly favored the experimental group: applied problem solving (g+ = 0.35, k = 31, p < .001) and role-playing (g+ = 0.61, k = 5, p < .001). For mentoring, no subcategory significantly favored the experimental group.

Content-Specific CT Outcomes

The same analysis approach that was used in the previous section on generic CT outcomes was also applied to studies that attempted to promote content-specific CT. Content-specific CT outcomes are those skills that are assessed using measures specifically designed to relate thinking skills to the content that is being taught in a course. The analysis began with 198 effect sizes. Table 7A shows the results after the removal of preexperiments. It is immediately evident that content-specific outcomes produced a higher average effect size compared to generic CT outcomes. The finding of g+ = 0.57 ranks at the low end of Cohen’s category of effects that are deemed to be of a moderate magnitude. However, as with generic effects, the distribution of 97 effect sizes was significantly heterogeneous (Table 7A), suggesting a large degree of between-study variation based on an I² value of 82.36%.

OPEN IN VIEWER

Table 7

Overall mean effect size (g+) and statistics for content-specific CT outcomes: (A) content-specific ct outcomes with pre-experiments removed; (B) mixed moderator variable analysis for measure source (k = 97)

		Effect size		95% Interval
Overall outcome	k	g+	SE	Lower	Upper	Test of null, z
A
Fixed effects	97	0.53	0.02	0.49	0.57	25.24*
Heterogeneity	Q-Total = 544.32, df(Q) = 96, p < .001, I2 = 82.36
Random effects	97	0.57	0.05	0.47	0.68	10.75*
B
Standardized	31	0.40	0.07	0.26	0.53	5.71*
Nonstandardized	66	0.65	0.07	0.52	0.78	9.29*
Q-Between = 6.67, df = 1, p = .01

Note. CT = critical thinking; df = degrees of freedom.

*

p < .05.

Source of Measure With Preexperiments Removed

The results shown in Table 7B examine the difference in average effect sizes between standardized and non-standardized outcome measures. The average effect fell to g+ = 0.40 for the 31 effect sizes derived from standardized measures. Regardless of the measure source, the average effect sizes for CT instruction were significantly greater than zero for this category of outcome.

The analysis of substantive study features for content-specific CT outcomes revealed a similar pattern to the results for generic outcomes. In particular, the effects of CT instruction were significant (p < .05) for all of the substantive study features including educational level, subject area, duration of instruction, and type of instruction.

CT Dispositions

The original collection of included studies produced 82 effect sizes. After removing preexperiments, this number fell to 25 effect sizes with an average effect size of g+ = 0.23 that was significantly greater than zero (Table 8) but with moderately high heterogeneity. Due to the small number of effect sizes, methodological, assessment, and substantive features were not explored in detail.

OPEN IN VIEWER

Table 8

Overall mean effect size for CT dispositions with preexperimental studies removed

		Effect size		95% Interval
Overall outcome	k	g+	SE	Lower	Upper	Test of null, z
Fixed effects	25	0.19	0.04	0.11	0.28	4.40*
Heterogeneity	Q-Total = 82.32, df(Q) = 24, p < .001, I2 = 70.84
Random effects	25	0.23	0.09	0.06	0.40	2.64*

Note. CT = critical thinking; df = degrees of freedom.

*

p < .05.

Summary of Findings

The global findings are summarized in Table 9. The column heading “Magnitude of Average Effect” applies Cohen’s (1988) qualitative criteria to the average effects of generic CT skills, content-specific CT skills, and CT dispositions. The column heading “Improvement Index” refers to the percentage of gain under the normal distribution between the experimental condition compared to the control condition. One way of interpreting this index is to consider the fact that an average student in the experimental condition (50th percentile) would place X% higher among students in the control condition (i.e., 50th percentile + Xth percentile). Finally, the column titled “Significantly Heterogeneous” is a result of a fixed-effects analysis and indicates whether between-study variability exceeds what is expected by chance. It also indicates whether there is sufficient unexplained variation to warrant moderator variable analysis.

OPEN IN VIEWER

Table 9

General findings from three CT measures and content outcomes derived from the reduced collections

Outcome measure	k	g+ a	Magnitude of average effect	Improvement index c	Significantly heterogeneous d
Generic CT Skills	341	0.30	Low b (0.20 to <0.50)	11.79%	Yes
Achievement	140	0.33	Low (0.20 to <0.50)	12.93%	Yes
Content-specific CT skills	97	0.57	Moderate (0.50 to <0.80	21.57%	Yes
CT dispositions	25	0.23	Low (0.20 to <0.50)	9.10%	Yes

Note. CT = critical thinking.

a

Random-effects model.

b

Based on Cohen’s (1988) qualitative descriptions.

c

Improvement index = (Φ(g+) − 0.50) × 100), where Φ is the cumulative frequency of g+ under the normal curve.

d

Derived from the fixed-effects model.

Discussion

This review addressed several basic questions: (a) Can CT skills and dispositions be taught? (b) What are some promising strategies for teaching students to think critically? (c) Which students benefit from CT instruction? (d) Are there curricular areas for which CT instruction works best?

At the most general level, this analysis clearly reveals that a variety of CT skills (both generic and content specific) and dispositions can develop in students through instruction at all educational levels and across all disciplinary areas using a number of effective strategies. There are some caveats attached to this finding, but we will postpone discussion of these until our “Outstanding Questions” section.

Looking more specifically at the question of possible instructional strategies, it is clear that two general types of instructional interventions are especially helpful in the development of generic CT skills. Notably, the opportunity for dialogue (e.g., discussion) appears to improve the outcomes of CT skills acquisition, especially where the teacher poses questions, when there are both whole-class teacher-led discussions and teacher-led group discussions. Similarly, the exposure of students to authentic or situated problems and examples seems to play an important role in promoting CT, particularly when applied problem solving and role-playing methods are used.

In addition, as Table 6 indicates, it also appears as though dialogue and authentic instruction are effective in combination, particularly when mentorship is added to the mix. As our findings demonstrate, studies that featured all three types of intervention (A + D + M) generated significantly larger effect sizes than either (A + D) or A or D alone. This is particularly interesting in light of the fact that mentoring did not generate especially strong results when analyzed on its own. In light of these results, it appears as though mentoring may serve in a catalytic capacity for CT; it can augment other strategies in a powerful way but is not especially successful if pursued in isolation.

It may be illustrative to examine more closely three interventions in which all of the instructional elements of the three-category scheme are strongly present. First, Yang, Newby, and Bill (2008) studied the effectiveness of structured Web-based bulletin board discussions on CT skills of distance veterinary students. Learners were required to contribute their thoughts and ideas to a series of course exercises and were encouraged to think critically about the course content and especially about questions and feedback comments from both their peers and the instructor. In this study, all three major instructional dimensions were coded equally (+1) in favor of the experimental condition. The effect on CT skills (as measured by the CCTST) in comparison with the control students, whose Web-based discussions were unstructured and unguided, was g = +0.53.

Second, in a study conducted by Arrufat (1997), the high prevalence in the experimental condition of all three major instructional strategies (i.e., Discourse—coded +3, Authentic Instruction—coded +3, and Mentoring—coded +2) resulted in the effect size of g = +0.638. In this study, undergraduate Psychology students received an intervention intended to promote identity development. The instruction focused on fostering an increase in the use of exploration and critical problem solving with respect to making personal life choices (Authenticity) with guidance offered by the teacher, both individually (Mentoring) and in whole-class discussions (Discourse).

Third, and finally, Pellegrino (2007) conducted a study in which high school students in the experimental condition were taught American History through systematically engaging in “historical thinking” activities. Typically, students were asked to present their views on a historical period in question and offer their explanations of what developments had shaped it. Students’ responses were then aligned with various sources of historical information and were asked to judge, for instance, the relative reliability of similar events used to illustrate conflicting viewpoints. The teacher tried to relate course themes to learners’ previous experience and knowledge in class discussions. Activities were designed to allow both the teacher and the students to recognize and appreciate various perceptions of historical content in its complexity, conduct an independent inquiry, find and examine documents, and build their understanding of the dynamics of historical events. There were several role-play games and series of discussions involved. The major difference between the experimental and control conditions was on the dimension of Authentic Instruction (+3) with a supplement of Discourse, and this intervention resulted in the effect size of g = +1.13.

In all three studies, we see large effect sizes resulting from a combination of all elements of our three-category scheme. Furthermore, it is clear from the descriptions that all three examples are compelling, longer duration studies in which a robust multifaceted intervention achieved impressive results, and the results of this meta-analysis clearly point toward a need for further pursuit, exploration and refinement of these especially successful strategies.

Outstanding Questions

Before concluding, we feel that it is worthwhile to consider three significant objections that could be levied against the analysis so far: first, that this review tacitly endorses a quasi-causal view of teaching CT in which successful instruction is simply a matter of adopting the correct instructional processes; second, that our contention that CT can be taught is subject to some significant caveats. As we will explain, the first objection is mitigated by the nature of our analysis, whereas the second objection deserves serious consideration.

The first objection holds that within some strands of educational thought, there is a prevailing misrepresentation of the teaching process that makes it seem as though good teaching is nothing more than applying the “magic recipe” of teacher behaviors that will produce the desired product. One could further argue that this mentality is present in our own approach, which has attempted to determine which types of instructional interventions yield the most promising results in terms of generating favorable results on standardized tests that purport to measure CT.

The first part of the objection has some substance and captures a line of flawed thinking that has been influential. One can see a particularly clear and virulent version of it in Skinner’s (1950, 1959) educational reform efforts, and a more moderate version is present within the process–product research tradition (Gage & Needels, 1989). Furthermore, it is probable that many of the studies analyzed in this meta-analysis reflect this view to some extent. The very fact that these studies took the approach they did (in most cases, a relatively short-duration intervention followed by a standardized test) arguably indicates that they tacitly favor a process–product view of teaching.

However, as we indicated in our section on the contested definition of CT, the mere fact that this analysis synthesizes articles that occasionally take a simplistic approach to CT does not imply that we endorse such a view. We regard teaching (and, more specifically, teaching CT) as a complex and multifaceted process, in which there is no magic recipe for the “production of learner success.” There are, however, ingredients that appear to be particularly promising, and it is our task as meta-analysts to highlight these. The techniques that we have identified as being promising (i.e., discussion, authentic situations, mentorship) are not an exhaustive list of promising ingredients, but they are a useful starting point for thinking about the challenges teachers face, as well as for further research efforts in the area of CT. Even if one endorses a highly contextual “quilting bee” approach to CT, as Barbara Thayer-Bacon (2000) does, there is still some usefulness in taking stock of the fact that certain kinds of CT tools seem to be particularly helpful.

A second major objection, to the findings presented above, holds that the results presented do not show that CT can be taught, nor do they show that certain types of interventions tend to facilitate the development of CT. What the results do show, the critics maintain, is that certain pedagogical interventions are associated with better performance on a CT test that measures a narrow band of skills that have only a tenuous link to genuine CT. These critics may hold a more dispositional view of CT or they may hold a view that CT is a complex, multifaceted practice, in line with the position of Thayer-Bacon (2000) that was sketched earlier.

These critics have a point: Although a wide variety of measures are captured by our analysis (including teacher-developed and dispositional measures), there is a potential bias among the measures toward a skills-focused conception of CT, which is, arguably, to borrow from Paul’s (1990) terminology, a relatively weak sense of CT. Yet, even after conceding this point, one could still argue, as we did in the section on the contested definition of CT, that these skills are important potential precursors to a more ambitious and robust form of criticality. To reject these results completely would be tantamount to discarding the good for the sake of a nonexistent and unobtainable better, at least as far as meta-analysis is concerned.

Still, one can easily imagine arguments that reject this line of reasoning as well and hold that a skills approach to CT, “effective” though it may be, is completely counterproductive. Again, regardless of the merits of this critique, we come up against the limitations of what a meta-analysis of education research can show. A meta-analysis is only capable of answering questions that have already been asked in certain very specific kinds of ways, and its claims must always be somewhat modulated. Like a crude early map, a meta-analysis of education research charts terrain that has already been visited many times and provides some modest degree of guidance for future visitors to the area. One will still see plenty of blank spots and indications to the effect that “Here there be dragons.”

Beyond these two serious objections, a number of other smaller questions remain outstanding. Perhaps most significantly, although the three-category scheme revealed some important differences between types of instructional interventions, the analyses of methodological and substantive study features failed to identify factors that explained completely the heterogeneity in effect sizes. Although the findings appear generally robust across educational level, subject area, treatment duration, and type of instruction, we are stymied in our attempts to understand why a minority of the CT instructional interventions were unsuccessful, resulting in lower levels of CT performance than in the control conditions. The frustration of the reviewer is that she/he can only explore factors that are evident in a sufficient number of studies, and code and aggregate those factors. There may, therefore, be factors other reviewers will see that were not seen by us or that the next round of new research will elucidate. In that sense, this review may act as a heuristic for future thought and research efforts.

The links between CT instruction and both content-specific CT outcomes and achievement outcomes make clear that student learning can be positively affected by the integration of CT-linked content. However, there is no guarantee of this being the case, especially if what is to be learned focuses on factual knowledge, even when such knowledge is a prerequisite for deeper, critical understanding. To the extent that there is a perceived conflict among curricular objectives, such a conflict may relate, in part, to why, despite the evidence that it may have a halo effect on other kinds of learning, CT instruction is not more widespread.

An additional outstanding question relates to age. Viewed from the perspective of developmental psychology, the fact that there is no significant difference among age levels in CT is quite surprising. According to at least some theories, young children should lack the developmental capabilities to perform well on certain types of CT tasks. One possibility here is that standardized measures (not to mention the aggregation involved in meta-analysis) simply do not capture these subtle differences particularly well, but another possibility is that young children may be more capable of CT than some have surmised. Regardless, this is a question that merits careful analysis.

Finally, the findings also seem to call for further investigation into the broad instructional strategies that were especially successful. An initial attempt was made to divide these categories into subcategories (see pp. 289–291) and to analyze the results according to them, but the results of this investigation, although intriguing, have been somewhat limited thus far. Some subcategories (e.g., teacher-led whole class discussions, applied problem solving, role-playing) favored the experimental group, but in the cases of many of the subcategories, the number of studies in the subcategory was insufficient to draw conclusive results. This is, in part, an inevitable difficulty in meta-analysis, but further research into more precise classification and alternative taxonomies is nonetheless advisable given the promising findings for the three general instructional strategy categories.

Despite these caveats, we are confident that the empirical evidence we were able to synthesize supports the notion that there are a number of promising teaching strategies for helping students develop CT skills and dispositions. Specifically, there are strong indications that dialogue, authentic instruction, and mentorship are effective techniques for the promotion of this goal. These techniques appear to be particularly effective when combined. Nevertheless, we must also concede there is a great deal more investigation to do in this area, and our tentative prescriptions would benefit from being embedded in robust normative frameworks for the teaching of CT.