Teaching urban youth about controversial issues: Pathways to becoming active and informed citizens

Abstract

Although American schools are required to meet civic education goals of preparing students to become active and informed citizens, high-quality civic opportunities (e.g. service learning and volunteering) are consistently less available to youth of color who are typically enrolled in schools located in high-poverty communities. The purpose of this study is to evaluate efficacy of the Word Generation program to improve students’ self-reported civic engagement (N = 5798) in the context of a randomized trial that was conducted in several middle schools located in a West Coast metropolitan area of the United States. Word Generation is a cross-content literacy program that instructs students to learn academic words, which are embedded in brief passages covering a different controversial issue each week. Participants completed survey items on how often they helped their friends, community, and school, as well as voting interest. Results provide support for the primary research question – participation in the Word Generation program has a significant impact on students’ self-reported civic engagement, but not for voting interest. These results suggest that students’ opportunities to debate on social issues are crucial to envisioning oneself as an active participant in civic affairs.

Keywords

civic education controversial issues teaching civic engagement randomized control trial

More than half of eligible young Americans (aged 18–29 years) did not vote in the 2012 elections, which raises concerns about the political health of this country (The Center for Information and Research on Civic Learning & Engagement (CIRCLE), 2013). Adler and Goggin (2005) explain that fulfilling one’s civic duty to vote is connected to a broader field of civic engagement, which includes the community context (e.g. volunteering and helping a neighbor) and electoral context (e.g. voting, donating money to political candidates, and protesting). As much as family, friends, and media can influence youth to become active in their communities, schools can also play a pivotal role in providing civic learning opportunities to adolescents (Gibson and Levine, 2003). In the landmark report called The Civic Mission of Schools (Gibson and Levine, 2003), practitioners and scholars emphasize that exposing students to controversial issues is critical to their civic engagement development (Carnegie Corporation and CIRCLE, 2003). Controversial issues refer to complex and socially relevant issues (e.g. gun control laws and stem cell research) that can generate high levels of disagreement (Dawson and Venville, 2009). A randomized trial of the Word Generation (WG) program was conducted in several middle schools located in an American urban school district with an ethnically diverse student population. This study examines whether participating in this intervention results in positive effects on students’ self-reported civic engagement.

Literature

Civic engagement among youth

A 2006 report by the CIRCLE published results from civic engagement surveys completed by 1300 American youth aged 15–25 years (Lopez et al., 2006). The report indicates that Hispanic youth lagged behind their White, Asian, and African American counterparts in various civic engagement activities including volunteering for a non-political group, engaging in community solving, and voicing their political opinions (e.g. contacted an elected official and joined a boycott). The 2006 CIRCLE report also suggests that Asian American youth were behind other racial groups in several key areas of civic engagement: volunteering for a charity and discussing current events with friends and family. High-quality civic learning opportunities are also consistently less available to ethnic minority youth (Levinson, 2012). In a study of students’ experiences (N = 2366) in high school social studies classes, students who identified as Asian, Filipino, or Pacific Islander were less likely to report having discussions of controversial issues and freedom to express their ideas in comparison with their Caucasian counterpart (Kahne and Middaugh, 2008). In contrast, students with high educational aspirations (as indicated by intended enrollment in post-secondary school) and high academic grade point average (GPA) were more likely to experience frequent civic learning opportunities. The strength of this study is expanding research so that schools can consider possible solutions to improving civic engagement among youth from ethnically diverse populations.

The Word Generation (WG) program

The Word Generation (WG) program is a cross-content literacy program delivered at the classroom or grade level that instructs students to learn five all-purpose academic words, which are embedded in brief passages covering a different controversial issue each week (Snow et al., 2009). The program features controversial topics based on a broad range of political and science-based controversies, such as whether the government should fund stem cell research or animal testing. As part of the WG program, teachers in the four main content areas – English language acquisition (ELA), social studies, science, and math – present materials related to the controversial issue of the week and explore academic language that is embedded in the curriculum through discourse and writing.

For 15 minutes a day, teachers and students engage in program activities related to vocabulary instructions and classroom discussions (see Figure 1 on sequence of weekly activities). On Monday, the ELA teacher presents a text that starts with a narrative about a girl who was accidentally shot and paralyzed, but for whom stem cell research might offer a cure; the rest of the brief text presents arguments against (e.g., religious objections, use of embryos) and in favor of federal funding of stem cell research (e.g., scientific and medical advances, global competitiveness). On Wednesday, the social studies teacher facilitates debate with the whole class, where students can choose to defend arguments for or against funding stem cell research. The weekly sequence was developed to provide students with recurrent exposure and opportunities to learn about the controversial topics in various subject-specific contexts (Snow et al., 2009). The unique strength of the WG program is focusing on the collective responsibility among social studies, ELA, math, and science teachers to instruct students on controversial issues.

Figure 1.

Sample weekly schedule of the Word Generation program.

Various studies have been conducted on quasi-experimental and experimental trials of the WG program that was implemented in several urban school districts across the United States. A broad range of research topics has been explored, including writing outcomes (Mancilla-Martinez, 2010), vocabulary development (Lawrence et al., 2015), and differentiated effects for English language learners (Hwang et al., 2015). This is the first study to assess whether enrollment in the WG program can improve students’ self-reported civic engagement and voting interest.

Controversial issues as a basis for classroom discussion

Few studies have assessed the effects of using controversial issues in the classroom (Hess and Posselt, 2002; Venville and Dawson, 2010). One study (Venville and Dawson, 2010) examined a 10-week intervention designed to improve students’ argumentation skills by engaging in classroom debates on the moral dilemmas associated with genetic engineering. In the quasi-experimental study, 10th grade students participated in treatment (n = 46) and control (n = 46) conditions. The argumentation course was taught by an experienced Biology teacher who participated in a professional learning session specifically on argumentation. At baseline, students in both groups were comparable on pretest writing assessments of their argumentation skills. The authors found that treatment was associated with a medium effect size (r = .30) on students’ argumentation quality and reasoning skill in their post-test writing assessments. We hypothesize that the WG program can improve discussion quality in the classrooms:

H1. Classrooms in WG schools have higher ratings on discussion quality than those that did not use the WG program.

Exposure to issues-centered curricula can improve students’ civic engagement

Alex Romeo Lin (2015) published a review documenting the last 25 years of research on civic education programs and highlighted four issues-centered curricula – CityWorks, Kids Vote, Project Citizen, and Student Voices – that have been tested in various research contexts (Hartry and Porter, 2004; McDevitt and Chaffee, 2000). Classroom discussions of various controversial topics are usually encouraged as part of the curriculum. Compared to evaluations of the other issues-centered curricula (e.g. Cityworks, Kids Vote and Project Citizen), evaluations on the Student Voices (SV) program are notable for its quasi-experimental design (with comparison groups) that involves multiple research sites (Feldman et al., 2007; Pasek et al., 2006; Syvertsen et al., 2009). Syvertsen et al. (2009) examined the effects of the SV program on high school students who enrolled in social studies classrooms that participated in treatment (n = 48) and control conditions (n = 32). Treatment (n = 776) and control (n = 894) students completed pre- and post-test surveys on how often they engaged in community volunteering to measure civic engagement, as well as their voting interest in the national election. Based on regression models that controlled for students’ grade level, race, socioeconomic status, and political interest, the SV program was found to have a 0.03 and 0.21 effect size on students’ self-reported civic engagement and voting interest, respectively. Although these evaluations provide evidence that issues-centered curricula can improve civic engagement among youth, the three evaluation studies on the SV program were conducted in the context of a quasi-experimental design, which opens the possibility that teachers implementing the SV program were already inclined or experienced in using the program than those in the control group:

H2. Participating in the WG program will impact students’ self-reported civic engagement and voting interest.

Assessing ethnic minority youth’s experience with issues-centered curricula

This section reviews current understandings about how students of color may respond to issues-centered curricula in schools. Compared to youth from other ethnic backgrounds, classroom supports for discussions of controversial issues is particularly important for Hispanic youth. One study (Torney-Purta et al., 2007) examined school and neighborhood factors that contribute to the civic engagement gap between Hispanic (n = 380) and non-Hispanic ninth graders (n = 2373). Multilevel models that controlled for students’ immigration status and political interest provide evidence that Hispanic students lagged behind their counterparts in terms of civic knowledge and voting interest. However, the degree that Hispanic students perceived an open classroom climate to discuss controversial topics virtually closed the civic engagement gap with their non-Hispanic counterparts. This analysis will consider estimating the extent that treatment effects differ among Asian, African American, and Hispanic students.

H3. Non-White students will benefit more from the WG program than White students.

Methods

Research design

One urban school district located in the West Coast of the United States participated in an evaluation study of the WG program. The Strategic Education Research Partnership (SERP) recruited leaders from the school district, who then invited their school leadership teams to participate in the randomized study. School district leaders informed parents and students about the study. Students obtained parental consent to participate and release information on their academic and psychological performances for research purposes. In the first phase of the randomization process, state accountability data were used to rank schools on several school-level variables based on student enrollment and prior academic achievement. Each school received general composite scores computed from state accountability data that were used to rank each school on several school-level variables based on the percentage of students identified as ethnic minorities, low-income status, English language learners, as well as the school’s prior academic achievement. The composite scores were then used to determine each school’s ranking. Each sequential pair of schools was assigned into dyads and the final process of randomization occurred within each dyad. As a result, six schools were assigned to the treatment and six schools into the control conditions. Teachers in the treatment conditions were required to implement the WG program; whereas teachers in the control conditions did not use the WG program and participated in “business as usual” instructions (Lawrence et al., 2015).

Preliminary analysis indicates a consistently high level of implementation during specific times that observers were present; however, students’ completion rates of workbooks varied and tended to decrease precipitously toward the end of the year. Observers found that teachers implemented the Day 1 program activities in a “smooth and efficient” manner in 83% of the 96 observed class periods. In 89% of the observation periods, observers found that teachers and students brought their workbooks to class. Students’ workbook completion rate varied among schools from 44.4% to 89.9%. Furthermore, workbook completion rate decreased over time from the start in Week 1 (82%), midpoint at Week 12 (69%), and at the end in Week 24 (57%). These preliminary analyses suggest that program implementation was quite consistent in the beginning, but may be noticeably inconsistent throughout the school year.

Participants

Although the study analyzes an equal number of schools that were randomly assigned to treatment and control conditions, the analytic sample contains more treatment students (n = 3488, 60.3%) than control (n = 2298, 39.7%) students (see Table 1). Several schools experienced changed leadership or circumstances, which may have had an adverse effect on their participation in the study. For these reasons, there were more challenges in data collection on civic engagement in comparison with reading and vocabulary measures.

Table 1.

Grade level contributions by treatment conditions (analytic sample).

	Sixth graders		Seventh graders		Eighth graders		All students
	n	%	n	%	n	%	N	%
Control schools
Duffie Oak	154	8.2%	62	3.3%	136	6.7%	352	6.1%
Evergreen	56	3.0%	20	1.1%	35	1.7%	111	1.9%
Honeysuckle	81	4.3%	124	6.6%	230	11.3%	436	7.5%
Maple	244	13.1%	205	10.9%	216	10.6%	667	11.5%
Rosemary	49	2.6%	202	10.8%	187	9.2%	438	7.6%
Vineland	97	5.2%	87	4.6%	110	5.4%	294	5.1%
Total	681	36.5%	700	37.3%	914	44.8%	2298	39.7%
WG schools
Apple	362	19.4%	352	18.8%	337	16.5%	1052	18.2%
Arbola	269	14.4%	239	12.7%	248	12.2%	756	13.1%
Flower Square	113	6.1%	71	3.8%	96	4.7%	280	4.8%
Hemlock	60	3.2%	58	3.1%	45	2.2%	163	2.8%
Moon	98	5.2%	134	7.1%	123	6.0%	355	6.1%
Palm	284	15.2%	322	17.2%	276	13.5%	882	15.2%
Total	1186	63.5%	1176	62.7%	1125	55.2%	3488	60.3%
All schools	1867	32.2%	1876	32.4%	2039	35.2%	5786	100.0%

WG: Word Generation.

Pseudonyms were used to protect the schools’ identity.

Table 2 indicates that the sample analyzed in this study has slightly more females (52.3%) than males (47.7%). Nearly two-thirds of the participants were from low-income background (63.9%), as indicated by their eligibility in the Free and Reduced Lunch (FRL) program at their respective schools. A substantial number of students come from extremely diverse ethnic background: Asian (54.2 %), Hispanic (19.4%), and African American (6.9%). Among language minority students, there were quite a number of students classified as Redesignated (44.2%) and Limited English Proficient (12.6%).

Table 2.

Demographic variables of students by treatment status.

	Control schools		Word Generation schools		All schools
	n	%	n	%	N	%
Female	947	49.0%	1565	48.3%	2702	52.3%
Special Education	172	8.9%	262	8.1%	434	8.4%
Free and Reduced Lunch Eligible	1340	69.4%	1961	60.6%	3301	63.9%
Gifted and Talented Education Program	775	40.1%	1528	47.2%	2303	44.6%
Ethnic/Race
Asian	1118	57.9%	1682	52.0%	2800	54.2%
Southeast Asian	92	4.8%	129	4.0%	221	4.3%
African American	138	7.1%	217	6.7%	355	6.9%
Hispanic	394	20.4%	610	18.8%	1004	19.4%
White	110	5.7%	370	11.4%	480	9.3%
Language Status
Limited English Proficient (LEP)	303	15.7%	347	10.7%	650	12.6%
Redesignated (RFEP)	917	47.5%	1367	42.2%	2284	44.2%
English proficient	526	27.2%	1164	36.0%	1690	32.7%
Fluent	184	9.5%	356	11.0%	540	10.4%
Primary Language
English	806	41.7%	1549	47.9%	2355	45.6%
Mandarin	568	29.4%	949	29.3%	1517	29.3%
Spanish	262	13.6%	400	12.4%	662	12.8%
Other Language	296	15.3%	339	10.5%	635	12.3%

Language Status – not listed were seven students who were categorized as “pending”; among Asian students, 2240 (78%) were identified as Chinese; primary language measured by the most frequent language used at home; 629 (63.5%) of Hispanic students indicate Spanish as their primary language; 5169/5786 (89.3%) of the students had demographic data.

Procedures

Classroom observations were conducted in all major content-area classrooms (ELA, math, science, and social studies) across treatment and control schools. Observers attended training sessions that instructed on using standardized codes to rate conversational moves observed among students and teachers in video footage of classroom discussion. In the classroom evaluation phase, participating teachers were randomized and a schedule was created identifying specific observation periods to be conducted on a given day.

Prior to test administration, we conducted cognitive lab interviews with middle school students enrolled in the school district to develop meaningful survey items appropriate to the context of youth from diverse population. The data were analyzed to develop appropriate survey items on civic engagement. During the two testing days in May 2012, students in treatment and control conditions completed a number of civic engagement measures. At the time of assessment, students in the treatment group were essentially at the point of fully completing the program.

Students also completed the Gates-MacGinitie reading comprehension assessment in Fall 2011 and Spring 2012. The test is a 48-item assessment used to measure students’ overall reading comprehension. The test includes 14 short multiple-paragraph passages from a wide range of domains including narrative, autobiographical, biology, and history to measure comprehension at the passage level. For the purposes of evaluating the WG program, the school district provided demographic data with information on students’ gender, grade level, ethnicity, school status (language minority status, GATES, and special education), and eligibility in the FRL status program. Contribution rates of demographic data were quite high among students (n = 5169, 89.3%).

Measures

Classroom observations

Support for participation – Classroom observers rated the extent to which the teacher created a well-ordered environment that enabled engagement with lesson content and participation in discussions. Based on a 3-point scale, observers gave a rating of 1 for classrooms lacking a well-ordered environment and 3 for classrooms where students were consistently engaged and minimally distracted. In observed classrooms with more than one observer, 13 out of the 13 observations received exact matches in this category.

Student engagement – Classroom observers rated the number of students participating and attending to classroom discussions. Based on a 3-point scale, observers gave a rating of 1 for classrooms where less than a quarter of students participated in discussion and a 3 for classrooms with at least half of the students participating in discussion. The interrater reliability of classrooms with more than one observer was quite high (κ = 0.83). All but one of the 13 observations received exact matches in this category.

Teacher talk moves – Classroom observers rated teachers’ frequency of initiating open-ended questions and follow-up questions requiring students to explain their thinking. Based on a 5-point scale, observers rated classrooms as 1 for those where teachers mainly posed closed-ended questions that imply an evaluation on students to add certain information. Observers gave a rating of 5 for classrooms where teachers typically initiate open-ended questions that encourage students to construct and justify their claims. All but 1 of the 13 observations received exact matches in this category.

Substantive contributions – Classroom observers rated students’ intellectual contributions during classroom discussions. Based on a 5-point scale, observers rated classrooms as 1 for those where students mainly provided cursory or surface-level understandings of the material. Observers gave a rating of 5 for classrooms where a majority of students contributed ideas that reflect thoughtful consideration of how their claims may be reconciled with competing views. The interrater reliability of classrooms with more than one observer was also quite high (κ = 0.71), with 11 out of the 13 observations getting exact matches in this category.

Student surveys

Civic engagement – On the surveys, students responded to the following three items, which asked, “How often do you help your (1) friend, (2) school, and (3) community?” Each item was answered based on a 5-point Likert scale of (1) never, (2) rarely, (3) sometimes, (4) often, and (5) always.

Voting interest – On the surveys, students responded to the item “How interested are you in voting for the President when you get older?” The item was answered based on a 5-point Likert scale of (1) not at all interested, (2) slightly interested, (3) somewhat interested, (4) quite interested, and (5) extremely interested.

Political interest – On the surveys, students responded to two items assessing the degree to which they are interested in politics and events going on in the world. Each item was answered on a 5-point Likert scale of (1) not at all interested, (2) slightly interested, (3) somewhat interested, (4) quite interested, and (5) extremely interested.

District data

For all analyses, demographic variables were included for students’ treatment/control assignment, ethnic-racial identity, gender, and socioeconomic status. Eligibility in the FRL program at school was used as a proxy indicating students from low-income background. Regarding students’ designated school status, students were coded as whether they participated in a Special Education and Gifted and Talented Education (GATE) program. Language minority students consist of students receiving additional ELA at school (Limited English Proficiency) and students who achieved a minimal English proficiency and do not have additional English support (Redesignated). Students’ academic ability measure is based on students’ pretest scores (Fall 2011) on the Gates-MacGinitie reading comprehension.

Data analytic plan

This section provides an overview of how the data were prepared and then discusses the analytical strategies used to answer each of the research questions.

Data preparation

This study replicates the approach used in a past WG evaluation (Lawrence et al., 2015), which involves creating a composite of the four rubrics used to measure the quality of classroom discussion and later transformed into a weighted school-level discussion quality ratings. This process facilitates comparisons of scores across classrooms that accounts for different content areas observed:

Classroom discussion quality rating – Rubric scores were divided by the number of possible points available in that category, in order to place the 3- and 5-point ratings on the same scale. Adding these scores equaled a maximum of 4 points (CLASSROOM_DISCUSSION_QUALITY_RATING = PARTICIPATION/3 + ENGAGEMENT/3 + TALK_MOVES/5 + SUBSTANTIVE/5). Classroom discussion quality rating ranged from 1.07 to 3.80 (out of a possible total of 4.00).

Standardized discussion quality rating – Next, we standardized the overall classroom discussion quality rating to account for the fact that average CLASSROOM_DISCUSSION_QUALITY_RATING discussion scores varies systematically across content area. For example, the mean of CLASSROOM _DISCUSSION_QUALITY_RATING scores of all observations of math class is zero, with a standard deviation (SD) of 1 (similar to all other content areas). This standardized discussion quality rating was then used to create the weighted school-level discussion quality rating (described in the next section).

Weighted school-level discussion quality ratings – School-level discussion quality ratings (WEIGHTED_CLASSROOM_DISCUSSION_QUALITY_RATING) were created for each school by averaging the zCLASSROOM_DISCUSSION_QUALITY_RATING scores for the observations completed at the school level. This transformation facilitates comparisons of scores across classrooms despite differences in the number of content areas observed. The strength of this approach is to avoid penalizing schools with a higher proportion of observed science and math courses, where there tends to be a higher focus on lecture and less emphasis on discussion.

Civic engagement and political interest measure – The three civic engagement survey items (see Table 3 for factor loadings) were used to develop the scaled score (CIVIC_ENGAGEMENT) with an eigenvalue of 1.05. The civic engagement score was standardized to have a mean of zero and standard deviation of one to assist with interpretation. Two items were developed into a scaled score for political interest (POLITICAL_INTEREST) with a 0.61 factor loading.

Table 3.

Components of the civic engagement scaled score.

	Mean (0–5)	SD	Factor loading (varimax rotation)
How often do you help your friends?	3.94	0.90	0.54
How often do you help your school?	2.91	0.94	0.55
How often do you help poor people in your city?	2.52	1.01	0.57

SD: standard deviation.

Analytic strategy

Hypothesis 1 is tested by analyzing classroom observation data used to determine whether there are significant differences in classroom discussion between treatment and control conditions. Before discussing the statistical test used to answer this research question, it is important to clarify the approach of analyzing classroom discussion quality at the observation level similar to a previous WG evaluation (Lawrence et al., 2015). At the core of implementing the WG program are grade level teaching teams of ELA, science, social studies, and math teachers. However, there was an uneven distribution of observations conducted across content areas. For example, if a teaching-team observation score contained only social studies with no ratings for math and science, then this score could not be generalized to the other content areas classrooms associated with this particular teaching team. Thus, the analysis considers analyzing classroom discussion quality at the observational level, rather than teaching-team level.

Hypothesis 2 is answered by determining whether participating in the WG program has a significant impact on students’ self-reported civic engagement and voting interest. This analysis utilizes regression analyses with adjusted standard errors that accounts for clustering at the teaching-team grade level. Including a cluster option (“vce”) in the regression command (STATA software) assumes that observations are clustered within grade level teaching teams, which takes into account that observations may be correlated (Murnane and Willett, 2010). The hypothesized regression model is:

\begin{matrix} CIVIC ENGAGEMEN T_{i} = β_{0} + β_{1} TREA T_{i} + β_{2} FEMAL E_{i} + β_{3} GRADE_7_{i} + β_{4} GRADE_8_{i} \\ + β_{5} ASIA N_{i} + β_{6} BLAC K_{i} + β_{7} HISPANI C_{i} + β_{8} OTHER_RAC E_{i} \\ + β_{9} FREE_A N D_REDUCED_LUNCH_STATU S_{i} + \\ + β_{10} L E P_{i} + β_{11} RFE P_{i} + β_{12} GAT E_{i} + \\ + β_{13} SPECIAL_EDUCATIO N_{i} + β_{14} POLITICAL INTERES T_{i} + \\ + β_{15} ACADEMIC ABILIT Y_{i} + ε_{i} \end{matrix}

where CIVIC ENGAGEMENT_i is the student-reported civic engagement measure of the ith student. β₀ are the regression intercept and slope parameters, and ε_i are the residuals. The slope parameter β₁ TREAT_i represents the average treatment effect – the average difference in the civic engagement outcomes between students who participated in the WG program and those who did not. Preliminary analysis conducted on an imputed dataset that accounts for patterns of data missingness did not yield significantly different results; thus, the non-imputed dataset is presented as the final results of the study.

Hypothesis 3 is tested by analyzing interaction terms added to the model to determine whether students’ ethnic identification moderates the relationship between program participation and self-reported civic engagement. The hypothesized model consists of determining whether the moderator (racial ethnicity) alters the relationship strength between the predictor (WG program participation) and the outcome (civic engagement). A moderator effect is present if the interaction term is found to be statistically significant (Bennett, 2000; Frazier et al., 2004).

Results

H1. Classrooms in WG schools have higher ratings on discussion quality than those that did not use the WG program.

Table 4 presents classroom discussion ratings in the 139 observations sampled across content-area classrooms. Although control classrooms received higher discussion scores in science (Δ = 0.24, p = n.s.), treatment classrooms had higher discussion scores in Math (Δ = 0.26, p = n.s.) and ELA (Δ = 0.33, p = n.s.) than control classrooms. In social studies, treatment classrooms (M = 2.71, SD = 0.66) had significantly higher ratings than control classrooms (M = 2.21, SD = 0.56), t(34) = −2.02, p < .05. The next analysis examines the overall classroom discussion ratings.

Table 4.

Classroom discussion quality rating by treatment status and content-area observed with t-test results and estimated effect sizes.

	Word Generation schools			Control schools			All schools			Difference (Word Generation − Control schools	Effect size (Cohen’s d)
	n	M	SD	n	M	SD	n	M	SD	Difference (Word Generation − Control schools	Effect size (Cohen’s d)
ELA	20	2.69	0.45	16	2.37	0.58	36	2.55	0.53	+0.33
Math	17	2.41	0.59	17	2.15	0.59	34	2.28	0.62	+0.26
Science	18	2.23	0.69	16	2.48	0.68	34	2.35	0.69	+0.24
Social Studies	18	2.71	0.66	17	2.21	0.56	35	2.46	0.65	+0.49*
Overall (Non-Weighted)	73	2.52	0.62	66	2.29	0.61	139	2.41	0.63	+0.22*	0.35
Overall (Weighted)	73	0.17	0.96	66	−0.19	0.99	139	0.00	0.99	+0.37*	0.37

SD: standard deviation; ELA: English language acquisition.

Discussion quality rating is an average score (out of a possible total of 4) based on (1) teacher talk moves (out of 5), (2) substantive contributions (out of 5), (3) support for participation (out of 3), and (4) student engagement (out of 3).

p < .05; **p < .01; *** p <0.001.

Hypothesis 1 is tested by analyzing the weighted total scores of classroom discussion in treatment and control schools. Classrooms using the WG program received higher overall weighted scores on classroom discussion quality than those in the control schools. Based on t-test results, the difference of 0.37 between treatment (M = 0.17; SD = 0.62) and control (M = −0.19; SD = 0.99) was found to be statistically significant, t(137) = −2.05, p < 0.01. The difference in average scores was used to calculate the average treatment effect (Cohen’s d = 0.37). The results provide support for the Hypothesis 1: overall classroom discussion scores, on average, were higher in treatment than control schools.

H2. Participating in the WG program will impact students’ self-reported civic engagement and voting interest.

The first section is descriptive analyses based on comparing mean scores of students’ civic engagement (Table 5) and voting interest (Table 6) between those in treatment and control schools. Among sixth and seventh graders, students in the treatment schools reported significantly higher civic engagement scores than those in control schools (see Table 5). In regards to students across different ethnic identification, Asian and African American students who participated in the WG program were found to have higher self-reported civic engagement than their counterparts in the control schools. According to Table 6, seventh and eighth students who participated in the WG program reported higher voting interest than their counterparts in the control schools. African American students reported higher self-reported voting interest than their counterparts in the control schools; this was not found to be the case for Asian and Hispanic students. White students in treatment schools reported higher civic engagement and voting interest in comparison with their counterparts in control schools. The next analysis is used to answer the study’s research question, which consists of estimating the WG program’s effects on students’ self-reported civic engagement while taking into account possible confounding variables.

Table 5.

Comparison of students’ self-reported civic engagement standardized scores by treatment status organized by grade level, gender, ethnicity, and school status with t-test results indicating difference.

	Word Generation schools			Control schools			Overall sample			Difference (Word Generation − Control schools)
	n	Mean	SD	n	Mean	SD	n	Mean	SD	Difference (Word Generation − Control schools)
Grade level
Sixth Grade	1170	0.07	0.65	829	−0.06	0.68	1999	0.02	0.67	+0.135***
Seventh Grade	1169	0.04	0.66	722	−0.04	0.66	1891	0.01	0.66	+0.081**
Eighth Grade	1112	−0.01	0.66	949	−0.02	0.63	2061	−0.01	0.64	+0.005
Female	1855	0.14	0.62	1447	0.06	0.63	3302	0.11	0.62	+0.074***
Race
White	616	0.09	0.67	539	−0.03	0.68	1155	0.03	0.68	+0.116***
Asian	1937	0.02	0.63	1604	−0.03	0.63	3541	0.00	0.63	+0.058***
Hispanic	875	−0.03	0.70	819	−0.06	0.70	1694	−0.04	0.70	+0.033
African American	493	0.11	0.68	558	0.00	0.70	1051	0.05	0.69	+0.104***
Special Education	587	−0.08	0.75	540	−0.03	0.74	1127	−0.05	0.75	+0.053
FRL	2192	0.02	0.67	1846	−0.04	0.66	4038	−0.01	0.67	+0.059***
GATE	1795	0.05	0.62	1252	−0.03	0.62	3047	0.02	0.62	+0.079***

SD: standard deviation.

Civic engagement is based on a 3-item measure relating to students’ reported frequency of helping their friend, school, and community on a Likert scale of 1 “rarely” to 5 “always”; FRL refers to free and reduced lunch status.

p < .05; **p < .01; *** p < 0.001.

Table 6.

Comparison of students’ voting interest (unstandardized scores) by treatment status organized by grade level, gender, ethnicity, and school status with t-test results.

	Word Generation schools			Control schools			Overall sample			Difference (Word Generation − Control schools)
	n	Mean	SD	n	Mean	SD	n	Mean	SD	Difference (Word Generation − Control schools)
Grade level
Sixth Grade	1163	3.21	1.34	821	3.29	1.31	1984	3.24	1.33	−0.0771
Seventh Grade	1163	3.00	1.33	720	2.96	1.37	1883	2.99	1.35	+0.041
Eighth Grade	1109	3.08	1.30	945	3.02	1.27	2054	3.05	1.29	+0.058
Female	1846	3.16	1.30	1440	3.14	1.33	3286	3.15	1.31
Race
White	611	3.37	1.36	536	3.15	1.39	1147	3.27	1.38	+0.219***
Asian	1929	2.99	1.29	1595	3.00	1.29	3524	3.00	1.29	−0.011
Hispanic	870	3.13	1.37	812	3.07	1.36	1682	3.10	1.37	+0.051
African American	489	3.31	1.33	554	3.17	1.40	1043	3.24	1.37	+0.145*
Special Education	535	3.18	1.37	581	3.02	1.39	1116	3.09	1.39	+0.160*
FRL	2184	3.05	1.32	1837	3.07	1.32	4021	3.06	1.32	−0.012
GATE	1787	3.16	1.31	1245	3.13	1.30	3032	3.15	1.31	+0.021

SD: standard deviation.

Voting interest is a single item measure that reflect interest in voting when student reaches legal age on a scale of 1 “not at all interested” to “extremely interested”; FRL refers to free and reduced lunch status.

p < .05; **p < .01.

Table 7 presents results from regression analyses with predicted effects on students’ self-reported civic engagement (Outcomes 1A-1E) and voting interest (Outcomes 2A-2E). The table presents models with the following characteristics: demographic and school covariates (Model A), covariates with treatment variable (Model B), covariates with political interest (Model C), covariates with treatment and political interest (Model D), and inclusion of interactions (Model E).

Table 7.

Regression models predicting students’ self-reported civic engagement (scaled score) and voting interest (single measure) with standardized coefficients and adjusted standard errors in parentheses.

Variables	Civic engagement					Voting interest
	1A	1B	1C	1D	1E	2A	2B	2C	2D	2E
Treatment		0.067* (0.03)		0.075* (0.03)	0.083 (0.08)		−0.087 (0.06)		−0.058 (0.04)	−0.093 (0.06)
Grade 7^a	0.034 (0.05)	0.034 (0.04)	0.034 (0.05)	0.035 (0.04)	0.035 (0.04)	−0.270** (0.08)	−0.271*** (0.07)	−0.161** (0.05)	−0.161*** (0.04)	−0.161*** (0.04)
Grade 8	−0.039 (0.05)	−0.032 (0.04)	−0.022 (0.05)	−0.015 (0.04)	−0.015 (0.04)	−0.193* (0.07)	−0.201** (0.07)	−0.148** (0.05)	−0.153** (0.05)	−0.151** (0.05)
Female^b	0.242*** (0.02)	0.243*** (0.02)	0.257*** (0.02)	0.258*** (0.02)	0.257*** (0.02)	0.109** (0.03)	0.108*** (0.03)	0.160*** (0.03)	0.160*** (0.03)	0.159*** (0.03)
FRL	−0.029 (0.02)	−0.027 (0.02)	−0.027 (0.02)	−0.025 (0.02)	−0.023 (0.02)	0.005 (0.03)	0.001 (0.03)	0.007 (0.03)	0.005 (0.03)	0.004 (0.03)
GATE	−0.027 (0.02)	−0.027 (0.02)	−0.028 (0.02)	−0.028 (0.02)	−0.028 (0.02)	0.034 (0.04)	0.034 (0.04)	0.032 (0.05)	0.032 (0.05)	0.032 (0.05)
Special Education	−0.133* (0.06)	−0.131* (0.05)	−0.139* (0.06)	−0.137* (0.06)	−0.136* (0.05)	−0.094 (0.10)	−0.096 (0.10)	−0.119 (0.10)	−0.12 (0.10)	−0.124 (0.10)
Race^c
Asian	−0.043 (0.03)	−0.04 (0.03)	−0.016 (0.03)	−0.013 (0.03)	−0.005 (0.07)	−0.352*** (0.07)	−0.356*** (0.07)	−0.266*** (0.06)	−0.269*** (0.06)	−0.309*** (0.08)
Hispanic	−0.103* (0.04)	−0.102* (0.04)	−0.062 (0.04)	−0.061 (0.04)	−0.038 (0.08)	−0.156* (0.07)	−0.157* (0.07)	−0.019 (0.06)	−0.02 (0.06)	−0.019 (0.08)
Black	0.084 (0.05)	0.089 (0.05)	0.064 (0.04)	0.071 (0.04)	0.021 (0.08)	0.162 (0.09)	0.155 (0.09)	0.101 (0.07)	0.095 (0.07)	0.072 (0.12)
Language minority^d
LEP	−0.068 (0.04)	−0.063 (0.04)	−0.124** (0.04)	−0.118** (0.04)	−0.119** (0.04)	0.059 (0.09)	0.052 (0.09)	−0.112 (0.08)	−0.117 (0.08)	−0.117 (0.08)
RFEP	−0.023 (0.02)	−0.02 (0.02)	−0.032 (0.02)	−0.028 (0.02)	−0.028 (0.02)	−0.124** (0.04)	−0.128** (0.04)	−0.139** (0.04)	−0.142** (0.04)	−0.142** (0.04)
Academic ability	0.008 (0.01)	0.005 (0.01)	−0.001 (0.01)	−0.005 (0.01)	−0.005 (0.01)	0.125*** (0.03)	0.129*** (0.03)	0.099*** (0.02)	0.102*** (0.02)	0.102*** (0.02)
Political interest			0.172*** (0.01)	0.172*** (0.01)	0.172*** (0.01)			0.586*** (0.02)	0.586*** (0.02)	0.586*** (0.02)
Asian × Treatment					−0.012 (0.07)					0.062 (0.09)
Hispanic × Treatment					−0.038 (0.09)					−0.008 (0.10)
Black × Treatment					0.09 (0.08)					0.034 (0.15)
Intercept	−0.013 (0.06)	−0.062 (0.06)	−0.047 (0.07)	−0.102 (0.07)	−0.108 (0.10)	3.451*** (0.06)	3.514*** (0.08)	3.329*** (0.04)	3.371*** (0.05)	3.395*** (0.06)
V	5149	5082	5082	5082	5082	5075	5075	5075	5060	5060
Adjusted R²	0.05	0.053	0.123	0.123	0.124	0.037	0.038	0.228	0.229	0.229

Standard error adjusted for 36 teaching-team clusters (in parentheses); Coefficients for Other-Non-White, Southeast Asians, Primary Language status not shown for the sake of space; FRL refers to eligibility in the Free and Reduced Lunch program; Academic ability based on Gates-MacGinitie reading comprehension; LEP refers to Limited English Proficient; RFEP refers to Redesignated.

Sixth graders used as reference group.

Male used as reference group.

White students used as reference group.

English proficient students used as reference group.

p < .05; **p < .01; ***p < .000.

Model 1A indicates that students identified as Hispanic are less likely to report being civically engaged (β = −0.103, p < .05) than their Caucasian counterparts. Model 1B indicates that treatment has a main effect on students’ self-reported civic engagement. In the next model, adding political interest improved the model fit (R²) from .05 to .12. Overall, WG has a significant positive effect (β = 0.075, p < .05) on students’ self-reported civic engagement while controlling for students’ gender, grade level, and race. For interpretation purposes, participation in the WG program is associated with a roughly 0.08 SD increase in civic engagement.

Models 2A–2E measure the extent to which treatment and various covariates are related to students’ voting interest. Students identified as Asians and Hispanics report significantly lower interest in voting than their Caucasian counterparts. On a side note, Academic ability and political interest is strongly related to students’ voting interest. Model 2D indicates that the WG program has no effect on students’ voting interest. In sum, these results support mixed support of Hypothesis 2: WG program improved students’ self-reported civic engagement, but not voting interest.

H3. Non-White students will benefit more from the WG program than White students

According to Table 7, Model 1E predicts students’ self-reported civic engagement and includes the interaction terms between treatment and various indicators of students’ ethnic background. No significant interactions were found between treatment and all ethnic identifications (i.e. Asian, African American, and Hispanic). Model 2E predicts students’ voting interest and also includes the same interaction terms examined in Model 1E. No significant interactions were found between treatment and ethnic identifications. These findings do not support Hypothesis 3 that Asian, African American, and Hispanic students gain additional benefits on their self-reported civic engagement and voting interest.

Discussion

Increasingly concerned about the national declines in civic engagement among adolescents, experts in the civic education field have been looking for innovative ways of connecting students with their school, community, and beyond (Carnegie Corporation and CIRCLE, 2003). Whereas past evaluations found that comprehensively designed curricula incorporating a broad range of civic activities can improve students’ civic engagement (Feldman et al., 2007; McDevitt and Chaffee, 2002), a modest intervention primarily focused on literacy development can achieve similar goals. Trained observers generally found higher levels of classroom discussion quality in classrooms using the WG program in comparison with those in control schools. More importantly, this study provides support for the central research question that participating in the WG program was effective in improving student’s self-reported civic engagement, though not for voting interest. These findings are particularly noteworthy because program effects were supported in the context of a randomized trial. Further exploration of these treatment effects indicate that certain ethnic groups did not gain additional benefits. This section provides implications on how the findings from this study contribute to relevant theories and research.

This study found that implementing the WG program led to more active and enriching discussions in classrooms than those in the control schools. Treatment classrooms had higher classroom discussion scores in math, ELA, and social studies, although not for science courses. These findings suggest that controversial issues can motivate classroom discussion, but may depend on the content-area context in which it is used. Further inquiry is needed to examine how classroom discussions of controversial issues are approached differently in science courses. Nonetheless, these findings provide promising evidence that modest school support of an issues-centered curricula can make a difference in improving classroom discussion quality.

Results from the study support past research showing that students identified as Hispanic and Asian are less inclined to engage in civic activities (Lopez et al., 2006). These findings affirm the need to help youth from these populations to improve their exposure to civic learning opportunities. Fortunately, there is support for the central research question that the WG program can improve students’ self-reported civic engagement. The WG program instructs students to engage in classroom activities that build on learning and engaging in discussions of controversial issues. Controversial issues play a critical role in informing adolescents about pressing issues that are relevant to their communities and beyond (National Council for the Social Studies (NCSS), 2007). Although the WG program does not explicitly support or provide students with community action tools to address social issues, the program exposes students to controversial issues on a daily basis that helps them develop social awareness. In turn, these program activities can help students build confidence in their capacity to become active citizens. One explanation is that controversial issues have a mobilizing function that helps students understand not only the significance but also the procedural knowledge associated with taking actions to resolve community issues (Scheufele, 2002).

Despite positive findings on the WG program’s impact on civic engagement, the program does not seem to have any impacts on students’ voting interest. These results suggest that getting younger adolescents (middle school students) to be interested in voting may be less feasible, in comparison with older adolescents (high school students). According to Hooghe and Wilkenfeld (2008), older adolescents are closer to the legal voting age and more likely to receive concrete information about voting registration procedures in their social studies courses. Despite being enrolled in the WG program, students may not be aware of when and where to vote because voting registration procedures are not usually communicated until high school (Callahan et al., 2008). Nonetheless, this finding considers how the WG program may be limited in its capacity to improve students’ voting interest.

Although the WG program was found to have positive effects on students’ self-reported civic engagement, students from particular ethnic minority backgrounds did not seem to gain additional benefits. This non-significant finding should not be interpreted to suggest that the program does not work for students from diverse backgrounds (Huey and Polo, 2008; Miranda et al., 2005). Rather, these results suggest that the WG program may have equally positive effects on students identified as ethnic minorities and Caucasian (Lochman and Wells, 2003). However, these findings suggest that possible modifications may be needed to ensure that the WG program is culturally adaptive and sensitive to the learning needs of ethnic minorities. (Huey and Polo, 2008).

Limitations

Although the WG program was observed to have a number of positive outcomes on students’ self-reported civic engagement, several limitations in this study are acknowledged. First, pretest surveys were not administered to students. Pretest data are useful in understanding students’ self-reported civic engagement at baseline that also providing more precise treatment effects. The study was also limited to examining a broad civic engagement measure that lacks specificity on whether students participated in certain civic activities, such as contacting public officials, donating money for specific causes, and volunteering in community services (e.g. soup kitchen). Another limitation was relying on students’ self-reported civic engagement. Although students’ self-reported civic engagement is considered a modestly reliable predictor of their future behaviors (Kahne and Middaugh, 2008), it is still possible that students may inaccurately report their civic engagement (McDevitt and Kiousis, 2006). Students may be overconfident in how often they help their communities because of biases based on social desirability or trying to appear as good citizens (Bouffard-Bouchard et al., 1991). Finally, there are other instruments that can be used to assess a broad range of civic outcomes. For example, a standard civic knowledge assessment relies on a battery of items testing whether students know how a bill becomes a law and the three branches of government (Galston, 2007). Addressing these limitations is critical to estimating more precise treatment effects as well as improving the generalizability of this study’s findings.

Conclusion and implications

In summary, the study provides evidence that exposing students to learning and discussing controversial issues through the WG program can positively impact students’ self-reported civic engagement. Civic engagement is important for adolescents in the broad context of developing awareness and responsibility of improving the community, as well as eventually fulfilling one’s civic duty to participate and vote in the national elections (Althof and Berkowitz, 2006; Carnegie Corporation and CIRCLE, 2003). This study makes a number of contributions that inform how schools can develop teaching strategies and curricula content used to improve students’ civic engagement.

A serious concern is that students enrolled in diverse and urban school districts have fewer opportunities to practice civic activities in comparison with students enrolled in more privileged schools (Ballard et al., 2014; CIRCLE, 2013). For example, schools located in high-poverty areas have fewer opportunities for service learning and community service outreach (Hart and Atkins, 2002). One reason explaining this trend is that school administrators are constantly pressured to ensure that students excel on standardized math and ELA assessments, which leaves less attention to improving social studies courses (Burroughs et al., 2005). The current study reports that ELA, science, and math teachers can also share in the responsibility of helping students to becoming informed and active citizens. Indeed, the implications of this study suggest that the lack of civic learning opportunities in highly diverse urban school districts can be addressed by considering a modest intervention requiring a reasonably low amount of instructional time and resources.

Footnotes

Funding

This work was supported by Grant Number R305A090555, Word Generation: An Efficacy Trial from the Institute of Educational Sciences, US Department of Education (Catherine Snow, PI).

References

Adler

Goggin

(2005) What do we mean by “civic engagement”? Journal of Transformative Education 3(3): 236–253.

Althof

Berkowitz

(2006) Moral education and character education: their relationship and roles in citizenship education. Journal of Moral Education 35(4): 495–518.

Ballard

Caccavale

Buchanan

(2014) Civic orientation in cultures of privilege: What role do schools play? Youth & Society 47 (1): 70–94.

Bennett

(2000) Mediator and moderator variables in nursing research: Conceptual and statistical differences. Research in Nursing & Health 23(5): 415–420.

Bouffard-Bouchard

Parent

Larivee

(1991) Influence of self-efficacy on self-regulation and performance among junior and senior high school age students. International Journal of Behavioral Development 14(2): 153–164.

Burroughs

Groce

Webeck

(2005) Social Studies education in the age of testing and accountability. Educational Measurement: Issues and Practice 24(3): 13–20.

Callahan

Muller

Schiller

(2008) Preparing for citizenship: Immigrant high school students’ curriculum and socialization. Theory & Research in Social Education 36(2): 6–31.

Carnegie Corporation and CIRCLE (2003) The Civic Mission of Schools. New York and Washington, DC: Carnegie Corporation and CIRCLE.

Dawson

Venville

(2009) High-school students’ informal reasoning and argumentation about biotechnology: An indicator of scientific literacy? International Journal of Science Education 31(11): 1421–1445.

10.

Feldman

Pasek

Romer

D, et al.

(2007) Identifying best practices in civic education: Lessons from the Student Voices program. American Journal of Education 114(1): 75–100.

11.

Frazier

Tix

Barron

(2004) Testing moderator and mediator effects in counseling psychology research. Journal of Counseling Psychology 51(1): 115–134.

12.

Galston

(2007) Civic knowledge, civic education, and civic engagement: A summary of recent research. International Journal of Public Administration 30(6–7): 623–642.

13.

Gibson

Levine

(2003) The Civic Mission of Schools. New York: The Carnegie Corporation of New York and the Center for Information and Research on Civic Learning & Engagement.

14.

Hart

Atkins

(2002) Civic competence in urban youth. Applied Developmental Science 6(4): 227–236.

15.

Hartry

Porter

(2004) We the People Curriculum: Results of Pilot Test. Calabassas, CA: Center for Civic Education.

16.

Hess

Posselt

(2002) How high school students experience and learn from the discussion of controversial public issues. Journal of Curriculum and Supervision 17(4): 283–314.

17.

Hooghe

Wilkenfeld

(2008) The stability of political attitudes and behaviors across adolescence and early adulthood: A comparison of survey data on adolescents and young adults in eight countries. Journal of Youth and Adolescence 37(2): 155–167.

18.

Huey

Polo

(2008) Evidence-based psychosocial treatments for ethnic minority youth. Journal of Clinical Child & Adolescent Psychology 37(1): 262–301.

19.

Hwang

Lawrence

E, and Snow CE

(2015) Differential effects of a systematic vocabulary intervention on adolescent language minority students with varying levels of English proficiency. International Journal of Bilingualism 19(3): 314–332.

20.

Kahne

Middaugh

(2008) Democracy for Some: The Civic Opportunity Gap in High School. Medford, MA: The Center for Information and Research on Civic Learning & Engagement.

21.

Lawrence

Crosson

Pare-Blagoev

Snow

(2015) Word Generation randomized trial: Discussion mediates the impact of program treatment on academic word learning. American Educational Research Journal. Advance online publication.

22.

Levinson

(2012) No Citizen Left Behind. Cambridge, MA: Harvard University Press.

23.

Lin

(2015) Citizenship education in American schools and its role in developing civic engagement: A review of the research. Educational Review 67: 35–63.

24.

Lochman

Wells

(2003) Effectiveness of the coping power program and of classroom intervention with aggressive children: Outcomes at a 1-year follow-up. Behavior Therapy 34(4): 493–515.

25.

Lopez

Levine

Both

D, et al.

(2006) The 2006 Civic and Political Health of the Nation. College Park, MD: The Center for Information and Research on Civic Learning & Engagement.

26.

McDevitt

Chaffee

(2000) Closing gaps in political communication and knowledge effects of a school intervention. Communication Research 27(3): 259–292.

27.

McDevitt

Chaffee

(2002) The family in a sequence of political activation: why civic interventions can succeed. Journalism & Communication Monographs 4: 6–42.

28.

McDevitt

Kiousis

(2006) Experiments in political socialization: Kids Voting USA as a model for civic education reform. Medford, MA: The Center for Information and Research on Civic Learning & Engagement.

29.

Mancilla-Martinez

(2010) Word meanings matter: Cultivating English vocabulary knowledge in fifth-grade Spanish-speaking language minority learners. TESOL Quarterly 44(4): 669–699.

30.

Miranda

Bernal

Lau

A, et al.

(2005) State of the science on psychosocial interventions for ethnic minorities. Annual Review of Clinical Psychology 1(1): 113–142.

31.

Murnane

Willett

(2010) Methods Matter: Improving Causal Inference in Educational and Social Science Research. New York: Oxford University Press.

32.

National Council for the Social Studies (NCSS) (2007) Academic Freedom and the Social Studies Teacher. Silver Spring, MD: NCSS.

33.

Pasek

Kenski

Romer

D, et al.

(2006) America’s youth and community engagement how use of mass media is related to civic activity and political awareness in 14- to 22-year-olds. Communication Research 33(3): 115–135.

34.

Scheufele

(2002) Examining differential gains from mass media and their implications for participatory behavior. Communication Research 29(1): 46–65.

35.

Snow

Lawrence

White

(2009) Generating knowledge of academic language among urban middle school students. Journal of Research on Educational Effectiveness 2: 325–344.

36.

Syvertsen

Stout

Flanagan

CA, et al.

(2009) Using elections as teachable moments: A randomized evaluation of the Student Voices civic education program. American Journal of Education 116(1): 33–67.

37.

The Center for Information and Research on Civic Learning & Engagement (CIRCLE) (2013) All Together Now: Collaboration and Innovation for Youth Engagement. Medford, MA: CIRCLE.

38.

Torney-Purta

Barber

Wilkenfeld

(2007) Latino adolescents’ civic development in the United States: Research results from the IEA civic education study. Journal of Youth and Adolescence 36(2): 111–125.

39.

Venville

Dawson

(2010) The impact of a classroom intervention on grade 10 students’ argumentation skills, informal reasoning, and conceptual understanding of science. Journal of Research in Science Teaching 47(8): 952–977.