Abstract
Algebra is critical to high school graduation and college success, but student achievement in algebra frequently falls significantly below expected proficiency levels. While existing research emphasizes the importance of quality algebra instruction, there is little research about how to conduct problem analysis for struggling secondary students. This article proposes an assessment model designed to analyze algebra skills for struggling students to assess basic skills in mathematics, algebraic thinking, and algebra content knowledge. Results of the study indicated sufficiently reliable data. Exploratory factor analysis of the data also found three separate factors (basic calculation skills, mathematics application, and algebra content knowledge) that underlie the data. Implications for the classroom, future research, and study limitations are discussed.
The National Mathematics Advisory Panel (NMAP; 2008) reported that algebra was critical to later achievement. Students who completed Algebra II had higher college grade point averages, were more likely to graduate from college, and had higher earnings later in life than those that did not (Gaertner, Kim, DesJardins, & McClarty, 2014). Unfortunately, the average score for a 12th-grade student on the National Assessment of Educational Progress Algebra test was 155, which is below the expected proficiency score of 176 and indicated only basic understanding of algebra and its applications and procedures (National Center for Education Statistics [NCES], 2014).
The term proficiency refers to what someone knows how to do, can do, and ultimately wants to do (Schoenfeld, 2007). A student who is proficient in mathematics is knowledgeable, flexible, and resourceful with a given skill or set of skills (Foegen, Olson, & Impecoven-Lind, 2008). The National Research Council (NRC; 2001) identified mathematics proficiency as possessing five different yet interwoven strands of conceptual understanding, procedural fluency, strategic competence, adaptive reasoning, and productive disposition. Conceptual understanding is recognizing and understanding the core underlying ideas of a subject such as the relationships and reasons that underlie the mathematics problems in a certain area (Byrnes & Wasik, 1991; Hiebert & Lefevre, 1986), and procedural fluency is the knowledge of rules, symbols, and sequence of steps required to complete mathematics problems (Zamarian, Lopez-Rolon, & Delazer, 2007). Previous research has shown that these two areas of mathematical proficiency translated well to basic computation (Burns et al., 2015), but the implications for algebra were less clear.
Algebra Proficiency
As described by the NRC (2001), algebra proficiency is more complex than using the procedures taught in an algebra high school class. Algebra can be defined as (a) a way of thinking; (b) a set of concepts and skills that enable students to generalize, model, and analyze mathematical situations; and (c) a way to systematically investigate relationships to better describe, organize, and understand the world (NMAP, 2008). The term algebra is often used to represent algebra instruction that occurs within public schools. The NRC (2001) identified the core components of school algebra as those that are learned throughout high school, and are most commonly split between Algebra I and Algebra II, but may also be interwoven though courses like Geometry, Trigonometry, and Statistics.
The first major topic of school algebra includes symbols and expressions (NMAP, 2008). The NMAP (2008) identifies three different skills under this heading, polynomial expressions, rational expressions, and arithmetic and finite geometric sequences. The second topic is linear equations (Kaufman & Schwitters, 2004). A student should be able to use linear equations to solve problems and graph linear equations, linear inequalities, and multiple linear functions (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). The third area includes quadratic equations and polynomials, under which the NMAP (2008) identifies four different critical skills. A student should be able to (a) factor, or multiply, quadratic polynomials with integer coefficients; (b) complete the square in quadratic equations; (c) understand the quadratic formula and be able to factor general quadratic polynomials; and (d) use the quadratic equation to solve problems (NMAP, 2008). The fourth content area is functions, which includes linear, quadratic, polynomials, nonlinear, exponential, logarithmic, and trigonometric (Kaufman & Schwitters, 2004). The final content area includes combinatorics and finite probability where students should be able to apply the binomial thermo and Pascal’s Triangle to develop combinations and permutations (NMAP, 2008).
Problem Analysis and Algebra
When a student is struggling, despite solid core instruction, interventionists can engage in problem analysis to better target instruction and intervention to increase proficiency. Problem analysis is the systematic assessment and evaluation of a problem to find the potential causes and to determine what is enabling the problem to continue, to isolate and intervene (Christ & Arañas, 2014). However, there is currently no algebra-specific problem-analysis model. In addition, problem analysis should be guided by the use of targeted assessments (Thompson, 2004), but few of these types of assessments exist.
Project AAIMS (Foegen et al., 2008) is one of the only evidence-based algebra progress monitoring tools currently available. AAIMS is divided into four areas: Basic Skills, Algebra Foundations, Translations, and Algebra Content. Basic Skills assesses a student’s ability to perform calculations, simplify expressions, and utilize proportional reasoning. Algebra Foundations assesses knowledge variables and expressions, graphing, solving simple equations, and using patterns. The Translations section assesses conceptual understanding of algebra content and relationships. Content Analysis assesses content found in Algebra classes (Foegen et al., 2008). AAIMS has been found to be an effective tool in monitoring student growth in Algebra (Foegen & Morrison, 2010). However, progress monitoring is a different function than problem analysis because problem analysis determines what intervention to target and progress monitoring determines if the intervention is effective (Shapiro, 2010). Although monitoring student progress is an important part of an assessment-to-intervention framework, problem-analysis data are equally important (Burns, 2010).
Purpose
The current study was conducted to expand on the literature regarding assessment of algebra proficiency by examining a model to systematically assess and identify core deficits for students struggling with algebra. The assessment was based on research compiled from numerous national reports on mathematics and algebra, and combined fundamental mathematics skills that form the basis of advanced mathematics with Algebra I content. The following research questions guided the study:
Method
Participants
The study included participants from two charter schools located in an urban Midwestern city. The first was a middle school with 225 students, and the second was a high school with 225 students. All of the students voluntarily enrolled in the charter school, but were randomly selected for enrollment through a lottery system because there were more students who wanted to attend the schools than could do so. Together the two schools consisted of students from sixth through 12th grades, with approximately 41% receiving free- or reduced-price lunch and 17% receiving special education services. In 2013, 62% of students at the middle school scored within the proficient range for mathematics on the statewide test, and 47.4% of the students at the high school scored in the proficient range on the mathematics state test. The high school reported a graduation rate of 92.6%.
All of the students within the charter schools were eligible to participate in the study. Consent forms were sent home to every parent in the school, and 376 students returned them and participated in testing. A total of 327 students were included in the final study. Of those participants, 42% were male and 58% were female. Less than 1% identified as Native American or Alaskan Native, 11% identified as Asian or Pacific Islander, 16% identified as Hispanic, 23% identified as Black, and 50% identified as White. Of the total number of participants, 10% were receiving special education services and 6% were receiving educational support through a 504 plan. The average age of the participants was 14.1 years, and the average grade was 8.5. The distribution of students among grades was as follows: Grade 6 had 62 students (19%), Grade 7 had 59 students (18%), Grade 8 had 50 students (15%), Grade 9 had 51 students (16%), Grade 10 had 40 students (12%), Grade 11 had 39 students (12%), and Grade 12 had 26 students (8%).
Measures
The study utilized multiple measures to address the research question. We used the mathematics test of the Measures of Academic Progress (MAP; Northwest Evaluation Association [NWEA], 2011) as the criterion, and we assembled a series of assessments as the battery for problem analysis in algebra. Both are described below.
Criterion
The MAP is a screening tool for reading and mathematics proficiency in Grades 2 through 12. The mathematics test served as the criterion measure of a student’s level of algebra proficiency. The overall MAP score was used for analyses. A MAP Algebra score was available, but the overall score was both more reliable and a better predictor of overall algebra performance (NWEA, 2011). The MAP score was based on measures of number sense and number systems, estimation and computation, algebra, geometry, measurement, statistics and probability, problem solving, reasoning, and proofs.
The MAP scores were in a Rasch Unit (RIT) scale and were classified as below grade level, grade level, or above grade level by grade level. Test-retest reliability of the MAP ranged from .79 to .94, while internal consistency ranged from .61 to .92 (NWEA, 2011). Concurrent and predictive validity ranged from .37 to .86, with the majority of values ranging from .65 to .85 (NWEA, 2011).
Algebra problem analysis
The problem-analysis assessment was comprised of three skills sections and within each skill section are different subskills. The skills included Basic Skills, Algebraic Thinking, and Content Knowledge. The selection of sections and respective content was based on evidence from multiple research studies and national reports identifying the core components of algebra proficiency.
Content
Basic skills expectedly provide the basis for all mathematics. Without them, a student will not be able to gain proficiency with more advanced skills (Kraiger, Ford, & Salas, 1993; Woolfolk, 2008). The Basic Skills section include, comparing and ordering whole and rational numbers (Booth, Newton, & Twiss-Garrity, 2014; Hallet, Nunes, Bryant, & Thorpe, 2012; Wu, 2001), calculation of whole and rational numbers (Kinach, 2014; Siegler et al., 2012), and solving word problem (Fuchs et al., 2012; Mayer, 1982).
Algebraic thinking helps evolve one’s understanding and generalization of basic skills, informs new facts being learned, guide problem solving with novel and abstract problems, and extend algebra learning beyond simple procedural recall (Cai & Moyer, 2008; Ferrucci, Kaur, Carter, & Yeap, 2008). The subsections with Algebraic Thinking included understanding the equal sign and variables (Christou & Vosniadou, 2012; Herscovics & Linchevski, 1994; Pillay, Wilss, & Boulton-Lewis, 1998), differentiating arithmetic and algebra (Herscovics & Linchevski, 1994), understanding patterns (Lee & Freiman, 2006; Markworth, 2012; Stacey & MacGregor, 1997), and using proportional reasoning (Bright, Joyner, & Wallis, 2003; Fujimura, 2001; Özgün-Koca & Altay, 2009).
Knowing the content of an Algebra course is unquestionably core to algebra success. When learning algebra content, students need to focus on developing a conceptual understanding of the content (Byrnes & Wasik, 1991; Hiebert & Lefevre, 1986; Rittle-Johnson, Siegler, & Alibali, 2001), direct teaching of vocabulary (Adoniou & Qing, 2014; Little & Box, 2011), and a clear and explicit teaching of algebra problem-solving procedures (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010; NMAP, 2008; B. Witzel, Smith, & Brownell, 2001). Being exposed to vocabulary can help increase a student’s understanding and retention of academic material (Little & Box, 2011). For struggling learners, language acquisition proves a major barrier to algebra proficiency (Rakes, Valentine, McGatha, & Ronau, 2010).
Test structure
The assessment was developed under the framework that students need to have foundational mathematics skills to be successful in Algebra (Powell, Fuchs, & Fuchs, 2013), including early skills like number sense (Newton, Star, & Lynch, 2010). Possessing mastery of foundational skills allows for more effective learning of new procedures, deeper understanding of concepts, and enhanced skill generalization (Woolfolk, 2008). If someone has gaps in the foundation, it is likely more advanced skills will not develop (Kraiger et al., 1993). Failure to acquire these basic mathematics skills can keep a student from becoming proficient in algebra (Wu, 2001). The importance of each set of skill being assessed and its subsequent relationship to algebra proficiency was described in the previous section. The assessment did not directly evaluate the strands of mathematics proficiency presented by the NRC (2001), but was constructed with the idea that understanding core skills is critical to possessing proficiency with advanced skills (Powell et al., 2013; Wu, 2001).
The Basic Skills section consisted of six different probes. The probes assessed an individual’s skills in comparing and ordering different values; calculating single and multidigit addition, subtraction, multiplication, and division problems; and solving word problems. For each skill there are two different probes, one for whole numbers and one for rational numbers. The comparing and ordering probe using whole numbers contained six items while the probe using rational numbers contained nine. The calculation whole number probe contained 100 problems while the rational number calculation probe contained 30. The word problem probes using whole numbers and rational numbers both contained six problems. All of the rational number probes used decimals, percentages, and fractions (common, irregular, and mixed).
In the Algebraic Thinking skills section, test items were adapted from available online copies of the Minnesota Comprehensive Assessment (Minnesota Department of Education, 2013), Trends in International Mathematics and Science Study (Mullis, Martin, Foy, & Arora, 2012), and National Assessment of Educational Progress (NCES, 2014). The questions were taken from assessments designed up through eighth grade. The Arithmetic to Algebra subskill required the student to provide the missing number in an equation, and used all four primary operations with whole numbers in the equations. The problems in the Patterns and Relation subskill required the student to identify the patterns in a series of numbers. In the Generalization subskill, students were given equations containing unknown variables on either side of an equal sign, and asked to find the value of one variable when given the value of another. The Proportional Reasoning subskill section had the students solve a series of proportional reasoning problems while showing their work. The Arithmetic to Algebra probe contained 24 problems, the Patterns and Relation probe contained nine problems, the Generalization probe contained six problems, and the Proportional Reasoning probe contained six problems.
All of the multiple-choice questions within the Content Knowledge skills section were directly adopted from the Pearson (2011) All-In-One Algebra 1 Teaching Resources. The items assessed critical algebra skills identified in the Common Core Standards, including solving inequalities, functions, exponents and exponential functions, and polynomials and factoring. The language fundamentals subskill assessment included questions focusing on the definitions and ideas required in entry Algebra classes. A pool of algebra terms was chosen from key terms in Algebra I textbooks. The students were given one definition and four terms, and had to select the correct term. Finally, the questions within the conceptual understanding subskill assessed a student’s knowledge and understanding of the reasons behind specific entry-level algebra problems. The conceptual understanding subskill assessment presented students a picture and presented the student with two equations, one correct and one incorrect, and then asked the student to choose which problem was correct. Having students identify correct or incorrect mathematics problems has been found to be an effective method of assessing conceptual understanding of basic calculation (Burns, 2011; Burns et al., 2015) and might be used to differentiate conceptual understanding of linear algebra (Booth, Lange, Koedinger, & Newton, 2013). The content knowledge probe contained 10 problems, the vocabulary probe contained nine items, and the conceptual understanding probe contained six problems.
Procedure
Standard setting
The first task in comparing test performance to a criterion measure is to identify an appropriate proficiency score. Proficiency scores for each assessment were identified through the Angoff method because it had strong evidence as an empirical group method of standard setting (Berk, 1986). Teachers at the participating school served as the expert panel because of their familiarity with both the students and the content of Algebra I, Algebra II, Geometry, Precalculus, and Calculus (Berk, 1986; Koffler, 1980; Kellow & Willson, 2008). All of the teachers had undergraduate degrees in mathematics and graduate degrees in education, and had all been employed as mathematics teachers for 3 years or more. The standard setting method involved four primary steps. First, the panel was presented with the concept of the borderline test taker. Second, they were asked to imagine a group of 100 borderline test takers, defined as the hypothetical student who scored lower than the high group of students, but higher than the low group of students. The panel was then instructed to identify the proportion of that group would answer each question correctly. Finally, the proportions for each problem were summed for each panel member, and then averaged across all of the members’ scores to identify a proficiency score for the skill.
Pilot testing
The first author developed test items in collaboration with an expert in the field of mathematics education. The expert had a doctorate in mathematics education and was an associate professor at a major research university. Test items were selected and revised multiple times so that they adequately represented the areas being measured. The item selection template was structured around broadly evaluating each skill and subskill area, as including a comprehensive, diagnostic measure of each subskill would prove impractical for a real-life testing situation. The Content section (Vocabulary, Conceptual Understanding, and Content Knowledge) contained a sampling of items based on the state’s mathematics standards, which closely aligned with the Common Core State Standards (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010).
Prior to data collection, the items were administered in a pilot phase, which allowed an examination of the appropriateness of the items and administration procedures as well as the appropriate time frame for completion of each skill. The pilot testing occurred in an 11th-grade precalculus class of 16 students and a sixth-grade mathematics class of 14 students. The students’ performance was analyzed and necessary changes made to the assessments and administration procedures. Changes to the assessment included clarification of directions, implementing an appropriate time frame for completion, item arrangement, and item appropriateness. Clarification of directions was assessed through student feedback and observations of student understanding of the task directions. The appropriate time frame for the assessment was assessed by timing the amount of time it takes for the pilot class to complete the assessment and student feedback. The appropriateness of the test items was assessed by reviewing the item discrimination and item difficulty for each problem.
Minor changes were made to the assessment after the pilot testing. The directions were clarified as to make the instructions more direct for the students. Items were rearranged in four skill sections of the test to maintain the easy-to-difficult progression of items. Finally, one item from the vocabulary skill section and one item from the content knowledge skill section were eliminated due to poor discrimination ability and new items were developed. The characteristics of each resulting test in the problem-analysis battery can be located in Table 1.
Characteristics of Subskill Assessments.
Administration
Data collection occurred over the course of 1 month. The assessment administration occurred during one class period, or approximately 50 min. During the data collection phase, the researcher administered the assessments to participants at a classwide level. Instructions were given to the class as a whole within their respective classrooms. The students were informed that they would be taking an assessment for a research project the day before completing the assessments. The following day, the researcher went to each classroom, described the purpose of the tests to the students, and distributed the test packets. The instructions clearly described the process of testing to the student. The basic skills section assessments were administered first followed by the algebraic thought skills section and finally the content skills section, but the order of the assessments within each section was counterbalanced. After each section was complete, the students were given instructions to turn the page and the next task was administered. Once students were finished, the administrator collected the assessment packet.
All of the students across grades were given the same test with the same content items, knowing that the sixth- and seventh-grade students likely had minimal exposure, if any, to the content of an Algebra course. This was done so the relations between skills and subskills could be measured across all the participants regardless of algebra exposure and the early relations among skills could be established.
Diagnostic accuracy
The diagnostic accuracy was assessed by measuring the sensitivity and specificity of each skill against the proficiency score from the mathematics test from the MAP (MAP-Mathematics) of 181. The sensitivity was measured by identifying students who score below the proficiency MAP-Mathematics score, and identifying the students above the proficiency score will measure the specificity. The diagnostic accuracy was calculated by measuring the agreement versus the disagreement between the cut score and diagnostic criteria (Stage & Jacobsen, 2001).
Results
Data were found to be missing at random. Data from 49 students were discarded prior to running the analyses. Data from 14 students were discarded because they either entered the class after the administration had already started or had to leave before the administration was complete. The data from 35 students across all grades were discarded because they were absent on one of the testing days and only completed either the assessment or the survey. All of the students who completed both portions of the assessment did so completely. Missing data were not due to student refusal or nonresponse, and there were no obvious skipped items or sections. There were no notable patterns in the attendance or behavioral data (e.g., suspensions) for the students with missing data.
The descriptive data for each skill measure by grade can be found in Table 2. Students in the sixth and seventh grade scored lower across all skill areas when compared with students in Grades 8 through 12, and students in ninth grade scored lowest across all skill levels when compared with their high school peers. The greatest change across grades occurs within the Content Knowledge section, whereas the smallest change occurred within the Algebraic Thinking section.
Descriptive Table of Skill Measures—Grade Level.
Correlation With Criterion Measure Using Transformed Scores
The analyses of the assessment were initially conducted using raw scores gathered from both the subskill and skill sections. However, it is important to consider the impact and importance of each subskill score on the skill scores, as accurate skill section scores are critical for establishing proficiency levels. For example, a student who scored very high on the integer calculation subskill section may have scored very poorly on the integer and rational word problem measures. Compared with a student who scored very high on the word problem measures but less so on the calculation measures, their scores may appear the same but fail to reflect their true overall level of proficiency with the larger skill.
The scores were transformed so they became standardized and comparable in a more meaningful way. To do this, z-scores were calculated for each skill and subskill section. The z-scores were calculated using the overall mean and standard deviation. Pearson correlations were calculated for the sections and the composite score of the test in relation to the MAP-Mathematics score. The correlations for the three sections using the z-scores and MAP-Mathematics were as follows: Basic Skills r(325) = .81, p < .001; Algebraic Thinking r(325) = .77, p < .001; and Content Knowledge r(325) = .68, p < .001. The correlation between the MAP and the composite score was r(325) = .85, p < .001.
Student Proficiency Across Sections
An analysis of the percentage of students who scored proficiently on zero, one, two, or three of the skill areas can be found in Table 3. The fewest number of students were only proficient with Basic Skills, while the greatest number of students lacked proficiency with all three skills. The number of students who were proficient with Content Knowledge was only slightly higher than students who were proficient with Algebraic Thinking only. Students were more likely to be proficient with both Algebraic Thinking and Content Knowledge than with either Basic Skills and Algebraic Thinking or Basic Skills and Content Knowledge. Almost 25% of the students were proficient in all three areas.
Percentage of Students Scoring Proficient Across Skill Sections.
One possible reason for the low levels of proficiency with only Basic Skills is that a student in middle or high school who has mastered basic skills is likely to also have mastered more advanced skills, and a student who struggles with basic skills have fallen behind his peers in both their proficiency level with those skills and in other areas. Proficiency with Content Knowledge may rely more heavily on current levels of Algebraic Thinking than Basic Skills, while Basic Skills appear to be important to the initial development of Algebraic Thinking.
Diagnostic Accuracy
The diagnostic accuracy was assessed by measuring the sensitivity and specificity of each section against the MAP-Mathematics proficiency score of 181, and the results can be found in Table 4. The sensitivity was measured by identifying students who scored below the proficiency MAP-Mathematics score, and identifying the students above the proficiency score measured the specificity. The diagnostic accuracy was calculated by measuring the agreement versus the disagreement between the cut score and diagnostic criteria (Stage & Jacobsen, 2001). The sensitivity value for the overall assessment was 76% while the specificity was 92%. While the Authentic Application and Engagement section scores were included in the overall assessment score, there were no sensitivity, specificity, or accuracy values calculated for them. This is because cutoff scores were not identified for those assessments due to a lack of traditional “correct” and “incorrect” answers.
Standard Setting, Sensitivity, Specificity, and Diagnostic Accuracy of Sections and Subsections With MAP for Mathematics.
Note. MAP = Measures of Academic Progress.
p < .01.
Comparison of Models Using Exploratory Factor Analysis
To promote a conceptualization of algebra proficiency that can potentially be used in a problem-analysis framework, the researchers conducted exploratory factor analysis to examine the distribution of data and identify potential latent variables. The fit indices for four different models can be found in Table 5. Based on the data, the three-factor model appears to be the best fit. The root mean square error of approximation (RMSEA) value is .05 with a 90% confidence interval of .03 to .07, with a rule of .05 being an indicator of good fit, and the Tucker–Lewis Index (TLI) is .97, with a cutoff of .95 (Cheung & Rensvold, 2002; Walkey & Welch, 2010). The factor loadings for the three-factor model can be found in Table 6.
Fit Indices for Models Using Exploratory Factor Analysis With Promax Rotation.
Note. RMSEA = root mean square error of approximation; CI = confidence interval.
Factor Loadings for Three-Factor Model.
Note. Factor loadings >.40 are in boldface.
The data indicate that the following variables were associated with Factor 1: ordering rational numbers, solving word problems using integers and rational numbers, generalization, and proportional reasoning. The following variables were associated with Factor 2: calculation with rational numbers, vocabulary, conceptual understanding, and content knowledge. The following variables were associated with Factor 3: ordering integers, integer calculation, patterns, and arithmetic to algebra.
Discussion
The current study was conducted to expand on the literature regarding assessment of algebra proficiency. Three questions guided the study: (a) what is the relationship between each of the skills within the assessment to an established measure of algebra, (b) to what extent do the skills within the assessment accurately identify the level of a student’s difficulty with algebra as measured by a criterion, and (c) to what extent does the data support a problem-analysis model to help assess algebra proficiency?
The correlation between the total score and the MAP-Mathematics test of r = .85 suggested a strong correlation between the two, and that the algebra assessment was a good predictor of overall mathematics proficiency. Sensitivity and specificity were calculated for all the sections scores and the total score for the Basic Skills, Algebraic Thought, and Content Knowledge test sections. The total score resulted in a sensitivity value of .76. This indicates that the test identified 76% of students that were proficient with algebra, but did not identify 24% of those who were proficient. The data fared better with specificity, with a value of .92. Thus, the data had a 92% chance of correctly identifying a student who was not proficient at algebra. This ratio of low sensitivity to high specificity is somewhat desirable, because in practice it is better to overidentify struggling students and deliver additional support than it is to fail to identify struggling students.
The section correlations indicated that the skill and subskill sections provided an important set of information regarding the mathematics and algebra proficiency. Among the sections, Basic Skills was strongly correlated with Algebraic Thinking, indicating that a student’s ability to order, calculate, and solve word problems using integers and rational numbers was closely related to their ability to perform tasks that utilize algebraic thinking. Basic skills seemed to have less impact on their ability to solve problems and recall facts using content knowledge.
Results from the exploratory factor analysis indicated that a three-factor model helped explain the distribution and relationship of the data. The factor including the ordering integers, integer calculation, patterns, and arithmetic to algebra skills can be interpreted as representing calculation skills. The second factor including ordering rational numbers, solving word problems using integers and rational numbers, generalization, and proportional reasoning can be interpreted as representing mathematics application and flexible thinking. The third variable including calculation with rational numbers, vocabulary, conceptual understanding, and content knowledge can be thought to represent algebra problem solving. The latent factors and variable distribution were similar to the assessment structure of Project AAIMS (Foegen et al., 2008).
Implications for Practice
The results of the current study also indicated that the proposed assessment was an appropriate measure of overall algebra proficiency. While the test was structured around a framework of testing basic skills, algebraic thinking, and algebra content knowledge, factor analysis indicates that algebra proficiency may better be conceptualized under calculation skills, mathematics applications, and algebra content. While the data did not provide a clear, linear, or causal model of learning algebra, they did provide support for the idea that proficiency with more fundamental skills was related to proficiency with advanced skills. In practice, the assessment may help in identifying students who are struggling, or at risk of struggling, with algebra. The assessment may also help guide practitioners in identifying specific areas where a student may have deficits. If a student is struggling in the classroom, a teacher may be able to use the assessment to identify potential areas of need for targeted instruction.
Implications for Future Research
Future research should continue to evaluate the validity and reliability of the assessment, replicate the study with different populations, and gather age- and grade-level norms. It is important that the assessment results are representative of actual student performance in the classroom. Using the current data as a predictor, and measure, of long-term success will also help to establish the usefulness of the assessment. Research should consider utilizing different modes of testing administration to see how it impacts students’ performance and the validity of the model. Future research should focus on identifying effective targeted interventions for the different related skills and subskill areas. There is a substantial body of research on improving the basic arithmetic skills of struggling students through intervention, but there is significantly less literature on delivering interventions for improving students’ algebraic thinking and content knowledge. Finally, future research should continue to expand on the validity of the three-factor model as a potential way to conceptualize algebra proficiency that could be applied in the context of problem analysis.
Limitations
Due to the preliminary nature of the current research, there are many limitations to this study that should be considered when interpreting the results. One major limitation of the study was the use of the MAP-Mathematics as a criterion measure. While the MAP-Mathematics is a robust measure of mathematics proficiency, it is less reliable when identifying algebra deficits specifically. However, it should be noted that the MAP-Mathematics total score, while more reliable and valid than the MAP-algebra score, is not always considered an algebra criterion measure. In the areas of assessment, assessing algebra proficiency is a major challenge, as there are few assessments that are designed to specifically target these skills.
Another major limitation of the study was that the measures and content used within the assessments were created for the purpose of testing. Preexisting measures with backing of literature were not used because none were available. The subskill measures used to examine performance in each area were designed specifically for this assessment. Prior to use they were piloted, the items analyzed for specificity and sensitivity. If appropriate measures are identified, concurrent and divergent validity studies could be conducted. In addition, alternate tests could provide more evidence for or against the three-factor model identified in the data.
A third limitation of the study was the design of the assessments themselves. The assessments were designed with a limited testing time in mind due to the practical nature of the assessment administration. Limited time can impact a student’s ability to complete the assignments even if they know the information. A student’s awareness of time limits may have had an impact on their levels of fatigue, anxiety, recall, or nervousness during testing, which could have had a negative impact on the assessment results. Future research should compare student performance on the current assessment with performance on tests with unlimited time to help determine the impact of the timing and how it can be addressed.
A fourth limitation is that the population used was a convenience sample. While all students were tested within the two schools, those schools only represented a certain sampling of middle and high school students. The assessment’s reliability and validity can be improved by applying the study across different populations.
Conclusion
The ongoing research on academic supports for secondary students seems far outpaced by the needs of that population. The current study works to address this gap by developing a method to analyze the skills of students struggling with algebra. The proposed assessment results in reliable data and from three underlying factors, basic skills, algebraic thought, and algebra content. While generalizability of these findings was limited due to the reasons listed above, this research took another step toward helping teachers identify and intervene with students struggling with algebra. Teachers can potentially use the current findings as one tool out of many to identify and work with struggling students while being better able to conceptualize with what that student may be struggling.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.