Abstract
This study reports on the relationships between test takers’ individual differences and their performance on a reading comprehension test. A total of 518 Chinese college students (252 women and 256 men; M age = 19.26 year, SD = 0.98) answered a questionnaire and sit for a reading comprehension test. The study found that test takers’ L2 language proficiency was closely linked to their test performance. Test takers’ employment of strategies was significantly and positively associated with their performance on the test. Test takers’ motivation was found to be significantly associated with reading test performance. Test anxiety was negatively related to their use of reading strategies and test performance. The results of the study lent support to the threshold hypothesis of language proficiency. The implications for classroom teaching were provided.
Keywords
Introduction
In recent years, language researchers have been concerned with identifying individual characteristics that are related to test takers’ performance on tests of English as a foreign language (EFL) and English as a second language (ESL) (Kunnan, 1995; Phakiti, 2008; Purpura, 1999; Zhang, Goh, & Kunnan, 2014). Personal attributes that affect test performance can be categorized into four major types: (a) background or demographic characteristics such as a person’s age, sex, and first language proficiency level; (b) socio-psychological characteristics such as attitude and motivation; (c) personality characteristics including a person’s self-esteem and anxiety; and (d) cognitive characteristics which refer to aptitude and learning strategies (Bachman, 1990; Bachman & Palmer, 2010; Purpura; 1999). According to Bachman and Palmer (2010), test takers’ language ability is their focal personal attribute while other characteristics are peripheral.
Researchers have conducted studies investigating the relationships between learners’ individual differences and language acquisition (Bernaus & Gardner, 2008; Dörnyei & Csizér, 1998; Gardner, 2007; Gardner, Masgoret, Tennant, & Mihic, 2004; Zhang, 2010). However, few studies examined how test takers’ individual differences affect their test performance collectively, although research in this area could contribute to the construct validation of the EFL/ESL test and the investigation of factors that influence test performance. This study was thus designed to investigate how test takers’ individual difference (e.g., language proficiency, motivation, test anxiety, and strategy use) affect their performance in an EFL reading comprehension test. More importantly, the current study will examine the relationship between test takers’ performance and several of their individual differences collectively and comprehensively in one study and provide implications for assessment and instruction of second language reading. This approach of combining the variables of individual differences synthesizes their effects in a way not explored in prior research.
Researchers have come to the consensus that readers construct meaning and achieve comprehension through interaction with text and context (Grabe, 2009; Kintsch, 1988; Pressley & Afflerbach, 1995). In the meaning construction process of reading comprehension, readers monitor their comprehension and repair their comprehension failure using various strategies. It is generally acknowledged that strategy use and comprehension monitoring distinguish skilled from unskilled readers (Grabe, 2009; Paris & Winograd, 1990). In addition, strategy use plays a very important role in enhancing test performance (Cohen, 2006). Previous research has shown that test takers’ strategy use explained 16%–30% of test performance (Phakiti, 2008; Zhang et al., 2014).
Regarding the relationship between students’ L2 proficiency and reading performance, it is found that the former is closely associated with the latter. According to the theory of a threshold level of language proficiency (Cummins, 1979) or linguistic ceiling (Clarke, 1978, 1980), readers would not be able to read effectively until they reach a threshold level in L2 language proficiency. This is congruent with Alderson’s (1984) famous conclusion that L2 reading “appears to be both a language problem and a reading problem, but with firmer evidence that it is a language problem, for low levels of L2 competence, than a reading problem” (p. 24). In other words, when reading in an L2, readers cannot use knowledge or intuitions gained from their L1 reading experience until they reach certain level in the L2 proficiency.
Researchers argue that the effective use of strategies in reading tests depends on factors related to test takers’ individual differences, such as their language proficiency level, test anxiety, and motivation. For example, most researchers have found that language proficiency is positively associated with strategy use (Griffiths, 2003; Magogwe & Oliver, 2007; Green and Oxford, 1995). Specifically, Green and Oxford (1995) found that proficiency level is significantly related to metacognitive and cognitive strategy use.
Test anxiety is viewed as a negative emotion associated with test-taking situations (Richardson, Abraham & Bond, 2012). It is an educational problem affecting students’ academic achievement (Nicaise, 1995; Suinn, 1968). On the other hand, it is possible that students’ previous academic performance affects their test anxiety. Test takers with test anxiety may sweat, experience faster heartbeat, or other physical reactions. In addition, highly anxious examinees may feel tense and apprehensive, which affects their performance on the test (Gierl & Rogers, 1996). Researchers found that test anxiety accounted for an important amount of variance in test performance (Cassady & Johnson, 2002). Studies also found that the impeding effect of stress on performance is moderated by strategy use (e.g., Rai, Loschky, Harris, Peck, & Cook, 2011).
The study of motivation and reading comprehension showed that intrinsic and extrinsic motivations play a salient role in affecting students’ performance in reading (Proctor, Daley, Louick, Leider, & Gardner, 2014; Wang & Guthrie, 2004). Intrinsic motivation is “a concern with learning and mastering the task using self-set standards and self-improvement,” and extrinsic motivation is “a focus on getting good graders and pleasing others” (Pintrich, 1999, p. 466). Specifically, intrinsic motivation refers to the motivation to engage in an activity simply because it is enjoyable and satisfying itself, whereas extrinsic motivation is the motivation to take part in an activity whose outcome, either reward or punishments, is separated from the activity itself (Ryan & Deci, 2000). Students’ motivation has a positive effect on their performance. For example, it was found that a higher level of intrinsic motivation and a medium level of extrinsic motivation are positively and closely related to students’ performance (Lin, McKeachie, & Kim, 2003). In addition, studies have shown that students’ motivation also affects their strategy use (e.g., Metallidou & Vlachou, 2007; Sins, van Joolingen, Savelsbergh, & van Hout-Wokters, 2008; Vrugt & Oort, 2008). Although researchers have examined the relationships among students’ strategy use, test anxiety, intrinsic and extrinsic motivation, and academic performance, few studies have investigated how these individual differences are related to their language test performance collectively and comprehensively in one study. The current study was designed to fill in the gap in literature.
The purpose of this study was to test a model of test takers’ individual differences and test performance. While test takers’ strategy use was included as an independent variable, the major concern was on the relative importance of motivation, test anxiety, and L2 language proficiency to the reading test performance. Specifically, eight hypotheses were tested: Hypothesis 1. Test takers’ strategy use has a direct effect on reading test performance. Hypothesis 2. Test takers’ motivation to learn has a direct effect on test performance. Hypothesis 3. Test takers’ motivation to learn has an indirect effect on reading test performance via strategy use. Hypothesis 4. Test takers’ L2 language proficiency has a direct effect on test performance. Hypothesis 5. Test takers’ L2 language proficiency has an indirect effect on reading test performance via strategy use. Hypothesis 6. Test takers’ test anxiety has a direct effect on reading test performance. Hypothesis 7. Test takers’ test anxiety has an indirect effect on reading test performance via strategy use. Hypothesis 8. Test takers’ L2 language proficiency and text anxiety are related to each other.
Method
Participants
Chinese college students (N = 518) were invited to participate in the study. They completed a consent form, responded to the questionnaire and the reading test. Among the participants, 256 (49.4%) were male and 250 (48.3%) female students who aged between 18 and 24 years (M = 19.26, SD = 0.98). A total of 12 students did not provide their gender information. They had learned English at school for an average of 8.68 (SD = 2.31). Their reported English score for the National Matriculation Entrance Test (NMET) 1 ranged from 83.00 to 149.00 (M = 125.64, SD = 10.27).
Measures
There are three measures used in the current study: the goal orientation and test anxiety scales from the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1993), the strategy use questionnaire, and an EFL reading comprehension test.
The goal orientation and test anxiety scales from the MSLQ
The subscales of goal orientation and test anxiety from the MSLQ is one of the most widely used measures of motivation and learning strategies (Duncan & McKeachie, 2005; Dunn, Lo, Mulvenon, & Sutcliffe, 2012; Winne & Perry, 2000). The MSLQ consists of two main scales: Motivation Scale and Learning Strategies Scale. The Motivation Scale has six subscales which are related to students’ goals, beliefs, skills, and anxiety in courses and tests, whereas the Learning Strategies Scale contains nine subscales regarding cognitive strategies. According to Duncan and McKeachie (2005), the 15 subscales can be used individually or collectively to assess the general or specific constructs regarding students’ cognitive activities. In this study, the subscales of goal orientation (i.e., intrinsic goal and extrinsic goal) and test anxiety were used to assess students’ intrinsic and extrinsic goal and test anxiety on the reading test. There are eight items measuring goal orientation and five items assessing test anxiety on a 7-point Likert scale, from 1 (Not at all true of me) to 7 (Very true of me).
The strategy use questionnaire
The Metacognitive and Cognitive Strategy Questionnaire (MCSQ) (Zhang et al., 2014) was used to measure test takers’ strategy use. Based on the theory of metacognition, Cohen and Upton’s (2007) framework, Pressley and Afflerbach’s (1995) constructively responsive reading model and test-taking studies, the MCSQ was designed to measure language users’ strategy use in reading comprehension test. Following established theories, metacognitive strategies comprise planning (for achieving pre-established goals), evaluating (for assessing tasks and personal cognitive abilities), and monitoring (for checking and regulating performance) strategies. Cognitive strategies are viewed as specific and conscious metal behaviors or activities used by test takers to solve the problems in the processes of reading comprehension. These strategies include initial reading (for engaging in general reading of the text), identifying important information (for refining understanding of the text), inference making (for bridging information gaps in the text), and integrating (for manipulating the text to fit information across the text) strategies.
There are 38 items in the questionnaire measuring seven subscales, including planning, evaluating, monitoring, initial reading, identifying important information, integrating, and inference-making strategies (For specific definition of each strategy, see Zhang et al., 2014). A 6-point Likert scale 0 (Never), 1 (Rarely), 2 (Sometimes), 3 (Often), 4 (Usually), to 5 (Always) was used.
The EFL reading comprehension test
A retired version of the College English Test Band-4 (CET-4) Reading (Fang, 2010) was used as a measure of students’ reading test performance. CET-4 is a national standardized test designed to assess Chinese college students’ English ability (Jin, 2008; Yang & Weir, 1998). The reading test used has 50 items and four sections: 10 items measuring students’ skimming and scanning skills (SKSN), 10 items of banked cloze (BCLZ), 10 careful reading items (RID), and 20 multiple choice (MCLZ) cloze items.
Analysis
Descriptive statistics
IBM SPSS Statistics 20 was used to examine the descriptive statistics. Mean, standard deviation, Skewness, and kurtosis were calculated at the item level. The internal reliability estimates of the questionnaire and test were estimated. IBM SPSS AMOS computer program Version 20.0 (Arbuckle, 2011) was used to perform the analysis as well. Maximum Likelihood technique was chosen as the method of parameter estimation. Mardia’s multivariate coefficient was also checked to ensure the multivariate normality of the data.
Confirmatory factor analysis
Sufficient empirical and theoretical support has been provided for the MSLQ (e.g., Pintrich et al., 1991) and the MCSQ (Zhang et al., 2014). Confirmatory factor analysis (CFA) was thus performed to assess the fit of the model of goal orientation, test anxiety, and strategy use to the current data (see Pupura, 1997; Phakiti, 2008). CFA was conducted on goal orientation and test anxiety, as they are different constructs. Similarly, CFA analyses would provide preliminary evidence of the test validity.
SEM analysis
Following the reviewed theories and empirical studies in the previous part, the researcher postulated a SEM model to make investigation of the relationships between test takers’ personal features and performance on the test.
The model fit was examined through a series of fit indices. The comparative fit index (CFI) and the Tucker-Lewis index (TLI) are used to evaluate the relative improvement in fit of the hypothesized model in comparison with the null model. A value greater than .90 is considered a good model fit. The absolute fit index root mean square error of approximation (RMSEA) should be less than .06 (Hu & Bentler, 1999). The narrow interval of the RMSEA 95% confidence interval shows better model fit. The ratio of the chi-square to degree of freedom (χ2/df) should be less than 3, which is indicative of a well-fitting model.
Results
Descriptive statistics
Descriptive statistics for the measures.
Note. NA = not applicable; EVA: evaluating strategies; MON: monitoring strategies; PLA: planning strategies; INI: strategies for initial reading; IDE: identifying strategies; INT: integrating strategies; INF: inference making strategies; SKSN; skimming and scanning; BCLZ: banked cloze; RID: reading in depth; MCLZ: multiple-choice cloze; intr_G: intrinsic goal orientation; extr_G: extrinsic goal orientation; Eng score: students’ self-reported English score for the National Matriculation Entrance Test; test_anx: test anxiety.
Confirmatory factor analysis
Fit indices for the tested CFA models.
Note. CFA = confirmatory factor analysis; df = degree of freedom; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root Mean Square Error of Approximation; RMSEA 90% CI = Root Mean Square Error of Approximation 90% Confidence Interval.
Structural equation modeling
Before the final model shown in Figure 1, several alternative models were tested. Considering adequate model fit and model parsimony, the model in Figure 1 was chosen as the final model.
2
Due to limited space, the information about model selection and comparison was not provided in this article. The SEM analysis produced good model fit indices as shown in Table 3. The ratio of χ2/df = 2.69, CFA = .92, TLI = .90, RMSEA = .057, and RMSEA confidence interval = .048, .066. The tested model with standardized coefficients was presented in Figure 1.
The tested model. Fit indices for the tested SEM model. Note. SEM = structural equation modeling; df = degree of freedom; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root Mean Square Error of Approximation; RMSEA 90% CI = Root Mean Square Error of Approximation 90% Confidence Interval.
Discussion
This study used structural equation modeling to investigate the relationship between test takers’ individual differences and their reading test performance. Based on relevant literature, test takers’ reading test performance was hypothesized to be affected by their L2 language proficiency, strategy use, motivation, and test anxiety. In addition, it was postulated that test takers’ L2 language proficiency, test anxiety, and motivation had direct effects on their strategy use. Results showed that the hypothesized model was plausible, as it is a well-fitting model (Table 3). Findings from this study have shed light on the relationship among test takers’ reading test performance, L2 language proficiency, strategy use, test anxiety, and goal orientation.
L2 language proficiency, strategy use, and reading test performance
Regarding the effect of L2 Language proficiency on L2 reading performance, this study found that L2 proficiency accounted for 80% of the total variance (i.e., 77% of variance for direct effect and 3% for indirect effect) in reading performance, which provided support for Alderson’s conclusion that L2 reading ability is affected by their L2 language proficiency. This finding concurs with Lee and Schallert (1977), whose correlation analysis indicated that 56% of the variance in L2 reading was accounted for by L2 proficiency. The discrepancy in the percentage could be explained through the method used in this study as error variance is taken into consideration in SEM (Byrne, 2011). In addition, the fact that L2 proficiency explained a large portion of variance in reading performance provided supportive evidence for the validity of the reading test as this shows that test takers’ language ability rather than other construct was assessed (Bachman & Palmer, 1996).
Apart from L2 language proficiency, strategy use was found to explain 5% of variance in L2 reading performance, which is similar to Song and Cheng’s (2006) finding that strategy use accounted for 8.6% of the variance in the CET-4 test. In addition, Phakiti’s (2008) and Zhang et al. (2014) also found that that strategy use had a weak to moderate effect on test performance, which could explained 3% to 30% of test takers’ performance.
Several plausible reasons may explain the limited role strategy use played in affecting test performance. First, Bachman and Palmer (2010) argued that test takers’ use of strategy is one of the features that may affect their performance. The dominating factor is their language knowledge. Furthermore, empirical studies (e.g., Cohen & Aphek, 1979; Paris, 2002; Song, 2005) also showed that strategies may have positive, negative, or no effects on language performance. Also, Anderson (2005) pointed out that successful use of strategies hinges on several conditions: (1) the strategy is related well to language tasks encountered by language users, (2) the strategy is linked closely with other strategies relevant to the tasks at hand, and (3) the strategy is consistent with the language user’s learning style. Furthermore, strategy use in test situations may be related to test takers’ self-assessment and evaluation (Phakiti, 2008). In other words, if test takers fail to assess or evaluate themselves properly, it may lead to poor strategy use. Thus, the limited role played by strategy use in Chinese college test takers’ reading performance in this study may be due to their failure to link suitable strategies to the tasks at hand or their learning style or their inappropriate self-assessment in responding to the test tasks.
In addition, this finding also concurred with Bachman’s (1990) argument that strategy use is only one of test takers’ personal features that would have an effect on their test performance. According to Bachman and Palmer (2010), language ability comprises two components: language knowledge and strategic competence. The core of strategic competence is metacognitive strategy use. Thus, strategy use is one part of their language ability and the dominating contributor to the variance of test scores is test takers’ language knowledge (Saito, 2003; Song, 2005).
Test anxiety, language proficiency, strategy use, and reading test performance
The study showed that test anxiety had negative effects on test takers’ strategy use (β = −.13) and reading test performance (β = −.17, p < .05). In addition, it correlated with students’ reported NMET score at a value of −.11 (p < .05). These results, in size and direction, are consistent with other studies of test anxiety and academic performance. For example, Eum and Rice (2011) reported a −.17 correlation between students’ cognitive test anxiety and their GPA. Cassady and Johnson’s (2002) found that the partial regression coefficients for the relationship between students’ cognitive test anxiety and academic performance varied from −.22 to −.26. Putwain, Wood, & Symes (2010) indicated that the correlation between students’ test-anxious worry and total academic points earned was a value of −.23.
In addition, literature shows that test anxiety may facilitate or debilitate test performance (Alpert & Haber, 1960). Normal and proper amount of stress during testing can enhance test performance (Salend, 2012). However, students who experience harmful test anxiety often have poor performance on tests. Studies show that a variety of factors can lead to test anxiety, such as unrealistic expectations, negative self-esteem, lack of confidence, poor prior performance, poorly designed tests, and unfavorable testing environments, and so on (Salend, 2012). From the perspective of testing research, the focus in studying test anxiety should be on its implication for the test validation. The finding in the current study showed that students’ test anxiety level for the reading test (β = −.17, p < .05) is similar to that for the NMET (−.11), suggesting that the reading practice test appeared to be as student-friendly as the NMET, the influential national standardized test. Additionally, the result also showed that test anxiety affected students’ strategy use negatively, indicating that students’ stress during testing indeed influenced their process of reading comprehension.
The effect of goal orientation on strategy use and reading test performance
Based on the results, goal orientation had a direct effect on strategy use (β = .45, p < .05). Though goal orientation did not affect reading test performance directly, it had an indirect effect on test takers’ performance through strategy use, which accounted for 10% (.45 × .23) of the variance of reading test performance. This finding supports previous research that motivation affects learners’ strategy use positively (e.g., Paris & Winograd, 1990; Vrugt & Oort, 2008).
Additionally, mediated through students’ deep cognitive processing in reading comprehension, motivation was found to affect test performance indirectly, lending support to Sins et al.’s (2008) study. This implies that though research shows that strategy use seemed to have a weak to moderate relationship with test performance (Phakiti, 2008; Song & Cheng, 2006; Zhang et al., 2014), other test takers’ personal features play a role in affecting test performance indirectly through strategy use. In other words, it suggests that while strategy instruction might be a plausible method to enhance test takers’ performance, researchers and school teachers should consider how to improve their performance through a combined enhancement of strategy use and motivation.
Conclusions and implications
Using a structural equation modeling approach, this study investigates the effects of test takers’ personal features on Chinese college students’ EFL reading test performance. Different from previous research, this study provided a comprehensive picture about how learners’ individual differences affect their reading test performance.
The findings in this study indicate that test takers’ personal characteristics have different effects on test performance. In line with the theory of the threshold level of language proficiency, this study found that test takers’ L2 proficiency had a direct and close relationship with their reading test performance. Though goal orientation did not affect test performance directly, it had an indirect effect through strategy use, which had a direct but weak effect on test performance. Consistent with previous research, test anxiety affected test performance negatively.
The results of the study point to the need to teach L2 reading by using effective approaches based on their proficiency level and other personal features (Lee & Schallert, 1997). For example, for readers with lower L2 language proficiency, emphasis of instructions should be put on the vocabulary and grammar. Once they reach the threshold level, the focus should be on the improvement of non-linguistic factor such as strategy use, motivation, etc. As for the readers with higher L2 language proficiency, instruction could focus on enhancing the non-linguistic factors such as how to use strategies effectively and how to reduce test anxiety, etc.
In conclusion, this is a small-scale study that examined the complex relationships among test taker’s individual differences and reading test performance. Although it revealed some interesting findings, several limitations are worth mentioning, as they have implications for future research in this area. First, this study investigated the effect of participants’ individual differences on reading test performance regardless of their L2 proficiency. To gain a better understanding of the influence of test takers’ personal characteristics, future studies are suggested to use a multi-sample approach to differentiate students with high, middle, and low L2 proficiency and compare the results of the analysis. Thus, a detailed profile for each group can be provided which is useful for differentiated classroom instruction. In addition, it should be noted that the anxiety questionnaire drawn from the MSLQ is a measure of students’ general course anxiety and may not be a very appropriate instrument to elicit students’ anxiety during the reading test. Therefore, their responses to the questionnaires may not reflect their test anxiety very accurately. It is suggested that a measure assessing test takers’ anxiety during the test be employed in future research.
On the other hand, this study adopted a quantitative approach to the investigation of test takers’ personal characteristics. Future research could investigate the problem from both quantitative and qualitative perspectives. It is recommended that for future studies researchers collect think-aloud data of how test takers engage in the reading comprehension test (Ericsson & Simon, 1993). It is hoped that findings from the analysis of think-aloud data can complement and triangulate the findings from quantitative analysis and give an all-inclusive picture of how test-takers’ individual differences affect their test performance.
