Abstract
Within the new era of artificial intelligence (AI), education industry should develop in the direction of intelligence and digitalization. For evaluating learners’ academic performances, English high-stakes test is not only a mere means for measuring what English as a Foreign Language (EFL) stakeholders know or do not know but also likely to bring life-changing consequences. Hence, effective test preparation for English high-stakes test is crucial for those who futures depend on attaining a particular score. However, traditional corpus-based approaches cannot simultaneously take words’ frequency and range variables into consideration when evaluating their importance level, which makes the word sorting results inaccurate. Thus, to effectively and accurately extract critical words among English high-stakes test for enhancing EFL stakeholders’ test performance, this paper integrates a corpus-based approach and a revised Importance-Performance Analysis (IPA) method to develop a novel frequency-range analysis (FRA) method. Taiwan College Entrance Exam of English Subject (TCEEES) from the year of 2001 to 2022 are adopted as an empirical case of English high stake test and the target corpus for verification. Results indicate that the critical words evaluated by FRA method are concentrated on Quadrant I including 1,576 word types that account for over 60% running words of TCEEES corpus. After compared with the three traditional corpus-based approaches and the Term Frequency-Inverse Document Frequency (TF-IDF) method, the significant contributions include: (1) the FRA method can use a machine-based function words elimination technique to enhance the efficiency; (2) the FRA method can simultaneously take words’ frequency and range variables into consideration; (3) the FRA method can effectively conduct cluster analysis by categorizing the words into the four quadrants that based on their relative importance level. The results will give EFL stakeholders a clearer picture of how to allocate their learning time and education resources into critical words acquisition.
Keywords
Introduction
Education industry should develop in the direction of intelligence and digitalization based on feasible Information and Communication Technology (ICT) assets. For academic performance evaluation, English as a Foreign Language (EFL) learners’ performance on English high-stakes tests (e.g. TOEFL®, IELTS™, college/high school entrance exams, etc.) has been used as accountability systems globally [1, 2]. Such test-based accountability systems are based on the beliefs that attaching incentives to standardized achievement tests is a relatively effective policy for improving student performance. Many educational systems consider such a test-based accountability as being an essential part of EFL learners’ learning process, but not only as a mere means for measuring what they know or do not know, moreover, as some English high-stakes tests are likely to bring life-changing consequences to EFL learners’ lives [3]. Hence, effective test preparation for English high-stakes tests is crucial for EFL learners whose futures depend on attaining a particular score [2, 3]. Prior researches have proven that there exists positive correlation between vocabulary knowledge and the performances of English high-stakes tests [e.g. 4–8], as Hiebert [9] stated vocabularies represent knowledge, and knowledge about a text’s topic strongly predicts EFL learners’ comprehension of a text, while Du et al. [4] explained that vocabulary knowledge is vocabulary size which is the number of words a language learner knows, furthermore, it has been gradually accepted among EFL practitioners that EFL learners’ vocabulary size is a key factor in successful comprehending English context, namely, once EFL learners who are automatic with the meanings of the majority of words, these researchers reasoned, will have a strong foundation for comprehension of English context [10–12]. When discussing the correlations between vocabulary knowledge, English high-stakes test preparation and English comprehension, EFL stakeholders seek to identify the so-called critical words, a set of words that account for a substantial percentage of the words in particular contexts, as they believed such words have strong likelihood of appearing in future English context that they may encounter; thus, memorizing critical words is widely regarded as the foundations of English acquisition and one of the most effective learning methods [e.g. 13–15]. However, systematically identify and extract critical words is not effortless. Fortunately, with the advancement of ICT, more and more EFL practitioners relied on corpus-based approaches to create wordlists for facilitating English high-stakes test preparation and language learning and teaching [16, 17].
Most corpus-based approaches were developed on the basis of corpus software, a Natural Language Processing (NLP) tool, and used to process big corpus data for extracting critical words or exploring linguistic patterns in various domains [18–20]. In processing of big corpus data, corpus software computes words’ frequency and range values, and ranks the words based on their frequency or range values respectively. That is, the wordlist produced by corpus software is based on a single variable (i.e. frequency or range) to rank words; but defining the actual importance level of critical words in this way does not seem to be sufficient to meet empirical needs of domains. Moreover, as discovered from previous researches [e.g. 21–23], high frequency words are usually grammar/function words, important elements for constructing meaningful and grammatical sentences, but such words are usually unrelated to the domains’ linguistic analysis needs. Hence, prior researches set further criteria to filter unimportant words for making the optimized wordlist more aligned with the empirical purposes, for example, Bi [24] used a corpus-based approach that based on criteria of removing the basic word list (including function words), frequency variable, and range variable to extract 356 word families outside the 3,300 items within students’ knowledge to form the computer science vocabulary list to address the needs for comprehending Chinese computer science textbook. In addition, Otto [14] used AntConc 3.5.8 [25] to generate keywords of a custom-made civil engineering corpus, then, set exclusion criteria for filtering keywords, which included (1) a keyword’s frequency value must higher than 30, (2) a keyword’s range value must higher than 19 texts and 5 of the 7 registers, (3) excluding non-words, acronyms, or anonymizations, commonly used nouns, proper nouns, and technical vocabulary. The selected keywords in Otto [14] research represented (1) critical to English for specific purposes context, (2) difficult for EFL learners, and (3) well appropriate to teaching through data driven learning. However, neither corpus software nor prior corpus-based approaches [e.g. 14, 19, 24, 26] can simultaneously take the frequency and range variables into consideration for integral computation during the process of defining critical words, so that their results of the critical words may be inaccurate. In other simpler example, when selecting an honor grade student in a class, the scores of all test subjects are usually computed by averaging or weighting to obtain an overall evaluated score, then, the school will arrange this score in descending orders to determine the students’ rank. Hence, the researchers argue that if frequency and range variables cannot be considered simultaneously, the accuracy of defining critical words of wordlist (i.e. high frequency words) will be questioned and challenged.
Recently, many studies have gradually realized the issue of word filtering and parameter integration in corpus research, hence, relevant algorithms were integrated to address such problems [e.g. 23, 27–29]. Notably, Term Frequency-Inverse Document Frequency (TF-IDF) is a widely used statistical method in NLP that measures the importance of terms in a corpus. It is used to identify the most significant terms or words in a document and can be used for various applications such as text classification, information retrieval, and document similarity analysis. For example, Seethappan and Premalatha [30] conducted the automatic classification of euphemistic phrases in a document. They collected a large dataset of 100,000 sentences from various resources and tested several approaches to improve euphemism classification, including lexical n-gram features, feature-weighting schemes, and deep learning classification algorithms. They found that Convolutional Neural Network (CNN) achieved the highest accuracy in detecting euphemisms when using a combination of unigram and bigram features with TF-IDF feature weighting, outperforming other six classification algorithms. In addition, Lin and Bu [31] proposed an advanced TF-IDF mechanism called TF-IDF-COR and develops a CNN-based prediction model to predict the tendency of public emotion and mental health in network public opinion analysis. The TF-IDF-COR mechanism integrates the correlation coefficient of word embeddings to TF-IDF to extract more comprehensive semantic information from text features. Their COR-CNN model outperforms traditional classification models, such as support vector machine and k-nearest neighbors, in accurately judging the emotional tendency of network users. However, although the TF-IDF method can handle the integration problem of word parameters and effectively evaluate the importance of words in the corpus in a weighted manner and rank them, it seems that this method still relies on other algorithms for cluster analysis of words. It is unclear how many words on the word list should be considered important words, and there is no definite answer yet.
In this paper, a potential method that can be used to process frequency and range variables simultaneously called Importance-Performance Analysis (IPA) method. IPA method was firstly proposed by Martilla and James in 1977 [32]. General speaking, IPA is practical and useful method, which can assist decision makers to identify the importance and performance of service/product to customers, then, properly allocate resources to gain higher satisfaction of customers. For a long time, IPA has been active in related fields such as tourism industry [33], information management [34], policy improvements [35], and so on. Even, IPA method is also gradually adopted in education-related researches, for example, Fan [36] adopted IPA method to investigate teachers’ core competencies that striving to engage or better practice maker education in primary and secondary schools. The results of IPA method help to identify the principles to improve the professional development of maker educators. Ha and Park [37] adopted IPA method to analyze the educational content related to elder abuse among undergraduate nursing students in Korea. As they discovered, Quadrant I of IPA showed that demonstrated high levels of both importance and performance. The results implied courses about elder abuse including the educational needs that identified by undergraduate nursing students, are required to improve the professional ability of nurses who care for the elderly. Basically, the IPA method can be applied in various fields as long as it is properly adjusted. The meaning behind it is to integrate two variables simultaneously through IPA graph, then, attribute each indicator to the four quadrants to define their importance and performance level so that decision makers can allocate and invest resources properly.
Likewise, in terms of English high-stakes test preparation, EFL stakeholders must allocate the proper resources and time to learn the critical words that really matter to improve test performance, especially for those who are preparing college/high school entrance exams. During the period of preparation of college/high school entrance exams, English test is only one of the subjects of entrance exams, that is, in addition to the English test, EFL stakeholders also faced with test preparation for other subjects so that the resources and time they can use are very limited. Hence, in order to effectively and accurately identify and extract critical words among English high-stakes test for enhancing academic performances, this paper propose a novel frequency-range analysis (FRA) method which integrates a corpus-based approach and a revised IPA method, furthermore, Taiwan College Entrance Exam of English Subject (TCEEES) from the year of 2001 to 2022 are adopted as an empirical case of English high stake test and the target corpus for verification.
The remainder of this paper is organized as follows: Section 2 reviews prior researches and relevant methods systematically. Section 3 describes detailed steps of the proposed FRA method. Section 4 uses TCEEES as the case for verifying the proposed method. Section 5 discusses the contributions of the proposed method and provides implications of English high-stakes test preparation. Section 6 is conclusion.
Preliminaries
Corpus-based approaches based on a single variable to rank words
Since vocabulary knowledge has a direct influence on EFL learners’ language proficiency, the acquisition of vocabulary is considered to be the most important component of learning English [24, 39]. O’Keeffe et al. [40] claimed that frequency counts are the basis for determining core vocabularies that native speakers use frequently. For a long time, the frequency with which a word is used is an indicator of its utility for learners, and wordlists based on this indicator to create aims to help EFL practitioners to set goals for their students in vocabulary learning [41]. Based on the above theoretical basis of language acquisition, the development of corpus software emphasized on computing the frequency of word occurrences in the target corpus [e.g. 42, 43]. With the current software settings, frequency variable is an important benchmark used to assess the important level of words, moreover, the default word ranking setting of the wordlist is also based on words’ frequency values. From the wordlist interface of AntConc 3.5.9 [42], it is clear to see that the rank of words is sorted by frequency variable, while range variable isn’t considered (see Fig. 1), furthermore, similar feature occurs on the wordlist interface of WordSmith Tool 8.0 [43], although frequency and range (i.e. Texts) variables are both shown on the interface, it still only based on a single variable (i.e. either frequency or range) to sort words (see Fig. 2). However, as Chen and Chang [28] argued, if linguists adopt the corpus software that based on a single variable to rank words and define high frequency words as critical words may cause bias analytical results because it neither optimizes the corpus data nor simultaneously compute words’ frequency and range variables, hence, the alleged critical words with high frequency would be problematic. In modern ICT era, the aforementioned issues in corpus-based researches are awaiting potential algorithms or methods to handle.
AntConc 3.5.9’s single-variable word sorting interface.
WordSmith Tool 8.0’s single-variable word sorting interface.
TF-IDF is a well-known method for computing the weight of terms in textual data, and it is used in information retrieval [31]. The method calculates a numerical representation of the data based on the distribution of terms in it. Search engines use this mathematical tool to retrieve web pages that are relevant to a user query and rank them based on their relevance [30]. Additionally, TF-IDF is used as a text representation technique in NLP. TF-IDF combines two metrics: term frequency (TF) and inverse document frequency (IDF). The term frequency (TF) measures how frequently a word appears in a document, while inverse document frequency (IDF) measures how rare or common a word is across the entire corpus. Next, TF is used as the weighting value and multiplied by IDF value to obtain an aggregate value, which serves as the basis for ranking the words. The higher the TF-IDF score of a term, the more significant it is to the document. Based on these, the calculation of TF-IDF score can be described as follows [30, 44]:
By utilizing TF-IDF, we can extract important words from a corpus and use them for various purposes such as topic modeling, information retrieval, and sentiment analysis. It is a robust method that helps to improve the efficiency and accuracy of NLP tasks.
IPA method was proposed by Martilla and James [32] and used for analyzing customers’ satisfaction towards an organization’s product or service. Traditionally, IPA measures the satisfaction from customer satisfaction survey based on two variables (i.e. importance and performance) of product or service attributes, which represents the importance of a product or service to a customer and the performance of organization in providing that product or service [32, 34]. For a considerable period of time, IPA method has been used for knowing customers’ needs and desires, hence, the organization is able to develop marketing strategies to respond. Related application areas include higher education [45], system evaluation [46], tourism [47], information management [48], and so on. Typically, IPA method used two-dimensional graph to simultaneously consider the two variables and evaluate the weakness and strengths of attributes. In IPA graph, x-axis represent a variable of performance of the attribute, while y-axis represent a variable of importance of the attribute [32, 46]. In addition, the mean scores of the two variables are plotted into a two-dimensional graph as the reference lines of x-axis and y-axis to comprise four quadrants (see Fig. 3).
IPA graph.
Quadrant I (keep up the good work): Attributes in this quadrant represents high performance and high importance, indicating the organizations should continue to maintain their competitive advantages of these attributes. Quadrant II (concentrate here): Attributes in this quadrant represents they are important to customers but the performance does not meet customers’ satisfaction. Hence, allocating resources to improve the performance of attributes in this quadrant should take priority, indicating that it would be a key success factor for organizations’ future development. Quadrant III (low priority): Attributes in this quadrant represents their low importance and low performance to customers, which are considered as no requirement of a great deal of priority for improvement. Quadrant IV (possible overkill): Attributes in this quadrant represents they are considered as low importance to customers, but the organization seems to offer high levels of performance. Thus, the organization should reallocate resources committed to attributes in this quadrant to other quadrants in which the attributes need to improve their performance.
Traditional corpus-based approach is mainly based on words’ frequency variable to sort words and extract high frequency words, however, simply adopt words’ frequency variable to rank them and determine which word is important seems inadequate and risky due to it may misjudge the actual importance level of words, for example, a word with high frequency but too much concentrated in few texts (i.e. few sub-corpus data) may not be critical for EFL stakeholders [28], thus, many researches has started to add words’ range variable as another criterion to filter words [e.g. 22, 49]. However, as discovered, such methods filtered words’ frequency variable and range variable respectively, which lacked of integrative. Hence, in order to simultaneously consider words’ frequency and range variables to conduct data clustering and classification for effectively evaluating words’ importance level, this paper integrates a corpus-based function word eliminating approach [21] and a revised IPA method to develop a novel FRA method that can be used for future corpus-based researches.
To avoid the interference caused by the extreme frequency and range value of function words, this step adopts Chen et al.’s [21] function word list as the exclusion baseline to remove the function words of the target corpus. Input Chen et al.’s [21] function word list to AntConc 3.5.9 as “stop list”, then, generate the optimized wordlist.
When generate the optimized wordlist, AntConc 3.5.9 will simultaneously calculate each word’s frequency and range value. Word’s frequency and range value are the two variables that will be considered and processed in following FRA method.
The framework of FRA method is revised from the IPA method. In order to make the IPA graph can process the corpus data, it is necessarily to re-define x-axis and y-axis. This paper defines x-axis as frequency, while y-axis as range. Notably, the researcher has also set x-axis as range, and y-axis as frequency to find the differences, however, after the researchers input corpus data into this graph, it is discovered that the graph was easy to cause confusion when explaining the meaning of each quadrant and is less align with the spirit of the IPA method. Thus, after comparing the graphs, the researchers decide to set x-axis as frequency and y-axis as range.
Once the x-axis and y-axis are re-defined, the meanings of the four quadrants will also change. Thus, the definitions of the four quadrants also need to be re-defined (see Fig. 4).
FRA graph.
Quadrant I (critical words): Words in this quadrant represents high frequency and high range, indicating the words are frequently and widely occurred on English high stake tests, hence, those words are critical to EFL practitioners. Quadrant II (secondary important): Words in this quadrant represents low frequency and high range, indicating the words are widely occurred (i.e. high range value) but have low frequency values on English high stake tests, hence, those words should be emphasized after EFL practitioners handle the critical words on Quadrant I. As Chen and Chang [28] mentioned when two words have the same frequency value (frequency = 100), one word occurs on 10 articles (range = 10), while another occurs on only one article (range = 1), the importance level of those should not be the same, rather, they believed the word with higher range value should be more critical. In view of this, the researchers define Quadrant II as secondary important. Quadrant III (low priority): Words in this quadrant represents low frequency and low range, indicating the words are not widely occurred on English high-stakes tests, hence, when the time of leaning or teaching is limited, EFL practitioners can take less efforts on them. Quadrant IV (possible overestimated): Words in this quadrant represents high frequency and low range, indicating that although its frequency value is high, it is too concentrated in a few texts (i.e. range value is low), thus its actual importance may be overestimated [28].
Due to the scales of frequency and range values are quite different, it is necessarily to normalize them to make those values are scaled proportionally to facilitate the integration process of the FRA method. Thus, assume that v represent the variable value of a vocabulary, while the v
i
x and v
i
y is the frequency and range value of the ith vocabulary. The value of r
i
x and r
i
y are the normalization of v
i
x and v
i
y, defined as follows.
After the normalization process, the grand mean score of vocabularies’ frequency and grand mean score of vocabularies’ range are calculated. To effectively divide the FRA graph into four quadrants, the researcher set the grand mean scores of frequency and range as the base lines, hence, before plot all vocabularies on the x-axis and y-axis of the FRA graph, the standardized value of w
i
x and w
i
y are the value of r
i
x and r
i
y minus the grand mean scores of vocabularies’ frequency and range,
Once the standardized value of w ix and w iy are calculated, all words based their w ix and w iy values to be plotted on the x-axis and y-axis of the FRA graph respectively, which is to achieve the purpose of distributing vocabularies to the quadrants.
After all vocabularies of the target corpus is distributed to different quadrants, according to this paper’s revised definition of each quadrant, education decision makers (including EFL practitioners and stakeholders) should firstly focus on the critical words in quadrant I; secondly, the words with high range but low frequency in quadrant II; thirdly, the words with high frequency but low range in quadrant IV; finally, the words with low appearing probabilities in quadrant III. Due to English test may just be a part of college entrance exam test subjects, in order to enhance the efficiency and performance of English high-stakes test preparation, the proposed approach will provide a clearer picture for education decision makers to understand where to invest time and resources to learn critical words.
The target corpus
The strong positive correlation between vocabulary knowledge and EFL academic performance make the booming development of critical wordlists [4–8]. Prior researches have also implied the creation of wordlists is the key to success in learning English of different domains [e.g. 13–16]. TCEEES is an English high-stakes test in Taiwan, and the test results are correlated to whether a student can be successfully accepted by his/her ideal university or the judgement that whether a school’s English education is success or fail. Thus, finding critical words of TCEEES will facilitate the developments of English curriculums in senior high schools and students’ learning performances.
The researchers collected the tests of TCEEES from 2001–2022, including 7,948 word types and 68,537 running words. Conducting composition analysis will better understand the target corpus belongs what type of English usages such as English for General Purposes (EGP), English for Academic Purposes (EAP), or English for Specific Purposes (ESP). The researchers adopted the function word list [21], New General Service List (NGSL) [50] and Academic Word List (AWL) [51] as the measurement tools for word classification to further conduct the composition analysis of the target corpus. Notably, the overlap parts of the word lists were eliminated by a well-designed corpus method with the assistance by AntConc 3.5.9 to set clear boundaries between word lists for making word classification task more accurate. From the perspective of word types, NGSL words (N = 4,356) accounted for the largest proportion (54.81%), off-list words (N = 3,166) that didn’t belong to the aforementioned wordlists accounted for the second large proportion (39.83%), while the function words (N = 216) and AWL words (N = 210) accounted for the least proportions (2.72% and 2.64%). For running words, the function words (N = 34,883) accounted for the largest proportion (50.9%) which was consistent with Chen et al.’s [21] research results; NGSL words (N = 26,104) accounted for the second large proportion (38.09%); off-list words (N = 7,172) and AWL words (N = 378) accounted for the minor proportions (10.46% and 0.55%) (see Table 1).
Composition analysis of the target corpus
Composition analysis of the target corpus
After rough inspection of the off-list words, it was discovered that off-list words embraced EGP words (e.g. best, better, deer, elephants, forks, etc.), ESP words (e.g. Capoeira, zebrafish, angelfish, cactus, weta, etc.), names (e.g. John, Jeanie, Morgan, Morrison, Wesla, etc.), and locations (e.g. America, Swiss, Europe, Japan, Africa, etc.). Although the off-list words accounted the second large proportion of word types, their running words accounted for only 10.46% indicating that they infrequent occurred in the target corpus. As prior researches have suggested that EFL practitioners should focus on high-frequency words because they may encounter high-frequency words in future English contexts with higher likelihood [9, 52]. In view of this, NGSL words had the largest proportions in both word types and running words (excluding function words) of the target corpus, which was consistent with that NGSL covers the most frequently used lemmas in the English language [50]. To sum, after the analysis made from word classification and composition analysis, the target corpus was defined by EGP-type due to the target corpus was primary composed by NGSL words, in addition, the results also aligned with the standards of test designs which set by Taiwan college entrance examination center [53].
Corpus software is one of the indispensable and important tools in many corpus-based researches. However, wordlist of traditional frequency-based word ranking which generated by the software seems to be insufficient in facing and processing of complex corpus data. Therefore, many researchers have to set multiple criteria to filter and define critical/core words after the software has produced the wordlist [e.g. 14, 19, 24, 26]. Although this paper also relied on AntConc 3.5.9 to process the target corpus to compute words’ frequency and range values, the wordlist of corpus software still implied some potential problems which summarized as: (1) The wordlist is based on single variable (i.e. frequency value) to rank words, which doesn’t take range variable into consideration, hence the actual importance of top rank words may be challenged; (2) As shown on Fig. 5, high frequency words on the wordlist are all function words and meaningless letters which are waiting researchers to filter and remove manually, indicating the processing of corpus data is inefficient.
AntConc 3.5.9’s single-variable word sorting results.
After obtaining the relevant parameters (i.e., frequency and range values) of words, the researcher calculated the words’ TF-IDF value based on Equations 3). Then, all words were sorted and a word list was generated based on their TF-IDF value. The researcher extracted the top 50 words from the TF-IDF word list for demonstration, as presented in Table 2.
Top 50 words from the TF-IDF word list (partial data)
Top 50 words from the TF-IDF word list (partial data)
After excluding the function words [21], the target corpus remains 7,732 word types and 33,654 running words, then, is processed by the FRA method. The FRA method simultaneously considers the words’ frequency and range variables to conduct data clustering, as Fig. 6 shows, all words were marked on two-dimensional graph of the FRA graph, moreover, they are effectively distributed to four quadrants.
Results of two-dimensional graph of the FRA graph.
Quadrant I includes 1,576 word types that account for 20.38% and 21,043 running words that account for 62.53% (see Table 3). As defined, Quadrant I embraces critical words of the target corpus. The critical words’ frequency and range values are higher than the grand mean scores of frequency and range variables, indicating that they have appeared frequently and widely in TCEEES over the years. When preparing for English tests, EFL stakeholders need to focus on words having both high frequency and range values (i.e. words of Quadrant I) as such words have higher likelihood of appearing in future tests (see Table 4).
Results of word distribution by the FRA method
Top 20 words in FRA Quadrant I (partial data)
Quadrant II includes 705 word types that account for 9.12% and 2,486 running words that account for 7.39% (see Table 3). As defined, words of Quadrant II are secondary important, indicating that after EFL stakeholders have acquired the critical words of Quadrant I, these words should also be learned. Although words of Quadrant II do not have high frequency values, their range values are higher than the grand mean scores of range variable, which means they appear widely in TCEEES over the years (see Table 5).
Top 20 words in FRA Quadrant II (partial data)
Quadrant III includes 5,190 word types that account for 67.12% and 8,015 running words that account for 23.82% (see Table 3). As defined, words of Quadrant III are low priority as their frequency and range values are much lower than the grand mean scores of frequency and range variables, even, there are 3,124 word types with original frequency and range value of 1, which account for 40% (see Table 6).
Top 20 words in FRA Quadrant III (partial data)
ABC represents ABC news that used in a reading comprehension test in 2007 TCEEES. cm represents centimeter.
Quadrant IV includes 261 word types that account for 3.38% and 2,110 running words that account for 6.27% (see Table 3). As defined, the substantial importance of words of Quadrant IV may be overestimated. Although their frequency values are higher than the grand mean scores of frequency variable, their range values are lower than the grand mean scores of range variable, indicating that these words are too concentrated in certain TCEEES and its dispersion is not enough (see Table 7).
Top 20 words in FRA Quadrant IV (partial data)
Comparative analysis of corpus-based approaches, TF-IDF method and the proposed novel FRA method
In order to highlight the advantages of the proposed novel FRA method, this paper compares the FRA method with other three corpus-based approaches [21, 42] and the TF-IDF method [44] that published in Science Citation Index (SCI) and Social Science Citation Index (SSCI) journals (see Table 8); furthermore, it is discussed from three aspects including machine-based function words elimination, simultaneously take frequency and range variables into consideration, and conduct cluster analysis.
Corpus-based approaches vs. the proposed novel FRA method
Corpus-based approaches vs. the proposed novel FRA method
For the aspect of machine-based function words elimination, the wordlist of AntConc 3.5.9 presents all word types of the target corpus; on its wordlist, high frequency words are usually grammar words, it is necessary excluded either manually or automatically before conduct further analysis. However, as Chen et al. [21] claimed that adopting manual elimination of grammar words will reduce the efficiency of corpus-based approaches. Back in 2015, Munoz [22] already adopted Nation’s grammar wordlist (GrWL) and used the Match List tool of WordSmith Tool to remove GrWL before extracting high frequency words of agriculture semi-popularization articles, which greatly improves the analysis efficiency. Similarly, Chen et al. [21] established a function wordlist as a standard exclusion baseline and adopted Stoplist tool of AntConc 3.5.8 [25] to eliminate high frequency function words of the target corpus, which also improved the analysis efficiency. Moreover, when using the TF-IDF method [44] to process the target corpus, commonly occurring terms in various documents, such as function words, letters, or overly generic terms, will have their IDF value calculated as 0 during the calculation process. Consequently, their TF-IDF value will also be 0. As a result, when sorting the word list based on TF-IDF values, these types of terms will automatically be sorted last, achieving the purpose of filtering out such terms using a machine-based method. While, the FRA method adopted Chen et al.’s [21] function word elimination approach, therefore, it inherited their machine-based function words elimination technique.
For the aspect of simultaneously take frequency and range variables into consideration, as mentioned, the wordlists that generated by AntConc and WordSmith Tool are based on a single variable (either frequency or range) to sort words, which cannot simultaneously integrate the two variables. As Chen and Chang [28] claimed, frequency-based word ranking approaches may cause bias when considering high frequency words as critical words due to they didn’t integrally take words’ frequency dispersion and concentration criteria into consideration. Although Munoz [22] adopted General Service List (GSL) [54] and AWL to categorize words into general words, academic words and other words, the high frequency words of each group were processed and determined by WordSmith Tool 4.0 [55], which was still based on a single variable (i.e. frequency) to sort the high-frequency words. In addition, after automatically excluding the function wordlist, Chen et al.’s [21] interpretation of the optimized wordlist or keyword list were based on the optimized results of AntConc 3.5.8, which indicated that neither high-frequency words nor keywords with high keyness values took the range variable into account. The TF-IDF method [44] used the weights to integrate a word’s frequency and range by multiplying TF with its IDF. The multiplication of TF and IDF assigns a higher weight to words that are frequent in a specific document but rare across all documents in the corpus, as such words are deemed more informative and discriminative. In contrast, words that are frequent across all documents in the corpus are assigned a lower weight, as they are less informative and less discriminative. This way, the TF-IDF method [44] can provide a numeric representation of corpus data based on the distribution of words occurring in it, which can be used to rank words based on their relevance to a query or to perform text classification and clustering. While, the FRA method marked each words on the two-dimensional graph, in which the x-axis represented frequency variable and y-axis represented range variable, then, all words were distributed to the four quadrants, indicating that the distribution process were based on the integration results of frequency and range variables.
For the aspect of conduct cluster analysis, cluster analysis (i.e. clustering) is a task that categories a set of objects in the same group, which is a common technique for statistical data analysis [e.g. 56, 57]. The FRA method inherits the cluster analysis ability of IPA method, which makes the FRA method is able to cluster the words with similar important levels into the same group (i.e. each quadrant) to form clearer standards to evaluate and define the substantial importance levels of the words. Different from traditional corpus-based approaches and the TF-IDF method [44], the FRA method does not focus on word ranking, but clusters the words based on the integration results of frequency and range variables. Until now, there is still no standard answer for how many words should to be listed as the so-called high-frequency words (e.g. top 50? top 100? or even top 500?), as Munoz [22] cited Nation’s [52] statements, the dividing line between high-frequency and low-frequency words is based on the researcher’s arbitrary decision. We may wonder under such a man-ruled situation, whether the substantial importance of the so-called high frequency words will be over- or underestimated.
Vocabulary knowledge is considered as the decisive element affecting successful English comprehension and test performances [3, 52], in other words, larger vocabulary knowledge generally leads to better English comprehension and obtain better scores in English high-stakes tests. Although it is better to memorize as many words as possible, for foreign language learners, the time invested in foreign language learning is limited. Therefore, not all vocabulary of the target corpus needs to be memorized. As Hiebert [9] stated, not all words need to be taught, and known critical words can be the basis for expanding EFL learners’ knowledge of rare words. Namely, critical words enable EFL learners to comprehend most parts of the content of English contexts and based on their current vocabulary knowledge to infer the meaning of rare and unlearned words through contextual meanings, and further to understand the information that the context is intended to convey [38, 52]. However, based on the traditional corpus-based word ranking results, there is still no clear answer as to how many core words must be memorized, in other words, it’s almost impossible to define exactly how many top rank words are important and should be learned due to the results were varied depending on the criteria set by different researchers.
Different from traditional corpus-based approaches, the proposed novel FRA method conducts cluster analysis to distribute the words into the four quadrants; among them, the degree of importance given to the words are different, which provide EFL stakeholders a clearer view to understand where they should attribute their time and learning resources to focus on. We suggest that EFL stakeholders should firstly make sure the words of Quadrant I are memorized because these 1,576 word types accounted for more than 60% running words of TCEEES corpus, in addition, these words have both high frequency and high range features, and will appear in future TCEEES with higher likelihood. After acquire the vocabulary knowledge of Quadrant I, EFL stakeholders can move on to the words of Quadrant II, then, IV and III, or they can start to practice skills of listening or reading comprehension based on the foundation of critical words of Quadrant I. In addition, we emphasize that a focus on the critical words does not mean that rare words can be ignore or disregarded, instead, the critical words can be the means of building the vocabulary knowledge that occur less frequently in text.
To sum, it can be known that the proposed novel FRA method simultaneously take words’ frequency and range variables into consideration, in addition, it also set clear boundary lines to cluster words, so that EFL practitioners and stakeholders can measure the existing pedagogical resources and self-learning status to formulate the most appropriate teaching and learning strategies to facilitate English high-stakes test preparations and further to improve test performances.
Conclusions
Having robust vocabulary knowledge will improve the performance of English high-stakes test directly [5, 8–11], thus, as prior researches stated, expanding vocabulary knowledge is an important part of English high-stakes test preparation [4, 15]. Although traditional corpus-based approaches have been widely used for assisting in extracting and categorizing words for building wordlists that used for pedagogical and learning purposes, these approaches cannot effectively and simultaneously integrate frequency and range variables during the process of assessing critical words, which may cause the evaluated importance level of words inaccurate. Moreover, although TF-IDF can effectively integrate the frequency and range parameters of words, it falls short in excluding unnecessary words and cluster analysis, resulting in the generated word list covering many grammatical words and meaningless letters, which affects the accuracy of determining critical words. In order to address this issue, this paper proposed a novel FRA method that integrate a corpus-based approach and a revised IPA method. The proposed FRA method will give education decision makers a clearer view and goal to allocate their learning time and education resources into vocabulary learning and acquisition to enhance language learning efficiency and academic performances. Based on this paper’s revised definition of each quadrant, education decision makers should firstly focus on the words in quadrant I, then, quadrant II, then, quadrant IV, and finally, quadrant III.
To sum, the significant contributions of the proposed novel FRA method embrace: (1) the FRA method uses machine-based technique to exclude grammar/function words to improve the efficiency of corpus-based analysis; (2) the FRA method effectively and simultaneously integrate the words’ frequency and range variables into consideration by the revised IPA method that marks the words on the two-dimensional graph where x-axis represents frequency variable and y-axis represents range variable; (3) the FRA method is able to conduct cluster analysis (i.e. clustering) which categorizes the words with similar important levels into the same quadrant to form clearer standards than traditional corpus-based approaches in evaluating and defining the words’ substantial importance levels.
The major limitation of the present research is that the FRA method has not been coded into corpus software, hence, the analytical process needs to rely on two platforms (i.e. corpus software and statistic software) to process, which lacks of integration. This paper suggests future corpus-based researches should pay attention to the issue of integration of frequency and range variables, and can develop other corpus-based approaches with higher efficient and accuracy for facilitating data process and language learning.
Footnotes
Acknowledgments
The authors would like to thank the National Science and Technology Council, Taiwan, for financially supporting this research under Contract No. MOST 110-2410-H-145-001 and MOST 111-2221-E-145-003.