Language Testing

Assessing translation and interpreting (T&I) is essential in tertiary-level T&I education, professional certification, and foreign language testing. Recently, researchers have explored automating T&I assessment, with large language models (LLMs) emerging as a promising agent for automatic scoring. This study presents one of the first large-scale empirical investigations into the scoring reliability, severity, and validity of GPT-4o and DeepSeek-R1 in English–Chinese consecutive and simultaneous interpreting assessment. Using more than 500 pre-scored samples from the Interpreting Quality Evaluation Corpus (IQEC), the study configured eight e-raters per LLM, systematically varying three scoring parameters: reference availability (zero vs. four references), scoring granularity (segment vs. document-level scoring), and model randomness (temperature 0 vs. 1). A combination of correlation, linear mixed model, and Rasch analyses revealed that: (a) both LLMs demonstrated higher reliability than human raters; (b) DeepSeek-R1 applied significantly harsher scoring patterns than GPT-4o; (c) both LLMs achieved moderately strong correlations with human raters, with overall Spearman’s correlation coefficients ranging from .586 to .700; (d) GPT-4o exhibited higher scoring accuracy than DeepSeek-R1; and (e) LLM-based e-raters’ performance varied significantly across different scoring conditions. These results have important theoretical and practical implications, providing insights into optimizing LLM-based automatic scoring for interpreting and broader language assessment contexts.

The present study explored anxiety and its fluctuation during an artificial intelligence (AI)-based English conversation test as a test validation study. Applying an idiodynamic approach, 13 Japanese university students learning English as a foreign language (EFL) took the test and participated in pre- and post-test interviews, which were analyzed by conducting reflexive thematic analysis. Individual differences in the pattern of anxiety experience fluctuations were observed, while factors affecting their anxiety experiences were also explored. These were classified into either authentic factors observed in both testing and non-testing contexts (e.g., perceived English proficiency, perceived communication skills, a two-way relationship between cognition and anxiety, and features of conversation) or factors unique to the test (evaluative situation and AI’s unnatural behavior). From the perspective of test validation, this study emphasized the importance of paying attention to construct-irrelevant variance, such as anxiety unnecessarily arising from test-specific factors.

We studied Japanese English as a Foreign Language (EFL) learners’ cognitive processes when reading two texts and answering intratext or intertext questions, a format that demands attention toward discrete pieces of information across texts. In Experiment 1, 38 Japanese undergraduate and graduate students read 16 pairs of short texts and answered comprehension questions with four options each. Their eye movements were recorded, which was followed by post-reading interviews. Experiment 2 involved a think-aloud protocol study with 30 Japanese EFL learners. We found that cognitive processes in multiple-text reading largely overlapped across the question types. However, the intertext questions led participants to read more evenly, avoiding reliance on specific answer-related sentences. Questions identifying similarities between texts prompted deeper engagement and utilization of test-taking strategies compared to those detecting differences. The question-answering processes were consistent regardless of whether the participants answered each question correctly. However, those who responded inaccurately tended to frequently return to the texts. Through multiple rounds of data collection, we found that the question answering processes involved in multiple-text reading in a second language were consistent regardless of question type, but intertext questions were effective for eliciting readers’ knowledge of the text structure, one of the higher-level strategies.

This study examined how multiple factors, independent of research quality, influence the scientific impact of language assessment articles. A total of 447 papers published between 2018 and 2022 were investigated using a path analysis approach to identify both direct and indirect ways in which extrinsic factors affect citation counts. A conceptual path model was constructed based on existing literature and assessed using a comprehensive bibliometric dataset. The final model showed an excellent fit to the data (χ² = 65.6, df = 52, p = .097; standardized root mean square residual [SRMR] = .034, root mean square error of approximation [RMSEA] = .024, comparative fit index [CFI] = .980, Tucker–Lewis index [TLI] = .970) and accounted for 57.75% of the variance in citation counts (R² = .5775, p < .001). Among nine examined factors, seven demonstrated significant direct or indirect effects on citation counts: number of self-citations, article age, journal CiteScore, number of countries, number of authors, open access, and research topic. Based on these findings, the study deepens our understanding of citation practices and offers broader implications for citation-based research evaluation in language assessment.

Integrated assessment remains comparatively underused in local placement testing contexts despite attracting increasing attention in the language testing community. Two practical difficulties have been noted: First, rating integrated tasks can be time-consuming, and second, integrated assessments may not offer sufficient diagnostic information about receptive language skills. This Brief Report describes a validation study of an integrated placement test designed to maximize efficiency by minimizing rating bottlenecks that arise from attending to copying, source details, and citation conventions, while at the same time maximizing information about receptive skills by triangulating data from analytic subscores, multiple integrated tasks, and a handful of selected-response items. Internal validation measures of the resulting placement test, including rater agreement metrics and many-facet Rasch measurement, indicate that the test meaningfully distinguishes between test-takers of different ability levels and exhibits strong rater consistency. Diagnostic intake checklists indicate that the test supports student learning and results in few misclassifications, with similar accuracy for productive and receptive skills. These innovations on the placement test were accomplished without perceptibly lengthening testing or rating time. Despite some continuing friction points, the test illustrates the potential of adapting integrated assessment to accommodate local resource constraints.

Recent proposals to reorient validity theory toward justice through a Justice-oriented Antiracist Validity (JAV) framework have sparked vigorous debate in educational and language assessment. While the JAV approach seeks to disrupt systemic inequities by centering antiracist principles and critical quantitative methodologies, its rejection of traditional validity models raises profound philosophical and methodological concerns. In this viewpoint, we respond to Randall et al. by arguing that maintaining rigorous, falsifiable approaches to validation remains essential for meaningful progress toward fairness and justice. We examine historical, ethical, and methodological limitations of the JAV framework, highlight practical challenges in operationalizing its principles across diverse contexts, and question the viability of QuantCrit as an analytic stance. We conclude by advocating for a pluralistic, collaborative profession that acknowledges critical perspectives without abandoning the methodological rigor essential for equitable assessment.