A Tool for Agreement and Alignment Analysis in Binary Rating Tasks: The R Package scindex

Description: Inter-rater reliability in binary rating tasks is commonly assessed through chance-corrected agreement coefficients such as Cohen’s κ and Fleiss’ κ (Cohen, 1960; Fleiss, 1971). However, κ is sensitive to prevalence imbalance and marginal distributions, so observed agreement may not fully represent convergence in raters’ decision criteria (Byrt et al., 1993; Feinstein & Cicchetti, 1990; Warrens, 2015). Signal detection theory provides a framework in which binary judgements are modelled as decisions based on latent evidence and rater-specific thresholds (DeCarlo, 1998; Green & Swets, 1966). The Strategic Convergence Index (SCI) is a model-based quantity that summarises convergence in rater decision thresholds within a signal detection-theoretic framework (Gianeselli, 2026b). SCI complements classical agreement coefficients by distinguishing agreement in observed categorical outcomes from alignment in the latent decision structure of binary rating tasks (Gianeselli, 2026c). scindex (Gianeselli, 2026a) is an R package (R Core Team, 2026) for inter-rater reliability analysis in binary classification tasks involving two or more raters. It standardises user-supplied rating data into a common long-format structure, computes Cohen’s κ for two raters and Fleiss’ κ for multiple raters, and, when ground-truth labels are available, estimates rater-specific hit rates, false-alarm rates, sensitivity, specificity, and decision thresholds using signal detection-theoretic transformations. The package provides structured summaries and threshold-based diagnostics for binary rating tasks in which similar agreement coefficients may reflect different underlying threshold configurations.