A Changepoint Rule for Determining Bicluster Retention Cutoffs in Cheating Detection

Biclustering for Cheating Detection

Cheating detection research has long emphasized two complementary sources of evidence: coordination and speed. Coordination occurs when collusion or item preknowledge produces groups of examinees whose answer patterns are more similar than would be expected under independent test taking (Angoff, 1974; Wollack & Maynes, 2016). Such similarity may appear not only in shared correct answers but also in shared incorrect options, particularly when compromised information is incomplete or inaccurate, leading examinees to select the same distractors (Belov, 2011; Holland, 1996). Speed provides a second source of evidence. When examinees use preobtained answers, their responses to affected items may be unusually fast relative to the time typically required to solve those items (Meijer & Sotaridona, 2006; Sinharay, 2020; Toton & Maynes, 2019). When coordinated response patterns and unusually rapid responding occur among the same examinees on the same subset of items, the evidence for unauthorized information sharing becomes especially compelling.

These two signatures suggest a localized, two-way structure in the data: the signal is concentrated within a subset of examinees and a subset of items. Biclustering is well suited to this structure because it searches for coherent row-by-column submatrices rather than imposing a single one-way clustering structure, such as clustering only examinees or only items (Hartigan, 1972). In cheating detection, a bicluster can therefore be interpreted as a candidate cheating pattern that links a subgroup of examinees to a subset of items on which their responses are unusually similar and unusually fast relative to expected test-taking behavior.

Within the broader biclustering literature (Padilha & Campello, 2017; Xie et al., 2019), this study uses QUBIC, a qualitative biclustering algorithm introduced by Li et al. (2009). QUBIC identifies coherent local patterns rather than assigning all rows or columns to mutually exclusive clusters. This feature aligns well with cheating detection because cheating is typically rare and localized, involving only a subset of examinees and a subset of items. QUBIC also allows overlapping biclusters, which is useful when cheating groups share some compromised items or when an examinee’s suspicious behavior appears across multiple patterns.

To apply QUBIC to response data, item-level behavior is encoded in a matrix that incorporates both response accuracy and response speed. The encoding distinguishes correct responses from each specific incorrect option and includes an indicator for unusually fast responding. Because raw response times are affected by item time demands and examinee speed differences, response times are standardized at both the item and person levels. Item-level standardization compares an examinee’s response time on an item with the response-time distribution for that item:

Item - level standardization : Z item ( i , j ) = RT ( i , j ) − Mean Item R T j SD Item R T j .

(1)

Person-level standardization then compares the item-standardized response time with the examinee’s own response-time tendency across items:

Person - level standardization : Z person ( i , j ) = Z item ( i , j ) − Mean Z item ( i ) SD Z item ( i ) .

(2)

Together, these standardizations help identify responses that are unusually fast for a given item and unusually fast for a given examinee.

The QUBIC algorithm is then applied to the transformed response matrix to detect biclusters that may represent potential cheating groups. The minimum number of columns in a bicluster is set to 2, reflecting the possibility that cheating may occur on only a small number of items. The consistency level is set to 1 so that retained biclusters contain response patterns that match exactly across examinees. In addition, at least half of the items in a bicluster are required to be correct responses, consistent with the expectation that cheating is generally associated with increased accuracy. The maximum number of output biclusters is set to 100 to provide a sufficiently long ordered sequence for the feasible search ranges used later in the changepoint evaluation, while avoiding unnecessary computation. After bicluster extraction, results are aggregated across retained biclusters, and examinees appearing in at least one retained bicluster are flagged as potential cheaters.

Label-Based Benchmark for Changepoint Evaluation

This study proposes a changepoint-based approach for determining how many biclusters to retain when ground-truth cheating labels are unavailable. To evaluate the accuracy of the estimated changepoint, the study defines label-based benchmark cutoffs in data sets for which cheating labels are available. With known labels, detection performance can be evaluated across possible retention cutoffs by examining the trade-off between sensitivity and the false-positive rate. This trade-off is commonly summarized by a receiver operating characteristic (ROC) curve, which traces the increase in sensitivity as additional false positives are introduced.

When only the first few sequentially extracted biclusters are used to flag examinees, the ROC curve is expected to show a steep initial rise, indicating that sensitivity increases substantially with only a small increase in the false-positive rate. As more biclusters are retained, the curve is expected to flatten, suggesting that additional biclusters yield smaller gains in sensitivity while producing larger increases in false positives. The point at which this trade-off begins to deteriorate is treated as the reference changepoint. To operationalize this trade-off, this study evaluates several classification-performance indices at each step of the sequential bicluster extraction process. Specifically, Youden’s index, balanced accuracy, and the F1-score are computed for each possible retention cutoff, ranging from retaining only the first bicluster to retaining all extracted biclusters. For each index, the label-based benchmark cutoff is identified as the retention cutoff that maximizes the corresponding criterion.

Although several indices can be used to summarize classification performance, their usefulness differs in cheating detection contexts, where the target class is typically rare. Youden’s index, defined as sensitivity plus specificity minus one, is commonly used to summarize the joint performance of sensitivity and specificity. However, because it weights sensitivity and specificity equally, it may not fully reflect the operational cost of false positives in cheating detection. Balanced accuracy, defined as the arithmetic mean of sensitivity and specificity, similarly gives equal weight to correctly identify cheaters and noncheaters. When noncheaters greatly outnumber cheaters, even a small false-positive rate can lead to many falsely flagged examinees. Therefore, balanced accuracy may make a retention cutoff appear acceptable even when its precision is low. In contrast, the F1-score incorporates precision directly. Because it is defined as the harmonic mean of precision and sensitivity, the F1-score penalizes cutoffs that increase sensitivity by flagging too many noncheaters. Accordingly, this study uses the F1-score as the primary criterion for identifying the label-based benchmark cutoff.

Changepoint-Based Bicluster Retention Rule

This study proposes a changepoint-based method that does not require cheating labels but is designed to approximate the label-based optimal retention cutoff. Following Lee and Vispoel (2025a, 2025b), a p-value is computed for each extracted bicluster to evaluate whether the observed number of examinees sharing the bicluster response pattern is larger than expected by chance. For each item included in a bicluster, the shared response option defining the bicluster pattern is identified. The marginal probability of observing that response option is then estimated from the full response matrix for each item. These item-level probabilities are multiplied to obtain the expected probability of observing the full bicluster response pattern by chance. This probability is then multiplied by the total number of examinees to obtain the expected number of examinees matching the pattern. The observed count is defined as the number of examinees included in the bicluster. Finally, a Poisson upper-tail probability is computed as the probability of observing at least this many examinees under the expected count. Smaller p-values therefore indicate biclusters whose shared response pattern occurs more frequently than would be expected.

Importantly, these bicluster-level p-values are produced within an inherently sequential extraction process. The QUBIC algorithm first constructs a weighted graph in which examinees are represented as vertices, and edge weights reflect the number of test items on which two examinees share identical nonzero encoded responses. After applying a Poisson-based filtering step to remove edges that could plausibly arise from random similarity, QUBIC sorts the remaining edges by weight and uses high-weight edges as candidate seeds. Because larger edge weights indicate stronger pairwise response similarity, biclusters formed earlier in the extraction process are initialized from examinee pairs with more extensive shared response patterns. These early seeds therefore provide a stronger basis for expanding rows and columns under the specified consistency criterion. As high-weight edges and their associated coherent structures are used, subsequent biclusters are initiated from low-weight edges, making later biclusters more likely to reflect weaker localized structure. As more biclusters are retained, the procedure increasingly admits patterns that may reflect noise rather than cheating-related signal.

Because conventional significance-based filtering treats each bicluster as a separate hypothesis test, it does not explicitly account for this ordered extraction process. To incorporate the sequential structure of QUBIC, bicluster retention is formulated as a changepoint problem. The p-value associated with each extracted bicluster is treated as an indicator of relative pattern strength within the data set, and the ordered sequence is used to identify the position at which the extracted biclusters shift from stronger, signal-dominant patterns to weaker, noise-dominant patterns. The changepoint is estimated using piecewise linear regression fitted to the ordered sequence of bicluster p-values. The estimation is carried out in three steps, as described below.

Step 2: Log Transformation and Cumulative Representation

Because bicluster p-values are often extremely small, the raw p-value scale provides limited resolution for distinguishing differences across the extraction order. Values that differ substantially can appear similarly close to zero, obscuring changes in bicluster strength. Therefore, p-values are analyzed on a logarithmic scale (Box & Cox, 1964). Let  p k  denote the p-value associated with the  k th extracted bicluster among the  m  biclusters within the feasible search range. The transformed value is defined as

y k = − lo g 10 ( p k ) , k = 1 , … , m .

(3)

To reduce local variability and highlight structural changes, a cumulative representation of the transformed values is then constructed:

S q = ∑ k = 1 q y k , q = 1 , … , m .

(4)

This cumulative formulation produces a sequence whose local slope reflects the magnitude of the transformed p-values over successive extraction orders. When early biclusters contain strong cheating-related signals, their small p-values yield large transformed values, causing  S q  to increase rapidly. As later biclusters increasingly reflect weaker residual structure or noise, their larger p-values yield smaller transformed values, causing the growth of  S q  to flatten. The changepoint is therefore interpreted as the location where the growth rate of  S q decreases, indicating a transition from signal-dominant to noise-dominant detections.

Step 3: Piecewise Linear Changepoint Estimation

To estimate the transition from signal-dominant to noise-dominant biclusters, the cumulative evidence sequence S q is modeled as a piecewise linear function of extraction order. Piecewise linear regression identifies one or more breakpoints that divide the ordered sequence into contiguous linear segments. These breakpoints are estimated by minimizing the total residual sum of squares across all segments (Bai, 1997; Zeileis et al., 2003). In this framework, a breakpoint represents an extraction order at which the rate of cumulative evidence accumulation changes, separating adjacent regimes with different slopes.

Let x q  denote the extraction-order index of the  q th bicluster within the feasible search range. For segment  i , bounded by start and end indices  s i  and  e i , respectively, the piecewise linear model is written as

S q = β 0 i + β 1 i x q + ε q , s i ≤ x q ≤ e i .

(5)

For each segment i , the slope is computed as

Δ i = S e i − S s i e i − s i ,

(6)

where Δ i  represents the rate of cumulative evidence accumulation within that segment. Early segments are expected to have steeper slopes because they contain biclusters initialized from stronger shared response patterns. In contrast, later segments are expected to have flatter slopes as the retained biclusters increasingly reflect weaker residual structure or noise.

The changepoint is determined by comparing the slopes of adjacent segments. Slope ratios are computed for consecutive segments, and the breakpoint with the largest decrease in slope is selected as the estimated transition point:

k ^ = arg max Δ i Δ i + 1 .

(7)

Thus, k ^  represents the extraction order at which the cumulative evidence curve shows the largest relative reduction in growth rate, indicating the transition from signal-dominant to noise-dominant biclusters. Biclusters extracted up to and including  k ^  are retained, and examinees appearing in at least one retained bicluster are flagged as potential cheaters.

Methods

The empirical evaluation used the credentialing-test data set reported by Cizek and Wollack (2016), which contains two independent test forms with identified cheaters and compromised items based on operational investigations. Form 1 includes 1,636 examinees with 46 identified cheaters, and Form 2 includes 1,644 examinees with 48 identified cheaters. Each form contains 170 multiple-choice items, with 64 items flagged as compromised in Form 1 and 61 in Form 2. Response accuracy, response time, and response choice were used for biclustering.

For each form, biclusters were extracted using QUBIC (Zhang et al., 2017) implemented in R (R Core Team, 2024). The proposed changepoint procedure was then applied to the ordered sequence of bicluster p-values. To limit the changepoint search to a practically plausible range, maximum examinee-level flag rates of 0.050, 0.075, and 0.100 were considered.

To establish label-based benchmarks for evaluating the proposed changepoint rule, known cheating classifications were used to assess detection performance across candidate bicluster-retention cutoffs. At each cutoff, examinees appearing in at least one retained bicluster were classified as flagged, and performance was summarized using the F1-score, balanced accuracy, and Youden’s index. For each metric, the label-based benchmark was defined as the retention cutoff that maximized the corresponding performance criterion. The changepoint estimated by the proposed rule was then evaluated by comparing its location with these benchmark cutoffs and by comparing the detection performance obtained at the estimated changepoint with that obtained at each label-based benchmark.

Results

The ROC trajectories and corresponding cutoffs for both forms across the three feasible search ranges are presented in Figure 1. For Form 1, the F1-maximizing benchmark cutoff yielded a sensitivity of 0.326 and specificity of 0.982 across all three ranges. This cutoff occurred near the region where the ROC trajectory began to flatten, suggesting diminishing gains in sensitivity relative to additional false positives. In contrast, the balanced accuracy and Youden’s index benchmarks selected later retention cutoffs. Under the 0.050 and 0.075 search ranges, these cutoffs yielded a sensitivity of 0.370, with specificity ranging from 0.965 to 0.967. Under the 0.100 search range, both benchmarks moved farther along the flattened portion of the ROC trajectory, yielding a sensitivity of 0.435 and specificity of 0.933 for balanced accuracy and 0.927 for Youden’s index. Consistent with the discussion in the background section, these results illustrate the operational rationale for using the F1-score as the primary label-based benchmark in this study.

OPEN IN VIEWER

Figure 1.

ROC trajectories showing estimated changepoint and label-based benchmark locations.

The proposed changepoint rule yielded the same sensitivity as the F1-maximizing benchmark, 0.326, with specificity ranging from 0.973 to 0.975 across the three search ranges. Although these specificity values were slightly lower than those of the F1 benchmark by 0.007 to 0.009, the estimated changepoints remained closely aligned with the F1-based reference.

A similar pattern was observed for Form 2. The F1-maximizing benchmark yielded a sensitivity of 0.417 and specificity of 0.978, with the cutoff occurring near the region where the initially steep ROC trajectory began to flatten. In contrast, the balanced accuracy and Youden’s index benchmarks selected the same later retention cutoffs. These cutoffs yielded a sensitivity of 0.438 under the 0.050 search range and 0.458 under the 0.075 and 0.100 search ranges, while reducing specificity to approximately 0.963 and 0.960, respectively. The proposed changepoint rule again yielded the same sensitivity as the F1 benchmark, 0.417, with specificity ranging from 0.964 to 0.966 across the three search ranges. Although specificity was slightly lower than that of the F1 benchmark by 0.012 to 0.014, the estimated changepoints remained closely aligned with the F1-based reference.

Across both forms, the proposed changepoint rule identified cutoffs that closely aligned with the F1-based benchmark despite using no ground-truth cheating labels. The estimated changepoints were located near the point at which the ROC trajectories began to flatten. This pattern was consistent across all three feasible search ranges.

Methods

A simulation study was conducted to evaluate the proposed changepoint-based retention rule across a range of testing and cheating conditions. The data-generation procedure followed Lee and Vispoel (2025a). The sample size was fixed at 5,000 examinees across all conditions, and the proportion of cheaters was fixed at 0.05 to reflect a low-prevalence operational setting. Four cheating scenarios were crossed with test length (50, 100, and 150 items) and the proportion of compromised items (0.25, 0.50, and 0.75), resulting in 36 experimental conditions. For each condition, the proposed changepoint rule was applied to determine a retention cutoff without using cheating labels. To examine sensitivity to the feasible search range, the proposed rule was evaluated under two maximum examinee-level flag rates, 0.075 and 0.100. The 0.050 search range was not used in the simulation because the simulated cheating prevalence was fixed at 0.05, and changepoint estimation requires the search range to extend beyond the expected cheating proportion. Sensitivity and specificity were then used to evaluate the extent to which detection performance at the estimated changepoint reproduced that of the F1-based benchmark.

Results

Differences in sensitivity and specificity between the estimated changepoint and the F1-based benchmark are summarized in Figure 2 across cheating types, test lengths, and compromised-item proportions. The main results are based on the 0.100 feasible search range. Results under the 0.075 search range were nearly identical to those under the 0.100 search range. Averaged across conditions, the difference in sensitivity between the two search ranges was 0.003, and the corresponding difference in specificity was less than 0.001. Under the 0.100 search range, the replication-level difference between the estimated changepoint and the F1-based benchmark varied more for sensitivity than for specificity, with average standard deviations of 0.099 and 0.007, respectively, across simulation conditions. Under the 0.075 search range, the corresponding average standard deviations were 0.099 and 0.006. Detailed condition-specific means and standard deviations for the 0.100 and 0.075 search ranges are reported in Supplementary Tables S1 and S2, respectively.

OPEN IN VIEWER

Figure 2.

Differences in sensitivity and specificity between changepoint and F1-based cutoffs.

Across conditions, the estimated changepoint consistently yielded lower sensitivity than the F1-based benchmark, while specificity was comparable or slightly higher in nearly all conditions, with two minor exceptions under independent-time fast cheating. Specifically, specificity differences were slightly negative for the 50-item test with 75% compromised items (−0.001) and the 150-item test with 25% compromised items (−0.002), but these differences were negligible in magnitude. The magnitude of the sensitivity reduction varied by cheating type. Mean sensitivity differences were smallest under independent-time fast cheating (−0.096) and largest under dependent-time fast cheating (−0.187), whereas independent-time moderately fast and dependent-time moderately fast cheating showed intermediate reductions (−0.127 and −0.106, respectively). Mean specificity differences were much smaller in magnitude, at 0.001, 0.010, 0.013, and 0.019 for independent-time fast, independent-time moderately fast, dependent-time fast, and dependent-time moderately fast cheating, respectively. Overall, these results indicate that the proposed changepoint rule tended to select somewhat more conservative retention cutoffs than the F1-based benchmark, particularly under dependent-time fast cheating, although the specificity advantage was not uniform across all conditions.

The condition-specific results further show that this conservative tendency was not uniform across test lengths or compromised-item proportions. Mean sensitivity differences were larger for the 100- and 150-item tests (−0.159 and −0.147, respectively) than for the 50-item test (−0.082), whereas mean specificity differences remained small across test lengths (0.012, 0.010, and 0.010, respectively). A less consistent pattern was observed for compromised-item proportion. The mean sensitivity difference was largest when 50% of items were compromised (−0.153), compared with 25% (−0.101) and 75% (−0.133), while mean specificity differences remained modest and decreased from 0.016 to 0.010 and 0.006 across the 0.25, 0.50, and 0.75 conditions, respectively. Overall, the estimated changepoint remained close to the F1-based benchmark in terms of specificity, but sensitivity losses were more pronounced and more variable across conditions, indicating that the proposed rule more often erred on the side of retaining fewer biclusters than the label-based benchmark.

Summary and Interpretation of Results

The empirical findings provide initial support for the conceptual rationale of the proposed rule. Across both operational test forms, the estimated changepoints closely aligned with the F1-maximizing retention cutoffs and occurred near the point at which the ROC trajectories began to flatten. Because this region reflects diminishing gains in sensitivity relative to additional false positives, the results suggest that the sequential weakening of bicluster evidence can serve as a meaningful structural signal for retention decisions. In other words, the changepoint rule appeared to identify an operationally relevant point beyond which retaining additional biclusters became less beneficial.

The simulation findings extend this evaluation across a broader range of controlled conditions. Agreement was strongest under independent-time fast cheating and weakest under dependent-time fast cheating, suggesting that the distinctness of the evidence transition may vary across cheating mechanisms. The closest agreement occurred under independent-time fast cheating with 150 items and 25% compromised items, where the differences from the benchmark were minimal for both sensitivity and specificity. This pattern is notable in relation to the empirical analysis: although the cheating mechanism in the operational data was unknown, the 170-item test forms likewise produced benchmark-aligned sensitivity with only slightly lower specificity.

By contrast, differences across test length and compromised-item proportion did not follow a consistent monotonic pattern. This suggests that changepoint accuracy may depend less on any single design factor in isolation than on how clearly the ordered bicluster sequence separates stronger cheating-related evidence from weaker later evidence. When strongly informative biclusters are followed by a relatively abrupt decline in evidential strength, the cumulative evidence curve should exhibit a clearer changepoint and the estimated cutoff should more closely approach the F1-based benchmark. When that decline is more gradual or irregular, the estimated changepoint may deviate more from the benchmark.

Practical Implications

A key contribution of this study is to frame cutoff selection as a data-driven retention problem rather than as a test of whether each bicluster meets a fixed nominal significance threshold. Nominal alpha levels are appropriate when the objective is to evaluate statistical rarity under a specified null model, as in significance-based approaches such as Belov (2011). However, biclustering-based cheating detection poses a different operational question: How many sequentially extracted biclusters should be retained, and consequently how many examinees should be flagged for further review? The proposed changepoint rule addresses this question by identifying where bicluster evidential strength begins to weaken substantially across the ordered extraction sequence. In this respect, it is more consistent with a data-driven pattern-detection framework than with fixed-threshold decision rules.

It also differs from approaches that require externally specified contamination rates (Kamalov et al., 2021) or formal prior and loss-function assumptions (Sinharay & Johnson, 2025), because it derives the retention cutoff from the observed structure of the bicluster evidence sequence itself. Although the procedure uses a feasible search range to avoid implausibly late cutoffs, this range is not intended to specify the true cheating prevalence. Rather, it serves as a loose operational constraint that should be broad enough to include the expected transition from stronger to weaker bicluster evidence while limiting unnecessary searches over later, weaker patterns. The sensitivity analyses using alternative search ranges showed nearly identical results, suggesting that the findings were not dependent on a precisely specified upper bound.

This distinction is practically important because operational testing programs rarely have known cheating labels or a principled basis for selecting a universal p-value cutoff. By providing a label-free and data set–adaptive stopping rule, the proposed method offers a more defensible way to define the review pool while reducing reliance on ad hoc threshold choices that may retain too many or too few biclusters across testing contexts. The changepoint rule can therefore function as a transparent triage mechanism: It prioritizes biclusters supported by stronger evidence and limits the extent to which weaker later patterns drive examinee flagging. The resulting set of flagged examinees can then be combined with other forensic information, such as response-time anomalies, seating arrangements, or audit records, to support downstream investigations.

The practical value of this data-driven retention rule is further supported by its performance under additional noncheating sources of response aberrance. In the simulation study, two forms of noncheating rapid guessing were included: time-limited rapid guessing near the end of the test and low-motivation rapid guessing on a subset of difficult or late items. Despite the presence of these behaviors, the estimated changepoint generally approximated the F1-based benchmark cutoff. This finding suggests that the proposed rule remained informative even when cheating-related patterns coexisted with noncheating sources of anomalous accuracy and response-time behavior, a setting that more closely reflects the complexity of operational testing data.

The proposed procedure also appears feasible for use with large testing data sets. The simulation study included 5,000 examinees and tests ranging from 50 to 150 items. Across these conditions, bicluster extraction and changepoint estimation together required an average of 0.667 seconds per analysis, with a standard deviation of 0.296 seconds, using eight cores and 8 GB of RAM. These results suggests that the procedure is computationally feasible for large-scale cheating-detection applications.

Limitations and Future Research

Several limitations of this study suggest directions for future research. First, the proposed method requires a sufficiently long sequence of extracted biclusters to support changepoint estimation. The piecewise linear regression used in this study estimates an intercept and slope within each segment; therefore, each segment should contain at least three biclusters to leave a residual degree of freedom for segment-level fitting. For a model with one changepoint and two adjacent linear segments, a total of six observations is a minimum under this specification. When fewer than six observations are available, changepoint estimation may not be identifiable. Even when this minimum requirement is met, estimates based on very short bicluster sequences may be unstable because limited information is available to distinguish a meaningful shift in the cumulative evidence trajectory from local fluctuation. Future research should examine alternative retention rules or modified changepoint procedures for settings in which only a small number of biclusters is extracted.

Second, the simulation results showed that the proposed rule consistently yielded lower sensitivity than the F1-based benchmark, whereas specificity differences were generally small. This pattern indicates that the estimated changepoint tended to stop earlier than the label-based benchmark, resulting in fewer retained biclusters and, consequently, fewer flagged examinees. Although such a pattern may limit unnecessary expansion of the review pool, it also means that some cheating examinees who would have been flagged under the F1-based benchmark may not be identified by the proposed rule. Future research should examine whether alternative evidence summaries or hybrid decision rules can improve sensitivity while maintaining the generally small specificity differences observed in this study.

Third, although the simulation design systematically varied cheating type, test length, and the proportion of compromised items, it was limited to a specific set of testing conditions. Other operationally relevant factors, such as sample size, cheating-group size, and cheating prevalence, were not examined. In addition, although the simulation included two rapid-guessing conditions as noncheating aberrant behaviors, it did not fully represent more complex operational settings in which multiple aberrant mechanisms may coexist with cheating-related patterns. Future studies should evaluate the proposed rule under a broader range of conditions and validate its performance using additional empirical data sets. Such work would provide a more comprehensive assessment of the method’s robustness and generalizability across diverse testing contexts.