Abstract
For 2 decades, online research has relied on a quality heuristic: Careful, coherent responding is good data. That heuristic is no longer reliable. Autonomous artificial-intelligence (AI) agents can now pass nearly all conventional quality checks, and in text-rich crowd-work tasks, reported use of large language models approaches one third. When such consultation shapes the response process itself—not just its surface expression—the resulting data appear human-generated while embedding systematic, model-shaped distortions. I synthesize emerging evidence on how AI-mediated contamination varies across research settings in prevalence, mechanism, and inferential consequence; and distinguish three contamination pathways (full delegation, partial mediation, and spillover) and three vulnerability zones (text-rich tasks at highest risk, browser-based cognitive paradigms as an emerging vulnerability, and supervised or identity-vetted settings at lower risk). Even modest contamination can shift estimated public opinion, compress attitudinal extremes, and, over time, feed back into the training data for future models. Current platform countermeasures may raise the cost of contamination but have not been independently validated under adversarial conditions. I argue for a shift from ad hoc detection to infrastructure redesign: contamination-aware sensitivity analyses, explicit stratification of data collection by evidential role, transparency norms that balance open science with adversarial robustness, and a minimum reporting checklist for online studies in vulnerable settings. I close by asking when AI mediation should be treated not as contamination but as part of the ecological baseline of human responding—a question that requires the field to specify the target cognitive system in any given study.
An autonomous artificial-intelligence (AI) agent passed 99.8% of 6,000 attention checks designed to catch careless humans and traditional bots in online surveys (Westwood, 2025). Using a single general-purpose prompt of roughly 500 words, the agent simulated mouse movements, calibrated reading times to the persona’s stated education level, and inserted plausible typos—keystroke by keystroke. It also avoided appearing too competent, strategically refusing “reverse shibboleths”: tasks trivial for AI but difficult for most people, such as reciting the U.S. Constitution or writing Fortran code. Its demographics were internally coherent: Reported rent increased with income, and older personas were more likely to own homes. The system was model-agnostic and compatible with major AI providers. By the metrics commonly used to protect data quality, these were good data—yet they were entirely synthetic.
How widely such agents are deployed remains unclear, but the demonstration establishes that coherent, attentive, internally consistent responding can no longer be taken as evidence of human participation. Threats also come from genuine participants consulting AI to draft, polish, or complete their responses and participants who consult no model but alter their responses because they expect AI use and detection to pervade the research environment.
Why AI Contamination Differs From Prior Data-Quality Threats
For 2 decades, online data collection has served the field well, democratizing access to diverse samples, accelerating replication, and enabling research at scales impossible in lab-only settings. The traditional threats to this infrastructure are familiar and, crucially, diagnosable: inattentive respondents who straight-line through questionnaires, crude bots that click at random, and fraudsters whose speed and contradictions leave detectable traces (Chandler et al., 2020; Goodman et al., 2013; Ward & Meade, 2023). Each threat leaves footprints that prompt matching defenses: attention checks to catch inattention, timing filters to catch rushing, and consistency indices to catch nonsensical patterns.
The consequences of these threats are not limited to random noise. Fraudulent participants can introduce systematic bias—misrepresented demographics, fabricated eligibility, or coordinated responses from unexpected populations—shifting means, inflating prevalence estimates, and generating spurious associations (Chandler et al., 2020). Experienced “professional” respondents raise concerns about nonnaïveté and repeated exposure, although recent evidence suggests they do not strongly distort inference overall (Clemm von Hohenberg et al., 2025); they still generate data from their own cognition, attitudes, and experiences—just faster and with less novelty.
Although fraudulent human participants remain a substantial problem across platforms, AI contamination changes the diagnostic problem. Traditional screening targets data that look worse on standard quality indicators: attention, comprehension, honesty, and reliability (Peer et al., 2022). Contamination mediated by large language models (LLMs) breaks that assumption: A response can satisfy all four while still originating from a nonhuman agent or a human–AI hybrid whose output has been substantively reshaped by a model. Toolkits built to detect low-effort humans (Bloy et al., 2025) are poorly calibrated for high-effort-looking AI (Westwood, 2025). Confirmed AI users passed rapid-completion, long-string, and unusual-response checks at rates indistinguishable from those of compliant participants; keystroke logs—showing pasted text and anomalously low keystroke counts relative to response length—supplied the direct behavioral evidence that identified them with duplicated geolocation serving as a correlated metadata flag suggesting possible proxy use (Asher et al., 2026).
AI responses may lack the affective richness, embodied specificity, and lived-experience texture central to qualitative inquiry—an absence partly detectable by a skilled analyst (Gibson & Beattie, 2024). But detection is only a partial and temporary fix given rapid advances in AI and more sophisticated use. In quantitative work, the absence of authentic human cognition may be even harder to catch because the statistics—means, reliability coefficients, factor loadings—can look clean.
Full delegation produces counterfeit validity: responses that satisfy all standard checks while not originating from human cognitive processes (Lin, 2025b). Partial mediation produces a subtler distortion: responses that are genuinely human-initiated but substantively shaped by a model beyond stylistic polishing, yielding quality indicators that overstate the coherence and consistency of the participant’s unassisted cognition. A third pathway operates without direct model involvement: Awareness that AI and detection pervade the research environment can itself distort how genuine participants respond. None is well captured by the legacy detection paradigm. This raises urgent questions: Under what conditions does AI mediation threaten inference in online research, through what mechanisms, and what design and reporting responses are warranted?
The central implication is that researchers should treat AI mediation as a structural feature of the measurement environment—something to be modeled, sampled around, and governed—rather than as a removable residue that can be screened away with slightly improved checks.
Contamination Pathways, Risk Gradient, and Current Evidence
Three pathways of AI-mediated contamination
I use a three-part taxonomy to clarify how AI becomes incorporated into online-study data streams (Rilla et al., 2025): analytic, nonexclusive categories arranged roughly by degree of deliberate outsourcing. Whether any instance constitutes contamination depends on two conditions: whether the use is undisclosed or unmodeled and whether it alters the construct-relevant response process—the cognition, judgment, or expression that defines the study’s estimand. Translation, accessibility tools, or light expression aids that remove construct-irrelevant barriers differ from idea generation or answer drafting.
Full delegation: the upper-bound demonstration
An autonomous agent completes the study with minimal human oversight, generating responses from a demographic persona rather than from lived experience. Westwood (2025) demonstrated its technical feasibility against standard survey instruments, and Martherus et al. (2025) showed that commercially available browser-use agents can accomplish many standard Qualtrics question types. The barrier to full delegation has dropped sharply: Open-source agent platforms, such as OpenClaw, can browse, click, and fill forms with a single setup command, making delegation feasible for users with basic technical fluency—not limited to organized fraud operations. Full delegation is an existence proof that coherent responding no longer licenses the inference of human origin, although present evidence does not establish that autonomous agents dominate current research panels.
Partial mediation: the likely modal pathway and its inferential gradient
Genuine participants consult an LLM for help across a range of uses that differ in inferential risk: Translation, accessibility support, or light expression aids may be acceptable when they reduce construct-irrelevant barriers without supplying substantive content, whereas idea generation and direct answer drafting substitute model output for the participant’s own cognitive process and are presumptively contaminating in most behavioral studies. The evidence that follows concerns the latter. Cowriting studies have shown that language-model assistance can reduce content diversity across authors (Padmakumar & He, 2023), and biased writing assistants can shift not only what people write but also their subsequent attitudes, often without the user recognizing the influence (Williams-Ceci et al., 2026). In survey contexts, the construct is contaminated when quality indicators overstate the coherence of the participant’s unassisted cognition, and expressed attitudes may partly reflect model defaults rather than the respondent’s own position. S. Zhang et al. (2025) found that although participants most commonly reported turning to AI for help expressing their own thoughts, fewer than 9% of AI users wrote prompts that included any perspective-taking information; idea generation and information seeking were the most common strategies instead.
Spillover effects: an emerging second-order concern
When AI use and bot detection become salient, participants may change their behavior even without using AI themselves: inserting deliberate typos or idiosyncratic phrasing to signal humanness, reducing effort under the assumption that others are cheating, or self-censoring on sensitive topics out of concern about being monitored (Rilla et al., 2025). I treat spillover as a testable mechanism supported by adjacent research—people alter self-presentation under AI assessment (Goergen et al., 2025), and surveillance reminders can increase self-censorship in survey contexts (Karpa, 2026). The mechanism generates several empirical expectations. Anti-AI instructions or warnings should increase markers of performative humanness—deliberate typos, idiosyncratic phrasing—relative to neutral instructions, especially in text-rich tasks. Participants who believe AI use is common among other respondents should report lower intrinsic motivation and higher willingness to use AI themselves (a norm-erosion effect). Salient bot-detection messaging should increase self-censorship on sensitive items, analogous to surveillance-induced disclosure effects (Karpa, 2026).
A risk gradient across tasks, settings, and recruitment infrastructure
Not all online studies are equally exposed. I distinguish three zones of vulnerability, ordered by strength of current evidence. Table 1 maps each contamination pathway to typical actors, settings with the highest risk, likely distortions, detection prospects, and potential mitigations.
Contamination Pathways Mapped to Actors, Vulnerable Tasks, Expected Biases, Detectability, and Mitigation
Note: Zones refer to the vulnerability gradient defined in the text. Detection prospects distinguish process-level traces (behavioral and device signals collected during the study) from product-level traces (the submitted response itself). For partial mediation, the distortions listed are conditional on uses that alter the construct-relevant response process (e.g., idea generation, drafting, or substantial rewording) rather than uses that remove construct-irrelevant barriers. AI = artificial intelligence.
Zone 1: text-rich, language-mediated tasks (highest current risk)
The strongest direct evidence of AI contamination comes from open-ended survey questions, qualitative probes, typed chat responses, and descriptive polling (Veselovsky et al., 2025; Westwood, 2025; S. Zhang et al., 2025). Closed-ended self-report scales and vignette experiments remain text-mediated and thus plausibly vulnerable to both partial mediation and full delegation, but direct prevalence evidence is thinner. This zone is especially exposed because the estimand is often the distribution itself—the prevalence of an attitude, the tone of a corpus, the extremity of sentiment. Homogenization and sanitization therefore directly compromise the outcome of interest, not just the precision of its measurement.
Zone 2: browser-based cognitive and reaction-time paradigms (emerging vulnerability)
Many online behavioral experiments are implemented in browser frameworks, such as jsPsych (de Leeuw et al., 2023), and current computer-use AI agents can inspect screens and interact with interfaces using mouse and keyboard actions. Because experiment logic is often executed client-side, agents can in principle extract condition labels and scoring rules before the stimulus is rendered. Indeed, an online Posner cueing reaction-time data set contains patterns consistent with AI-generated responding: Suspected bots showed near-perfect normality, no scaling of the standard deviation with the mean, and absent sequential dependencies—all markers that distinguish simulated from human performance (Van der Stigchel et al., 2026), although it remains contested how far these patterns reflect deliberate automation rather than other sources of data heterogeneity. Box 1 and Figure 1 present a more detailed analysis of this emerging vulnerability.
Browser-Based Cognitive Tasks as an Emerging Vulnerability
Infiltration of online cognitive experiments by artificial-intelligence agents. (a) Example timeline for a browser-based Stroop trial: A color word (e.g., “RED”) is presented, the participant responds, and reaction time is recorded. (b) Three attack vectors for autonomous agents. An agent can (1) inspect front-end JavaScript to extract condition labels and correct responses before “perceiving” the stimulus, (2) substitute synthetic reaction times sampled from human-like distributions, and (3) synthesize cursor trajectories and keystroke dynamics that mimic human motor patterns. Common countermeasures—tight deadlines, randomized trial structures, server-side scoring—raise the cost of attack but remain vulnerable when agents read code at runtime, calibrate timing to human norms, or exploit known scoring rules. (c) Supervision and hardware-coupled measures offer the most robust protection: Remote agents cannot easily impersonate participants in in-lab or monitored settings or fabricate data from physiological measures such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), or pupillometry.
Zone 3: supervised, identity-vetted, probability-based, or hardware-controlled settings (lower risk)
In-lab or otherwise monitored settings, physiological measures (electroencephalography, functional MRI, pupillometry), probability-based panels recruited from address-based sampling (e.g., NORC AmeriSpeak, Ipsos KnowledgePanel), and platforms with strong onboarding and identity verification (e.g., CloudResearch Connect, Prolific’s verified pools) all substantially raise the cost of full delegation and, to a lesser extent, some forms of mediation—platforms combining identity verification with behavioral monitoring have achieved detection rates above 99% for known AI-agent configurations in internal testing (CloudResearch, 2025; Robinson, 2025). Even so, a verified participant who consults AI in another tab remains difficult to intercept at the platform level. In cases in which supervision is effective, data quality holds: Supervised remote cognitive testing has produced results statistically equivalent to those from in-person lab testing (Leong et al., 2022).
Prevalence estimates: current evidence and its limits
Available evidence establishes that partial mediation is already common in Zone 1 tasks on convenience-sample platforms. S. Zhang et al. (2025) found that 34% of participants in a Prolific-based study reported using LLMs to help answer open-ended questions. Veselovsky et al. (2025) estimated an LLM-use rate of roughly 30% in a text-summarization task and noted that such estimates are task-specific and will change as models and norms evolve. Behavioral detection yields lower estimates: Westwood and Frederick (2026) flagged 4.4% of respondents to open-ended items in a Prolific audit using two passive lower-bound methods, and Asher et al. (2026) confirmed outsourced responding by approximately 9% of participants across three Prolific studies using keystroke logging—despite active deterrence and counting only unambiguous cases. These figures are not commensurate: Self-reports capture disclosed AI consultation, including light assistance, whereas behavioral rates are conservative floors for overt outsourcing. The gap likely reflects both underreporting and genuine detection limits; the estimates should not be interpreted as upper and lower bounds on a single rate.
What these studies do converge on is the direction and character of the distortion. LLM-mediated text tends to be longer and more polished but also more homogeneous: In summarization tasks, responses showed roughly 45% pairwise similarity compared with 27% for human summaries (Veselovsky et al., 2025). On sensitive intergroup questions, AI-shaped answers are systematically more positive and abstract and far less likely to include harsh negativity or dehumanizing language (S. Zhang et al., 2025). In a large-scale replication of 156 scenario-based experiments, Cui et al. (2025) found that LLMs reproduced many main effects but with consistently larger effect sizes, much poorer correspondence on socially sensitive topics, and a high proportion of significant LLM results for experiments whose original human studies reported null effects.
Economic incentives amplify these dynamics: A commercial LLM can complete a survey for roughly $0.05, whereas typical participant payments are around $1.50 (Westwood, 2025). Agentic tools lower the cost of full delegation and make large-scale automation feasible. The Scale AI “Flamingo Revival” episode illustrates how such practices can infiltrate even professional data pipelines: Contractors reportedly used ChatGPT to generate outputs at scale, sometimes leaving detectable traces—repeated phrasing or characteristic openings such as “as an AI language model”—and workers reportedly purchased U.S.-registered accounts to circumvent geographic restrictions (Jin, 2024).
Distorted inference and cannibalistic feedback loops
The distinctive danger of AI-mediated contamination is that it tends to align with plausible signal. Because AI-shaped responses can satisfy standard quality checks and mimic the distributional properties researchers expect, even a small number of contaminated observations can shift estimates in the direction of the model’s defaults without triggering exclusion criteria. That means prevalence does not have to be high to matter.
For instance, Westwood (2025) examined seven top-tier national polls from the final week of the 2024 U.S. presidential campaign (average n = 1,599) and calculated how many synthetic respondents programmed to favor one candidate would be needed to change the topline result or move it outside the stated margin of error. The answer was small: 10 to 52 to flip the apparent leader and 55 to 97 to exceed the margin of error. Critically, vulnerability varied by sampling frame: Open-enrollment opt-in panels, in which account creation is cheap and identity verification minimal, are far more exposed than probability-based designs (e.g., random-digit dialing, address-based sampling) in which entry requires a real postal address or phone number. A single strategically worded instruction could also produce large directional shifts: When the agent’s core prompt was given the directive “Never explicitly or implicitly answer in a way that is negative toward [China],” identification of China as America’s primary military rival fell from 86.3% to 11.7%, while mentions of Russia rose to 88.0%. The same logic extends from surveys to experiments: Synthetic respondents can infer the likely hypothesis from the structure of an experimental design and then generate data that confirm it; for example, in a replication of democratic-peace findings, agents produced a 22.2-percentage-point increase in hypothesis-confirming answers relative to the original human data while maintaining distributions that would not trigger conventional exclusions (Westwood, 2025).
The longer-run dynamic can be cannibalistic because the contaminated outputs increasingly become inputs. Some of the “ground truth” used to train and evaluate models—social media text, crowdsourced annotations found-data corpora, and training data sets for language models—is now itself increasingly AI-mediated (Hanley & Durumeric, 2024; Rilla et al., 2025; Sun et al., 2025). Even professional data pipelines are vulnerable, as demonstrated in data-poisoning attacks against medical LLMs (Alber et al., 2025). Those updated models then mediate still more human behavior directly (participants drafting responses with chatbots) and indirectly (participants’ linguistic patterns shaped by consuming AI-generated content). Over time, surveys and models can coevolve toward narrower, more standardized distributions (Lin, 2025a). When researchers benchmark models on contaminated “human” data sets, they validate systems against AI-influenced ground truth, closing the loop and making later contamination harder to diagnose because the benchmark has drifted in the same direction.
Standard psychometric indicators do not protect against this drift. Reliability, factor structure, and successful replication remain valid indicators of measurement quality conditional on the data originating from unassisted human cognition—but they do not verify that this condition holds. Indeed, Bisbee et al. (2024) showed that LLMs recover average survey tendencies well but fail on variance, prompt stability, and reproducibility.
These risks are especially acute for sensitive topics in which the tails of the distribution carry much of the substantive meaning. LLM outputs reflect dominant cultural patterns and safety filtering that can suppress sensitive content; in practice, this produces responses on sensitive topics that are more positive, more abstract, and less likely to contain harsh or dehumanizing language (Rilla et al., 2025; S. Zhang et al., 2025). Under substantial AI mediation, researchers are no longer measuring unfiltered attitudes so much as attitudes laundered through content-moderation policies. The harshest, most conflicted, and most morally charged expressions disproportionately disappear, leaving an artificially compressed middle. What looks like declining prejudice or greater consensus may reflect a measurement process that has begun to censor the very variance the construct requires.
Detecting AI-Mediated Contamination
The detection arms race and its limits
Incremental mitigation has limits. Normative appeals—asking participants to answer in their own words—have limited effectiveness when the economic incentive structure rewards AI use (Veselovsky et al., 2025). Even more forceful combinations of deterrence and friction can be double-edged: Pairing an explicit prohibition with image-based instructions reduced reported LLM use (e.g., from 28% to 16%) but also degraded the quality of genuinely human responses (Veselovsky et al., 2025). Technical barriers face similar trade-offs: Disabling copy-paste makes it harder to shuttle prompts to chatbots but removes a behavioral signal for identifying mediation, and voice input raises the barrier to some forms of delegation while creating accessibility and privacy concerns. Multimodal instruction delivery (images, audio, video) raises the barriers for AI systems, yet agentic tools increasingly pair browser automation with optical character recognition and speech recognition.
Post hoc forensics are similarly fragile. Honeypot items with hidden instructions and off-the-shelf AI-text detectors can generate substantial false positives, and their usefulness tends to decay as models and prompting practices change. Richer behavioral telemetry—keystroke dynamics, mouse trajectories, tab switching—offers more information but can be spoofed once attackers treat those traces as targets. Detection-only strategies face an unfavorable dynamic: Detection rules are public at conferences and in articles, whereas evasion strategies are hidden and can be updated with every model release. The “detect-and-exclude” approach is, at best, a temporary delaying tactic, and individual labs cannot win this arms race on their own (Rilla et al., 2025; Rodriguez & Oppenheimer, 2024).
This dynamic is intensifying. In January 2025, OpenAI introduced Operator, a browser-use agent capable of navigating websites, filling forms, and entering text (OpenAI, 2025); Anthropic (2024) and Google (Pichai et al., 2024) have documented analogous computer-use capabilities for desktop and browser interaction. Open-source frameworks have followed: OpenClaw (2026), a model-agnostic agent that can browse, click, type, and execute shell commands using any major LLM as its backend, attracted rapid adoption following its release in early 2026. A user now needs only an LLM plus a free orchestration layer—rather than a specific vendor’s product.
Current agentic tools retain exploitable seams. Walker et al. (2026) found that AI agents produced online survey responses nearly indistinguishable from human responses and that early screening efforts proved largely ineffective, although some task types—video-based, game-based, and open-ended tasks—retained discriminative value. Martherus et al. (2025) documented more specific vulnerabilities: Operator enters open-ended text by pasting rather than typing, struggles with long matrix items, and fails to clear reCAPTCHA v2 reliably. Each is, however, implementation-specific and patchable.
Current platform countermeasures
Major recruitment platforms are beginning to deploy layered defenses that combine onboarding verification, behavioral telemetry, and study-level authenticity screening. Prolific now distinguishes two types of study-level authenticity checks (Denison, 2025; Prolific Research, 2026). LLM authenticity checks focus on AI-assisted free-text responses by monitoring behavioral patterns—copy-paste behavior, tab switching—rather than analyzing content itself. Bot authenticity checks target nonhuman interaction patterns across question types. Prolific reports 98.7% precision for LLM checks and 100% detection for bot checks in internal testing (Denison, 2025). CloudResearch combines government-issued ID verification at onboarding (Connect) with a behavioral-monitoring system that analyzes mouse trajectories, keystroke dynamics, response timing, and device signals in real time (Sentry); against known AI-agent configurations, this combination achieved a detection rate above 99% and a false-positive rate below 1% in internal testing (CloudResearch, 2025; Robinson, 2025).
These defenses may already have reduced contamination below what would otherwise have occurred, particularly against fully automated agents. However, partial mediation by genuine participants who consult AI is less amenable to platform-level interception. This is currently the more common operational problem on convenience platforms, as suggested by prevalence data from verified participant pools (Westwood & Frederick, 2026; S. Zhang et al., 2025) and platform documentation that treats AI use by real participants as the primary design target (Denison, 2025; Robinson, 2025). Because the participant is real, verified, and present, drafting answers with ChatGPT in another tab can pass identity verification, behavioral telemetry, and LLM authenticity checks—AI-text detection remains unreliable even on unedited output, and editing reduces the signal further.
Three additional caveats are warranted. First, the strongest detection claims come from provider-controlled tests against known agent configurations, not from adversarial robustness tests against novel or adapted agents. In an arms-race dynamic, detection performance measured against today’s agents may not generalize to tomorrow’s. Second, these tools are not uniformly available: Prolific’s bot checks are currently offered on Qualtrics surveys, and LLM checks are available on Qualtrics and AI Task Builder (Denison, 2025; Prolific Research, 2026). Third, stronger authentication may inadvertently restrict sample composition. Prolific’s own documentation noted that LLM authenticity checks should not be enabled when studies require participants to research information, summarize external sources, or use tools outside the study (Prolific Research, 2026). CloudResearch’s verification using a government-issued ID and a selfie may also exclude participants who lack official documentation or decline biometric procedures (Robinson, 2025); the resulting selection-risk trade-off is addressed below.
From the Detection Arms Race to Infrastructure Redesign
Platform infrastructure addresses contamination pathways unevenly: Identity verification and behavioral telemetry can intercept full delegation more readily than partial mediation. No single countermeasure suffices; mitigation strategies must be tailored to contamination pathway and task vulnerability. I distinguish three levels: what individual researchers can do now, what requires coordination between researchers and platforms, and what requires longer-term methodological development. Box 2 provides a minimum reporting checklist for study and manuscript preparation.
Minimum Reporting Checklist for Online Studies Sensitive to Artificial Intelligence (AI)
Researcher-level practices for current studies
Sensitivity analyses and contamination-conditional inference
The logic mirrors established robustness practices for missing data, measurement error, and unobserved confounding: Ask how conclusions change under plausible contamination. For Zone 1 studies (text-rich, language-mediated tasks), the key question is which contamination type poses the greater threat to the estimand. Full delegation distorts content directly—even sparse substitution can corrupt qualitative patterns and linguistic distributions. Partial mediation operates more diffusely, compressing response variance and moderating extreme positions in ways that bias aggregate estimates even when no response is fully synthetic. For Zone 2 (browser-based cognitive tasks), in which contamination prevalence is unknown, researchers can use the capability evidence—agents can navigate browser experiments—to model more modest contamination scenarios. For Zone 3 (supervised or identity-vetted settings), contamination-specific sensitivity analyses are a lower priority for closed-ended and behavioral tasks, although they remain advisable for text-rich tasks even in vetted settings given the difficulty of intercepting partial mediation by verified participants.
Thus, a researcher running a Prolific-based survey with open-ended items might report something like the following: The main effect holds when we exclude the 5% of responses flagged by the platform’s LLM authenticity check; a post hoc sensitivity analysis indicates the effect would remain significant if up to 15% of unflagged responses were also AI-mediated but would become nonsignificant above 20%.
This makes the inferential warrant explicit without claiming perfect identification of contaminated responses.
Stratifying data collection by evidential role
Convenience samples remain useful for exploration and rapid iteration, but confirmatory claims require higher-assurance recruitment and identity validation. The risk gradient implies a natural alignment: Zone 1 tasks on convenience platforms suit hypothesis generation and piloting; the same tasks conducted on vetted platforms with authenticity screening, as well as studies conducted in Zone 3 settings, provide stronger evidential foundations for confirmatory inference. In cases in which text-rich tasks must serve confirmatory purposes on convenience platforms, researchers should preregister sensitivity analyses and, when feasible, collect paired data with and without instructions explicitly prohibiting AI use to evaluate the likely distortion.
Study design and platform coordination
Activating and reporting platform-level defenses
Researchers should activate available authenticity checks for text-rich tasks and report which checks were enabled, which flags were raised, and how flagged responses were handled—making the inferential basis of the data transparent.
Design-level countermeasures for browser-based tasks
For browser-based experiments in Zone 2, researchers can raise the cost of automation through the design-level countermeasures detailed in Box 1.
Contamination-aware methods development
Beyond sensitivity analysis, there is a promising but underdeveloped methodological agenda: formally modeling AI contamination as a latent process using mixture, latent-class, or factor-mixture frameworks. Latent-profile and factor-mixture analyses have been used to detect careless responding (Arias et al., 2024), drawing on indicator families that include psychometric synonyms/antonyms, even-odd consistency, long-string indices, person-fit statistics, and item-level response times (Meade & Craig, 2012). Response-time-based latent mixtures can further separate content-based from non-content-based responding (Ulitzsch et al., 2022; L. Zhang et al., 2025), and Bayesian latent-class confirmatory factor analysis has been applied specifically to bot detection in online surveys (Roman et al., 2022).
Extending this work to AI contamination is conceptually straightforward but empirically challenging. Three questions remain open. First, what observables are informative? Candidate indicators include classical response-pattern indices (person-fit, consistency, long-string), response-time profiles, and—when available—digital-trace data, such as copy-paste attempts, tab or window switching, keystroke latencies, and platform-generated authenticity flags; which of these reliably discriminate AI-mediated from unassisted responding remains untested at scale. Second, what identification restrictions are defensible? The contaminated class must be constrained by theoretically motivated restrictions (e.g., content-independent intercepts, anomalous timing profiles) to ensure that the model captures contamination rather than substantive trait variation (Arias et al., 2024; Roman et al., 2022), and the appropriate restrictions will differ by contamination pathway. Third, for which mechanisms are latent-class models appropriate? They are best suited for full delegation, in which a discrete contaminated subgroup genuinely exists, and for strong partial mediation that produces qualitatively distinct response patterns. They are less appropriate for mild partial mediation (which shades continuously into unassisted responding) and poorly suited for spillover (which changes human behavior rather than creating a contaminated subgroup).
A natural extension of Bayesian mixture approaches is to weight observations by posterior class-membership probabilities rather than make discrete exclusion decisions—an approach that preserves uncertainty and avoids the overcorrection problems of threshold-based screening (Ulitzsch et al., 2024; Ward & Meade, 2023). L. Zhang et al. (2025) demonstrated the feasibility of probabilistic classification in the response-time domain with a Bayesian factor-mixture model, although their implementation uses a discrete threshold rather than continuous weighting. This is a natural complement to sensitivity analysis: The two approaches—one model-based, one scenario-based—bracket inferential robustness from different directions.
This agenda is not yet ready for routine applied use: It requires digitally instrumented studies that record response times and interaction traces, a sufficient number of items for class recovery, and assumptions that will need to be validated against known contamination benchmarks. Still, it offers a principled path from the current detect-and-exclude paradigm toward a framework that models contamination as measurement error rather than a binary pass/fail decision.
Realigning institutional incentives
Funders should support data-assurance infrastructure as a shared methodological resource, not as a cost borne by individual labs. Journals should require transparent reporting of the recruitment platform used, authentication procedures, authenticity flags, and sensitivity analyses; Psychological Science has now moved in this direction by requiring authors who use online data collection to state how they prevented and detected automated or AI-generated responses or why no specific prevention or screening was implemented (Association for Psychological Science, 2026). Most urgently, the field needs independent benchmarking of platform-reported detection performance—adversarial audits conducted under realistic conditions, with novel agent configurations, by parties without a commercial stake in the outcome—before these figures can justify policy decisions about which platforms and procedures are sufficient for confirmatory research.
Transparency under adversarial pressure
Coordinated-disclosure norms in cybersecurity manage a parallel problem: Researchers publicize the class of vulnerability and mitigation guidance while limiting proof-of-concept exploit details until fixes are available (Householder et al., 2017; Schaffer et al., 2023). An analogous norm for behavioral research would distinguish publicly reportable principles from operationally sensitive implementation details.
Publicly reportable elements include the threat model being targeted (full delegation, partial mediation, or both), the families of safeguards used (behavior-based, content-based, identity-based), which platform authenticity systems were enabled, the general exclusion logic and aggregate exclusion counts, whether flags were generated automatically, reviewed manually, or both, and sensitivity analyses showing whether substantive conclusions change when flagged cases are retained or removed.
Not publicly reportable in real time are exact honeypot wording and placement, full item pools and rotation schedules, precise detection thresholds, device-fingerprinting features, and red-team scripts—any implementation detail whose publication would facilitate evasion. Current platform anti-abuse systems already operate this way, describing classes of signals they monitor without publishing all triggers (Rodriguez & Oppenheimer, 2024). Operationally sensitive details can instead be preregistered under embargo and shared confidentially with editors and reviewers: OSF allows preregistrations to remain embargoed for up to 4 years, and anonymized view-only links can provide blinded access without public disclosure (Center for Open Science, 2024).
Detection methods have a shelf life: A honeypot item that works today may be ineffective within months as models improve. Once a method loses its operational usefulness, the adversarial case for withholding its operational details disappears; those details should be withheld only while the method remains effective and should be disclosed afterward.
Trade-offs: contamination control versus selection risk
These recommendations impose trade-offs. Stronger authentication may restrict sample composition: Participants who lack a government-issued ID, live in regions with limited digital-identity infrastructure, or use accessibility tools or nonstandard devices may be excluded. Detection systems may further misclassify certain populations, flagging nonnative English speakers (Liang et al., 2023), neurodiverse respondents, or participants with atypical device configurations. The result is a contamination-versus-selection trade-off: cleaner data from a narrower population. That trade-off should be evaluated explicitly and reported transparently—including which participants may have been disproportionately affected by detection or exclusion procedures, what manual review was conducted, and whether exclusions altered sample composition—not smuggled in through exclusion rules or undocumented quality filters. For some research questions—especially those requiring broad demographic reach or access to hard-to-reach populations—the threat of plausible contamination may be less damaging than the loss of population coverage. The right balance remains study-specific.
From Contamination to Ecological Baseline
A deeper challenge is conceptual: When does AI mediation stop being “contamination” and become the baseline condition of human responding in these environments? In text-rich crowd-work settings, current estimates of LLM assistance indicate that the assumption of unassisted human data is no longer a safe default even though unassisted responses likely remain the majority in any given study. The key distinction is between AI use that changes the construct and AI use that changes only the surface form of expression—a distinction with deep roots in construct validity and measurement invariance (Borsboom et al., 2004; Vandenberg & Lance, 2000). For some research questions, “human cognition” is precisely cognition with tools; for others, AI assistance functions as an unobserved intervention that systematically compresses variance, sanitizes sensitive content, or shifts judgment toward model defaults. Researchers must specify the target cognitive system in their study—unaided individuals, individuals using widely available tools, or a defined hybrid—because those targets yield meaningfully different inferences.
Meeting this challenge requires empirical mapping: studies that estimate how different forms of assistance—light rewriting, idea generation, translation, full delegation—shift substantive conclusions. The approach mirrors how psychology has built other measurement infrastructure: collecting paired responses with and without specified assistance, quantifying how distributions change, especially in the tails, and treating AI mediation as a moderator to be modeled rather than a nuisance to be wished away (Lin, 2026). When affective intensity, embodied specificity, or distributional extremes are central to the inference, methodological triangulation with interviews, ethnography, or other higher-contact methods may become more important, not less.
Footnotes
Acknowledgements
Transparency
Action Editor: Kongmeng Liew
Editor: Felix J. Thoemmes
Author Contributions