Evaluating LLM-Based Coders in Psychological Assessment: A Validation Framework With Application to the Rorschach Morbid Content Variable

Abstract

Large language models (LLMs) are increasingly used to support psychological assessment, but standards for evaluating their scoring accuracy remain limited. This article introduces a clear, reproducible validation framework to evaluate LLM-based scoring systems. The framework separates pre-validation steps (e.g., balancing base rates, refining prompts, and comparing models) from a standardized validation phase focused on reliability and validity benchmarks. We demonstrate its application with a case study of Morbid Content (MOR) scoring in the Rorschach task, using a two-agent LLM workflow. In an independent dataset (n = 84; 2,176 responses) with natural MOR base rates, the final LLM coder showed good response level agreement (kappa = .72–.74) and excellent protocol level agreement (ICC = 0.94–0.95) with assessors, near-perfect consistency with itself (ICC = 0.97–0.99), and replicated external validity (r = .59–.71) that matched human coders (r = .54–.65). This article offers a practical guide for evaluating automated coders in psychological testing and discusses practical decisions and ethical considerations.

Keywords

psychological assessment Rorschach morbid content psychometrics automated scoring validation framework

Introduction

Early automated-scoring efforts in psychological assessment were largely purpose-built using machine learning, natural-language processing (NLP), or computer-vision pipelines trained for specific instruments or behaviors. These systems often approached human-range reliability and reproduced expected criterion relations across diverse applications. Examples include open-ended story comprehension in children (Devine et al., 2023), psychotherapy session coding (Ewbank et al., 2021; Tanana et al., 2016), episodic-memory narratives (Shen et al., 2023), and image-based scoring in neuropsychological tests (Park et al., 2023).

Building on this technical foundation, language-based assessment researchers have increasingly leveraged advances in NLP. Whereas classical NLP relied on training models for a specific task, contemporary transformer models use pre-trained systems (Qiu et al., 2020). In particular, large language models (LLMs) can be instructed to perform classification tasks (i.e., using prompts), rather than training a new model from scratch (Brown et al., 2020; Liu et al., 2023). A prompt is a set of instructions, guidelines, decision rules, and illustrative examples that a pre-trained model uses as its primary input to predict the most likely outcome and generate a natural-language response. The LLMs can be operationalized as simply as a chatbot like ChatGPT or Gemini, or as more technical, such as using an Application Programming Interface (API) to access a specific model (e.g., GPT 4.1, GPT 4.1-mini, or Gemini 2.0 Flash).

There are various strategies for creating a prompt, such as providing a few examples of what qualifies or not (zero- or few-shot prompting; Brown et al., 2020) and instructing the model to provide intermediate reasoning before the final judgment (chain-of-thought; Wei et al., 2022). Prompts can be iteratively improved using error analysis, in which developers systematically review disagreements or problematic outputs to identify recurring failures and refine decision rules, exclusions, and examples. For a more thorough reading on prompting methods in NLP, see Liu et al. (2023), and for a shorter review applied to mental health, see Priyadarshana et al. (2024). For the current paper, we use the terminology LLM-based coder (or LLM coder for brevity) to describe a prompt-guided LLM performing a classification task, where developers specify a scoring rubric, decision rules, or examples directly in the prompt.

A growing set of studies demonstrates that LLM coders can function as raters with promising psychometric properties. Examples include ratings with the Social Cognition and Object Relations scales – Global Rating Method (Dauphin & Siefert, 2025), divergent-thinking scoring on the Alternate Uses Task (Organisciak et al., 2023; Saretzki et al., 2025), and “LLM rating scales” applied to therapy transcripts (Eberhardt et al., 2025). Preprints further suggest that LLM coders can reasonably score open-ended Theory-of-Mind responses (Wang et al., 2025) and support risk ratings in social media (Jeon et al., 2024). Overall, the conclusion is that LLMs and prompt engineering unlock new possibilities for language-based assessment.

In this emerging landscape, some argue that LLMs can supplement the field’s dominant reliance on self-report (Brickman et al., 2025), whereas others argue that natural-language assessments could eventually replace self-reports and shift the direction of the field (Kjell et al., 2024). Despite these differences, both viewpoints acknowledge the importance of developing LLM coders and evaluating their psychometric quality.

Developing LLM coders and introducing them as a new procedure in psychological assessment workflows is fundamentally a measurement change. It changes how observations are transformed into variables. Consequently, the existing evidence of validity for a measurement using human coders, which forms the foundation for interpretations (American Educational Research Association [AERA] et al., 2014), may not apply to LLM coders. Guo et al. (2024) reviewed the use of LLM in mental health applications and expressed serious concerns regarding the accuracy and reliability of LLM-generated output, as well as the lack of benchmarked frameworks. As a result, similar to other researchers (e.g., Kjell et al., 2024), we argue that LLM coders must be evaluated with the same rigor as any scoring algorithm or rater system, with a focus on reliability and validity.

From a psychometric perspective, supplementing human coders with LLM coders shifts the measurement process. This shift raises at least three questions. First, consistency in the LLM coders: Do independent sessions of the same LLM coder agree with each other as much as independent human raters? Second, LLM-human agreement: To what extent do LLM coders agree with independent human coders? Third, validity of LLM scores: Do the LLM-generated scores relate to external variables in the same direction and magnitude as human-coded scores? The first question targets the reliability of the LLM coder, and the last two address validity issues, with the third being an extension of the second. We address these questions by proposing a practical validation framework for LLM-based coders in psychological assessment. We view an LLM coder as a rating system whose evidence must be compared to the existing gold standard (most often, human coders). We separate our framework into two stages: (a) a pre-validation stage for project- and data-dependent tasks such as prompt design, dataset organization, and model selection, and (b) a validation stage, which would encompass real-world datasets, performance metrics, agreement indices benchmarked to human reliability, consistency checks across replications, and construct-relevant external relations.

We illustrate this framework with a case study using Morbid Content (MOR) coding in the Rorschach task, a thematic code with a long-standing empirical literature (Mihura et al., 2013) and clear coding rules (Meyer et al., 2011). MOR is coded when a Rorschach response contains some damage (e.g., “a butterfly with a broken wing”) or dysphoria (e.g., “a sad dog”). MOR has a relatively low base rate (about 5% of responses) and clinical interpretive value (good meta-analytic support, Mihura et al., 2013), making it a useful case to create and test an LLM coder.

For the case study, we developed a workflow with two different LLM-based coders that operate in series (one after the other), each of them responsible for a different part of the Rorschach coding: (a) clean and prepare the data, and (b) classify MOR subcomponents. We named the LLM-based MOR Coder, or LLM Coder for brevity, as we only use one code.

We do not advocate for a single “best” model or prompt, nor claim that LLM coders should replace trained assessors. Rather, our goals are to: (a) present clear and reproducible steps for evaluating LLM coders for psychological tests that any team can implement, (b) provide a guide for researchers and clinicians working with LLMs in psychological testing, and (c) demonstrate the framework through a case study. In the long run, such standards will facilitate cumulative science by making results comparable across settings and enabling better collaboration among assessment, AI, and LLM researchers.

Method for the Case Study

Datasets

We used two separate datasets. The first one was used to develop and refine the prompts with error analysis and to evaluate the performance of various models under the same constraints. The second dataset was used to assess reliability and validity.

Norms Dataset: Used for the Pre-Validation Stage

This dataset comprises a subset of protocols from the current Rorschach Performance Assessment System (R-PAS; Meyer et al., 2011) norms. We selected only cases with complete English transcripts for both the response and clarification phases, N = 145 protocols. Those cases are used for protocols that were administered following Comprehensive System (CS) procedures (Exner, 2003). Protocols contained on average 23.31 responses (SD = 7.94, range = 14–53). Together, these protocols had 3380 responses. From these, we semi-randomly¹ selected 30 positive and 30 negative classifications of the targeted Rorschach code (i.e., MOR). Because these data contribute to the current R-PAS norms, their existing codes, assigned and reviewed by trained assessors through standard R-PAS procedures, were treated as the gold standard (criterion). We used this subset of 60 responses to evaluate the LLM coder against the assessor’s coding to improve prompts through error analysis, with the expectation that the LLM coder would agree as much as possible with the assessor’s decision.

Chicago Dataset: Used for the Validation Stage

This is an archival dataset previously reported by Meyer (1997, 1999) and colleagues (Meyer et al., 2000), which originates from a University of Chicago hospital-based psychological assessment service. This broader archive includes 661 clinical protocols. Of these, 443 completed the Rorschach, and 362 completed both the Rorschach and the MMPI-2. Cross-method correlations between Rorschach variables and self-reports are typically low (r ~.08; Mihura et al., 2013) when response style is not considered. However, Meyer (1997) showed that when response styles are similar across these tests and reflect spontaneous engagement with the task, correlations increase, such that the Rorschach Depression Index (DEPI) correlates strongly with MMPI-2 affect-distress scales (r range: .42–.66). Although the original articles (Meyer, 1997, 1999) did not examine the correlations of MOR with the MMPI-2, MOR is a component of DEPI and has shown stronger validity with external criteria than the DEPI overall (MOR r = .29 vs. DEPI r = .19; Mihura et al., 2013). Also, in a more recent re-analysis of these data, MOR demonstrated stronger correlations with the relevant MMPI-2 scales (Pimentel & Meyer, 2025).

From the complete dataset, Meyer (1997) identified 87 patients who exhibited a similar engagement style on both the Rorschach and MMPI-2. We obtained access to the complete responses for 84 of them, totaling 2,176 responses. In this style-matched subsample, we know in advance (Pimentel & Meyer, 2025) that assessor-assigned MOR² codes correlate strongly with five MMPI-2 scales: Scale 2 (r = .54), Scale 7 (r = .65), Depression Content Scale (DEP; r = .63), Anxiety Content Scale (ANX; r = .59), and Personality Psychopathology Five Negative Emotionality/Neuroticism (PSY-5-Neg; r = .62). Hence, we used these MMPI-2 scales as external validity criteria for our applied (“real-world”) analyses to demonstrate that when LLM-generated MOR scores have good agreement with assessor-coded MOR scores, the LLM coded scores show comparable associations with external criteria.

Note that we used this dataset because it provides a response-style-matched subgroup in which MOR has already shown meaningful relations with selected MMPI-2 scales. We do not intend to treat self-report as a general validator of the Rorschach or to argue that LLM coding should strengthen Rorschach-MMPI correspondence. Rather, given strong LLM-human agreement, we asked whether LLM-generated MOR scores reproduced the same established pattern of criterion relations observed with human-coded MOR in this dataset.

The Rorschach Task

The Rorschach task consists of presenting the respondent with 10 semi-ambiguous inkblots, one at a time, and asking, “What might this be?” Given that the instructions to the task do not specify what an ideal response would look like, the respondent is left to behave according to their preferences, coming up with their own solution. Later, the assessor applies many codes to classify the respondent’s behavior, language, and imagery. Respondents provide a few responses per card, and when summing across cards, they average about 24 responses per protocol (Meyer et al., 2011). Codes are assigned at the response level and then summed at the protocol level. Thus, the task offers two levels of data: (a) response level, which indicates whether a specific code applies to a particular response, and (b) protocol level, which includes the totals (and composites) calculated across all responses within a protocol.

Morbid Content (MOR)

We chose to use MOR as our case study variable, a thematic code with relatively straightforward decision rules. This code is applied in two circumstances: (a) damaged objects, e.g., “spoiled lettuce,” “a bug smashed on a windshield;” or (b) perceptions or attributions of distress, dysphoria, or depression given to response objects, e.g., “a very sad face,” “red eyes, he’s crying” (Meyer et al., 2011, pp. 140–141).

LLM Coder Workflow

We developed this workflow as part of a broader project to build multiple LLM coders. The development followed an iterative, trial-and-error process grounded in the R-PAS manual guidelines, extra guidelines for the second edition of R-PAS, and discussion among expert coders. For design, we used a few-shot prompt (Brown et al., 2020) where, besides instructions on how to classify a response, we also included a few positive and negative examples of expected input and what the output should look like. In addition, we implemented a chain-of-thought design (Wei et al., 2022), adding instructions to provide a rationale before making a final classification decision. Our prompt specifies the agent’s name, goal, expected inputs and outputs, and a step-by-step decision process. For guidance on creating prompts, see Demszky et al. (2023). Our workflow uses two independent LLM agents that process the input in sequence. Below is an explanation of each. All agents had the temperature³ set to the lowest option:

Standardization Agent. Parses Response and Clarification Phases and outputs a standardized JSON record. It standardizes task-specific notations (e.g., when the assessor reads the responses back), notation about the respondent’s behavior, and, where possible, separates the assessor from the respondent’s transcription.

Classification Agent (MOR). Receives the standardized JSON file from the previous agent, evaluates each MOR subcomponent and criterion, provides a rationale for why or why not a criterion is applied, and records a True or False decision for each. To support transparency while protecting proprietary materials, we provide a high-level template describing the prompt architecture, accompanied by brief illustrative examples (Supplemental Material 1).

Data Preparation and API Processing

We organized responses in a long format with columns for card number, response number, Response Phase, and Clarification Phase. Each response (i.e., row) was turned into a JSON object and sent sequentially to the LLM coder workflow: (a) standardization and (b) MOR classification. We deployed all APIs using Copilot Builder (https://copilotbuilder.ai/).

Each response submission to the API starts a new session, so the LLM coder has no memory of previous responses. Given that sessions are independent, potential errors in the rationale or LLM hallucinations⁴ in one response do not impact decisions made in other sessions. Essentially, each session works like a Rorschach expert who processes one response, then passes it to another equally trained expert for the next.

Output Consolidation and Analysis Dataset

All API calls and data processing were done using R⁵ (R Core Team, 2025). We received the final output in JSON, then we converted it to tabular data, separating LLM-generated MOR decisions from the LLM coder’s rationale. The final file includes card number, response number, Response and Clarification Phase, JSON input and output, and the LLM-generated MOR code for each response. We then added the original assessor-assigned MOR code, organizing two data files for analysis: one at the response level and the other at the protocol level.

Metrics

We treated the assessor’s coding as the gold standard (i.e., “true” cases) and considered the LLM-generated outputs as the predicted cases. We compared them with confusion matrices in which columns are the true classes and rows are the LLM coder’s predictions by calculating True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). From this matrix, we computed: (a) Accuracy: Proportion correct across all cases (N), (TP+TN)/N, answers the question “How often is the LLM coder correct overall?”; (b) Cohen’s kappa: Agreement between LLM coder and assessor beyond chance that answers the question “How much do they agree once chance agreement is removed?”; (c) Positive Predictive Value (PPV): TP/(TP+FP), that answers the question “When the LLM coder says ‘positive,’ how often is it truly positive?”; (d) Negative Predictive Value (NPV): TN/(TN+FN) that answers the question “When the LLM coder says ‘negative,’ how often is it truly negative?”; (e) Sensitivity (Recall or True Positive Rate): TP/(TP+FN) that answers the question “Of all actual positives, how many did the LLM coder find?”; (f) Specificity (True Negative Rate): TN/(TN+FP) that answers “Of all actual negatives, how many did the LLM coder correctly reject?”

Two-Stage Evaluation

Stage 1: Pre-Validation Screening and Prompt Development (Balanced Samples)

To test the performance of models and refine prompts based on error analysis, we artificially balanced the data to approximate a 50/50 split between present and absent codes. Then, we compared across different LLMs and prompts. This artificial balance might not be necessary if researchers are using data that are common occurrences. It is helpful, however, for low or high base rates. This stage is flexible and varies depending on the specific project. It can take many forms depending on the test and variable and may include iterative prompt engineering, rule refinement, or model comparisons. Its purpose is to prepare the final prompt for formal evaluation by stabilizing its performance and selecting the most suitable model.⁶

Error Analysis

To refine the prompt, we used the Norms dataset and interactively updated the prompts by conducting error analysis, a structured review of cases in which an LLM output diverges from a reference standard to identify systematic failure and guide revisions. These revisions involved clarifying decision rules, adding counterexamples, strengthening exclusions, and adjusting the standardization of inputs and outputs.

Stage 2: Validation on Natural Data (Low Base Rate)

The validation phase is where we apply our proposed framework. We encourage both psychological test developers and automated-scoring developers to treat this stage as a standard for demonstrating that an automated system works reliably and preserves validity.

For our case study (MOR), we emphasize kappa (chance-corrected agreement), PPV (credibility of positive calls), and Sensitivity (ability to detect positives). These metrics remain most informative when positives are rare. Given the relatively low base rate of MOR, we prioritize not missing positives (i.e., adequate Sensitivity) while ensuring that predicted positives are trustworthy (i.e., adequate PPV), accepting that this balance may allow some FP. For protocol level reliability, we used the Intraclass Coefficient Class (ICC) and followed Cicchetti’s (1994) general interpretation: <.40 poor, .40–.59 fair, .60–.74 good, and ≥.75 excellent.

Framework – Pre-Validation

In the next section, we outline each pre-validation step with (a) the rationale, (b) how to implement it, and (c) an illustration using the MOR variable.

Step A (Pre-Validation): Building the Dataset

Rationale

Many variables in psychological tests occur infrequently, leading researchers to often deal with imbalanced datasets in which target behaviors and traits have low frequency. When evaluating an automated system on a dataset that mirrors a natural imbalance, the system can appear accurate simply by predicting the most common outcome (e.g., “absent”). For instance, with a 5% positive base rate, a trivial LLM coder that always outputs “negative” achieves 95% accuracy but misses all positive cases. NPV and Specificity can also seem inflated at low base rates because most cases are negative. Hence, one problem with imbalanced data is that several performance metrics become useless (He & Garcia, 2009). This leads to inflated accuracy scores that hide weaknesses in detecting the less frequent, yet often clinically significant, cases. A way to overcome this problem and provide informative development data is to use Random Oversampling and Undersampling (He & Garcia, 2009). In practice, this is done by selecting a similar number of cases from each class, in our case, positive and negative. A dataset with roughly equal positive and negative cases allows us to detect weaknesses and make fair comparisons across models, and ensure that metrics like Overall Accuracy and Sensitivity are meaningful.

How to Do It

Identify the variable to be evaluated (e.g., a scoring code or classification category).

From the gold-standard dataset (human-coded), select a fixed number of positive cases.

Randomly select an equal number of negative cases.

Use this balanced dataset only for early-stage development and model selection. It does not replace testing with real base rates.

Example With MOR

We processed 60 responses (30 positive and 30 negative) across nine LLMs. During this pre-validation pass, three models did not meet our prompt, formatting, or reliability requirements and were excluded from further testing: (a) Meta-Llama-3.1-8B: Produced inadequate and underspecified outputs; prompts required substantially more constraints and detail to approach the requested format; (b) Phi-4: Returned outputs inconsistent with the prompt specification (e.g., generated free text instead of JSON-only); (c) DeepSeek-R1: Markedly slower runtime and verbose outputs that did not follow the JSON-only requirement. It generated many more null results (rather than clear true and false decisions).

These issues reflected our specific settings and prompts. These models may behave differently with alternative setups and prompts. For this project, they were not reliable enough to advance. Other models, multiple versions of OpenAI GPTs, and Google Gemini, returned outputs close to the requested specification and were retained for the next step.

Step B (Pre-Validation) – Comparing Multiple LLMs and Prompt Development

Rationale

Given that OpenAI’s and Google’s models are built using different data, different numbers of parameters, and different architectural designs, they differ in how they follow instructions, handle ambiguous cases, or maintain output consistency. Hence, an early comparison across multiple candidate models ensures that a researcher can focus their refinement efforts on a model that is already well-suited to the task and the type of prompt they are working on, rather than trying to compensate for poor baseline performance later.

How to Do It

Choose several candidate models that could be used for the target LLM coder.

Send the responses separately to each model, with the same prompt and temperature.

Compare the LLM-generated outputs to the human-coded gold standard using agreement metrics such as Accuracy, Kappa, Sensitivity, Specificity, PPV, and NPV.

Decide which model to retain for further development and testing.

Conduct error analysis to refine the final prompt.

Example With MOR

After excluding three models during pre-validation, we compared six remaining models on the balanced set (see Table 1 and Figure 1). Each version of the LLM coder used a different model (but the same prompt) and was evaluated against the assessor’s coding. Among the tested models, GPT-4.1 achieved the highest chance-corrected agreement (kappa = .72), and its PPV = 0.92 and Specificity = 0.93 were also among the highest, so positive MOR calls closely matched assessor decisions and non-MOR responses were rarely misclassified as MOR. Gemini 2.0 Flash and GPT-4.1 Mini followed with similar performance across metrics, and GPT-4o showed a solid balance between Sensitivity and Specificity. By contrast, GPT-4.1 Nano showed the weakest overall performance (kappa = .42), driven primarily by lower Sensitivity = 0.50, meaning it missed more MOR-positive cases than other models.

Table 1.

Pre-Validation Performance Metrics for Each LLM Version.

LLM model	N	Accuracy	Kappa	Sensitivity	Specificity	PPV	NPV
GPT 4.1	57	.86	.72	.79	.93	.92	.82
GPT 4.1-Nano	43	.70	.42	.50	.95	.92	.60
GPT 4.1-mini	56	.84	.68	.82	.86	.85	.83
GPT 4o	57	.82	.65	.71	.93	.91	.77
Gemini 2.0 Flash	60	.85	.70	.80	.90	.89	.82
Gemini Flash	60	.83	.67	.80	.87	.86	.81

Note. N varies across models because some responses were not processed when the model provider’s content filtering or safety policies blocked the request, resulting in missing outputs for those items. NPV = Negative Predictive Value; PPV = Positive Predictive Value.

Figure 1.

Performance metrics by LLM for morbid contents.

Although GPT-4.1 was optimal, we prioritized performance and scalability for other Rorschach codes we are working on. Considering the relative price, with 4.1 Mini being 80% cheaper than 4.1, and practical performance across tasks, we selected GPT-4.1 Mini for the next stage. It delivered acceptable, stable results with our current prompt while offering a more favorable cost-performance profile for use with other coding agents under development.

Error Analysis and Prompt Refinement

We manually inspected disagreements between GPT-4.1 Mini and the assessors’ scores across several runs to identify recurring errors and revise instructions. Revisions focused on (a) clarifying definitional boundaries, (b) adding targeted inclusion and exclusion rules for recurrent edge cases, and (c) creating a standard output for atypical inputs. For example, early iterations over-classified MOR for fire or explosions alone, even when no object was described as damaged. The prompt was revised to require an explicitly damaged or destroyed object when “fire,” “explosion,” or similar content is mentioned. Additional refinements included: clarifying that intact skeletal percepts are not coded as Damaged Objects unless the percept indicates remains or damage; expanding examples for under-identified morbid representations (e.g., tombstones, cemeteries, plague masks) while adding negative thresholds for superficially related but non-morbid items (e.g., COVID-19 masks); constraining blood to cases implying injury (rather than the word “blood” alone); and distinguishing affect attributed to the object from the respondent’s reaction.

We also enforced the output structure with no follow-up questions and include a fallback template when essential inputs are empty. Finally, because the examples were sometimes applied too literally, we reframed them as illustrative examples rather than exhaustive lists and created a “minimum information needed” for each subcategory. This development continued until remaining discrepancies primarily involved ambiguous or threshold cases where reasonable human disagreement was expected (all FP and FN were reviewed by both authors). The final prompt was then frozen and evaluated in the next stage in an independent Chicago dataset.

Framework – Validation Process

After data preparation and model selection, researchers will formally evaluate the psychometric performance of the LLM coder. The goal is to test whether the LLM coder is reliable, both (a) stable across repeated runs (intrarater reliability) and (b) consistent with human coding (interrater reliability) and valid, meaning its scores relate to external criteria in theoretically coherent ways.

Agreement between the LLM coder and assessors can be interpreted in two ways: (a) as interrater reliability, because both coders apply the same scoring rules to the same responses, or (b) as criterion-related validity, because the LLM score is compared with a reference standard (Grove et al., 1981). In this framework, we emphasize the former because human codes are not error-free, so the assessor’s code is an imperfect gold standard, although aggregated reference scoring can reduce idiosyncratic error (McGraw & Wong, 1996; Shrout & Fleiss, 1979). Conceptually, LLM-assessor agreement indicates whether the LLM can reproduce human scoring, whereas validity against independent external criteria tests whether LLM-generated scores preserve the expected relations between the construct and external measures. If the LLM-assessor agreement produced identical scores, then validity analyses using external criteria would yield the same results for both coders. In this case, the validity evidence concerns the score and the construct it represents, rather than differences between coders. This also means that, under the same conditions, the LLM score could be a substitute for a human score.

Step 1: Intrarater and Interrater Reliability

Rationale

Reliability testing addresses a fundamental psychometric question: “If two raters score the same material, do they agree?” For LLM coders, this first means verifying that they are consistent across different sessions. Then, checking whether the LLM-generated scores align with those of trained assessors, both for individual responses and for aggregated protocol scores. Without sufficient reliability, specifically across sessions, validity is compromised because the coding process is unstable (Furr, 2022; Schmidt & Hunter, 2015).

Cohen’s kappa and ICC are typical metrics used when assessing reliability in performance-based tests such as the Rorschach task (McGrath et al., 2005; Meyer et al., 2011). Kappa values, although standardized, vary in magnitude depending on the base rates (Grove et al., 1981). Less frequent codes, by definition, have fewer positive cases, so the same amount of random error has a larger impact on reliability estimates than it would for more frequent codes (Viglione et al., 2012). When looking at dimensional variables, in the case of Rorschach protocol level information, Viglione and Meyer (2008) explain that ICC is preferable, while kappa would be recommended for nominal or categorical variables, as we see in the response level data.

Benchmarks and Thresholds

Although we can simply calculate kappa and ICCs and report the typical interpretation levels, we can go one step further by contextualizing the results and benchmarking them. The challenge is creating such benchmarks as they depend “on the availability of keystone datasets and requires a consensus on how to define and operationalize psychological constructs” (Demszky et al., 2023, p. 698). Until such field-wide resources and consensus emerge, we recommend pragmatic human-centered benchmarks. For example, for reliability, the LLM-LLM agreement would need to meet or exceed the assessor-assessor agreement for the same metric and level. Also, to keep validity at the same human coding level, we would like to see LLM-assessor agreement comparable to assessor-assessor agreement. On the contrary, if reliability is low, we would need to assess validity and require the LLM’s coder-criterion effect sizes to match or approximate the human-criterion effect sizes reported under comparable conditions. Because benchmarks are often construct- and dataset-specific, we illustrate how to select and interpret pragmatic reliability benchmarks in the MOR example, rather than treating any single numeric threshold as universally applicable.

How to Do It

Response level reliability: Compare the LLM-based coding to a gold-standard code (i.e., human coding) on each individual observation (e.g., test item, behavioral response).

Protocol level reliability: Aggregate scores to the protocol or person level. Aggregation can reduce the impact of isolated disagreements and test whether the LLM coders reproduce similar interpretation scores. Because interpretations are made at this level, protocol-level reliability is ultimately the most consequential. Enhancing response level reliability typically boosts overall protocol level agreement (Meyer et al., 2002).

For both response level and protocol level reliability, obtain human–human agreement for the same variable, either from the same dataset, from comparable datasets, or a meta-analytic summary to benchmark reliability against typical human performance.

Metrics to use:

a. For response level data: Because each response is a dichotomous decision, it could be treated as nominal or categorical data, and kappa would be used to evaluate reliability. Also consider transforming multilevel codes into separate prompts, each coding an independent level as a dichotomy. For less frequent variables, especially rare ones (<5%), any disagreement has a greater impact. Although we suggest evaluating all metrics, focus on kappa, PPV, and Sensitivity.

b. For protocol level data: Here we are dealing with the sum of positive cases of a score; now the scores could vary from 0 to as many responses as are in a protocol (e.g., 0–24). Data are now dimensional; hence, ICC would be more appropriate for assessing agreement on continuous scores. Use an absolute-agreement form, a two-way random effects model for a single measure (McGraw & Wong, 1996).

Example with MOR

Response Level

The LLM coder was evaluated against assessor coding using the Chicago dataset with natural base rates for MOR. This step assessed both agreement and the LLM coder’s ability to identify MOR cases in realistic conditions correctly. We ran all responses (N = 2,176) three times (T1, T2, and T3) in independent sessions to assess the consistency of the model responses. After turning off an initial provider-side moderation issue (which had blocked several dozen responses as inappropriate content), only a handful remained unprocessed. In two runs, five responses were not returned (analyzable N = 2,171; 0.23% missing). In the third run, four were not returned (analyzable N = 2,172; 0.18% missing) because the model correctly processed one previously rejected “abstract” response as a real response. Of the items never processed, three contained extreme content (sexuality, harassment, prejudice, or violence) or were interpreted as having some personally identifying information that continued to trigger provider content filtering.⁷ The remaining item was correctly not considered a response by the LLM. In this instance, the response was “Colors” with a clarification that simply mentioned several colors, which is not a genuine response, even though the original assessor coded it. To avoid post hoc bias, we did not edit the dataset or the prompt to process the non-coded responses (i.e., we did not conduct error analysis on this dataset). We treat these as genuine LLM-assessor disagreements and retain these responses as missing on the LLM side.

Table 2 summarizes the response level results. Across runs (i.e., T1, T2, and T3), performance of the agent relative to the assessor was virtually identical: kappa = .72–.74 (good chance-corrected agreement), Sensitivity = .79–.81, PPV = .69–.73, Specificity = .96–.97, NPV = ~.98, and Accuracy = ~.95. In practical terms, the agent detected roughly 80% of assessor-identified MOR responses (Sensitivity), and 70% of its positive calls matched assessor codes (PPV). High Specificity and NPV indicate few FP and more reliable negatives. Accuracy remained around 0.95 across runs, but should be interpreted with caution given the low base rate. The near-identical metrics across T1–T3 also support the LLM coder consistency under the same prompt and model settings. For illustration, Figure 2 presents the confusion matrices for T1–T3. Consistent with the metrics, the agent was highly effective at identifying non-MOR responses (high Specificity and NPV) and performed well at detecting MOR, with some FP.

Table 2.

Performance Metrics for the AI-Based Agent for Coding MOR Across Responses (N = 2,176).

LLM Model	N	Accuracy	Kappa	Sensitivity	Specificity	PPV	NPV
MOR T1	2,171	.95	.72	.79	.97	.71	.98
MOR T2	2,171	.95	.71	.79	.96	.69	.98
MOR T3	2,172	.95	.74	.81	.97	.73	.98

Note. NPV = Negative Predictive Value; PPV = Positive Predictive Value. T1–T3 = LLM coder Time 1–3.

Figure 2.

Confusion matrix for morbid content between agent (rows) and human codes (columns) across time 1 (T1), time 2 (T2), and time 3 (T3).

Overall, these findings highlight a good to excellent agreement with human coding, particularly in detecting TN and minimizing FN. The somewhat lower PPV shows that some LLM MOR-positive calls were not coded as MOR by the assessor. Whether these represent overcoding by the LLM coder or borderline and ambiguous cases requires further analysis.

Protocol Level

Because interpretations are ultimately made at the protocol level, we aggregated MOR totals per protocol and evaluated agreement across these 84 protocols. Table 3 shows that protocols had, on average, 25.90 responses (SD = 11.82). Assessor-coded MOR averaged 2.40 (SD = 2.96) per protocol. The LLM coder runs were overall similar to the assessor’s score, although slightly higher: T1 M = 2.63 (SD = 3.13), T2 M = 2.71 (SD = 3.25), and T3 M = 2.65 (SD = 3.21). Given MOR codes were relatively skewed (i.e., around or exceeding|1.8|; Table 3), we square-root-transformed these variables, reducing skew values to below|0.41|, which makes the scores more normally distributed (Table 3). Figure 3 illustrates these transformed distributions with violins and boxplots and no statistically significant differences across groups; the omnibus test was F(2.07, 171.97) = 2.78, p = .06. Also, post hoc Holm-adjusted pairwise effects were small (Hedges’ g: −0.25 to 0.08) and not statistically significant. Practically, if handed a random protocol, the MOR count alone would not reveal whether it was coded by an assessor or by the LLM coder. However, to a small extent, the LLM coder tended to overcode MOR compared to human assessors, assigning 10.2 codes per 100 responses versus 9.3 codes per 100 responses.

Table 3.

Descriptive Statistics of Morbid Coders by Protocol Level (N = 84).

Variables	M	SD	Med	Min	Max	Skew	M2	SD2	Min2	Max2	Skew2
R	25.90	11.82	22	14	72	1.68
MOR Assessor	2.40	2.96	1	0	14	1.70	1.21	.98	0	3.74	0.41
MOR T1	2.63	3.13	1	0	15	1.75	1.29	.99	0	3.87	0.36
MOR T2	2.71	3.25	2	0	18	1.98	1.31	1.00	0	4.24	0.36
MOR T3	2.65	3.21	1	0	15	1.76	1.29	1.00	0	3.87	0.40

Note. MOR Assessor: Morbid Content scored independently by humans; MOR T1–T3: scored at times 1–3 by LLM coder; R: Number of responses in a Rorschach protocol; the M, SD, Min, Max, and Skew columns present raw scores, while columns with headings ending in “2” contain the values used in the ANOVA and criterion correlation analyses, which were square root transformed for all variables except R, which was not included in any analysis.

Figure 3.

Comparison of MOR across assessor coder and LLM-based agent coder.

Protocol-level agreements were excellent using untransformed raw scores (Table 4). Assessor-LLM ICCs were 0.95, 0.94, and 0.95 for T1–T3, all p < .001. LLM-LLM agreements were higher: 0.98, 0.99, and 0.97, all ps < .001. Notably, the 95% confidence intervals for Assessor-LLM and LLM-LLM barely touch: the strongest Assessor-LLM upper bound is 0.97, which barely exceeds the lowest LLM-LLM lower bound (0.96). This visual separation indicates that the LLM reliably agrees with itself more often than it agrees with a human (Cumming & Finch, 2005).

Table 4.

ICC Summary Protocol Level Agreement (N = 84).

Comparison				95% CI
Comparison	ICC	F (83)	p	LB	UB
Assessor and LLM agreement
Assessor vs LLM T1	.95	41.47	<.001	.92	.97
Assessor vs LLM T2	.94	37.20	<.001	.91	.96
Assessor vs LLM T3	.95	37.78	<.001	.92	.97
LLM and LLM Agreement
LLM T1 vs LLM T2	.98	97.25	<.001	.97	.99
LLM T1 vs LLM T3	.99	140.05	<.001	.98	.99
LLM T2 vs LLM T3	.97	73.59	<.001	.96	.98

Note. LB = Lower Bound; UB = Upper Bound.

Benchmark Comparisons

At the response level, the LLM coder shows good chance-corrected agreement with the assessor (kappa ≈ .72–.74). To contextualize these findings, we would like to see what an expected kappa level is in practice among assessors. Meyer et al. (2002) examined the interrater reliability for scoring the Rorschach in several datasets and reported a kappa of .90 for experienced coders, a kappa of .81 for students, and .72 for a dataset with 30% random error. Thus, although the LLM coder kappa is good, it is slightly below the student kappa and far below that of experienced raters. It is closest to the 30% error condition. This shows room for response level refinement while still reflecting good agreement under natural base rates. In other words, this is a solid first milestone rather than an endpoint.

By contrast, at the protocol level, where interpretation occurs, we can see the big picture. Assessor-LLM ICCs of 0.94–0.95 match expert-assessor results reported by Meyer et al. (protocol level ICC = 0.95, 2002) and are at or above typical assessor-assessor agreement reported in Schneider et al. (2022), who reviewed ICC for R-PAS variables across four datasets. The ICCs across four samples for MOR were 0.93, 0.94, 0.78, 0.93 (unweighted mean 0.90, weighted mean 0.87). This pattern of higher protocol level reliability than response level reliability is expected and has already been explained and described by Meyer et al. (2002). However, the specific findings going from good reliability at the response level and excellent reliability at the protocol level in this case, may also imply that, although the LLM coder does not fully capture response level reliability at expert standards, it nevertheless yields protocol totals that are comparable to expert coders, likely because some LLM-flagged, assessor-negative responses (e.g., borderline cases) aggregate to a stable overall MOR count.

Finally, the observed LLM-LLM consistency (0.97–0.99) is numerically higher than assessor-assessor reliability (e.g., Meyer et al., 2002; Schneider et al., 2022). In practice, the protocol level agreement (the level used for clinical and research interpretation) meets established human benchmarks even as response level refinement remains open to improvement. Together, these results show that the LLM coder closely reproduces assessor totals with minimal bias and near-perfect stability across independent runs.

Step 3: Validity Against Independent Criterion

Rationale

In the previous section, we checked the consistency of the LLM coder across independent runs and the agreement with human coders. To demonstrate that the reliability with an already valid coder (assessor) generalizes to its validity, we also show that the LLM coder is measuring the same construct or behavior. Validity testing asks whether LLM-generated scores relate to external variables in the same pattern and with the same magnitude as human-coded scores. This typically means reproducing established associations with a relevant criterion. In a typical validity study, behavioral and clinically consequential outputs provide the strongest evidence and can be considered the “gold standard” when available. However, when the goal is to test whether an LLM coder preserves the meaning of an established human-coded score, it could be more practical to begin by replicating prior human-coded findings using the same criterion measures that have already demonstrated validity. For this part, we recommend starting by choosing a criterion with prior theoretical support and demonstrated validity. For instance, one used in earlier studies or available in an existing dataset. Then, replicate those associations with the LLM-generated scores independent of human scores. Pre-specify the expected direction and magnitude of effects and compare effect sizes with past findings.

How to Do It

Identify external measures that have well-documented associations with the variable being coded (e.g., prior studies or datasets).

Note predicted direction and magnitude.

Calculate associations between the LLM-generated scores and these external measures and compare them with correlations from human-coded scores.

Focus on whether the LLM preserves the direction, magnitude, and statistical significance of these associations.

Example with MOR

To evaluate validity against external criteria, we correlated protocol-level √MOR totals with MMPI indices that should relate to dysphoria and negative affect (Scale 2, Scale 7, DEP, ANX, PSY-5-Neg; Table 5 and Figure 4) in patients with matching response styles on both tests. All correlations were statistically significant (n = 84, df = 82, ps < .001). The pattern was consistent across LLM coders: associations ranged from .59 to .71, with Scale 7 the strongest and Scale 2 the lowest. Relative to human coding, the LLM coders produced numerically higher correlations on every scale (e.g., Assessor vs. LLM ranges: Scale 2 = .54 vs .59–.63; Scale 7 = .65 vs .70–.71; DEP = .62 vs .68–.69; ANX = .59 vs .65–.66; PSY-5-Neg = .62 vs .65–.68). Average correlations were .61 for the assessor and .66 across the three LLM coder T1–T3, indicating small, consistent gains while preserving the same substantive profile. The bar plot (Figure 4) illustrates these parallel patterns across coders. Hence, the LLM coder maintains the expected external relations and does so reproducibly across independent runs.

Table 5.

Pearson’s Correlations Between MOR and the MMPI Scale by Coder (n = 84, df = 82, ps < .001).

MMPI	Assessor	LLM T1	LLM T2	LLM T3
Scale 2	.54	.63	.60	.59
Scale 7	.65	.70	.71	.71
DEP	.62	.68	.68	.69
ANX	.59	.65	.66	.65
PSY-5-Neg	.62	.65	.67	.68

Note. All MOR variables (Assessor, LLM T1–T3) were square root transformed prior to analysis. Assessor: Morbid Content scored independently by humans; LLM T1–T3: Morbid Content scored at times 1–3 by LLM agents. DEP = Depression Content Scale; ANX = Anxiety Content Scale; PSY-5-Neg = Personality Psychopathology Five Negative Emotionality/Neuroticism Scale.

Figure 4.

Correlation sizes across LLM-based coders and human coder (N = 84, df = 82, all ps <.001).

Discussion

We present a practical, test-agnostic framework that adapts established psychometric practice to LLM-based scoring and aligns with Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis +AI (TRIPOD+AI; Collins et al., 2024) and with Guidelines for Reporting Reliability and Agreement Studies (GRRAS; Kottner et al., 2011). Our contribution is a step-by-step process with two separate stages: a flexible, project-specific pre-validation phase and a standardized validation phase. Concretely, we recommend (a) balanced screening samples to avoid inflated accuracy, (b) response level agreement emphasizing kappa, PPV, and Sensitivity under low base rate conditions, (c) protocol level agreement via ICC, and (d) validity against independent criteria by replicating established relations, allowing direct comparison of LLM coders to human coders, particularly when reliability between the LLM coder and human coders is modest or weak.

For pre-validation, we emphasize pragmatic decisions whenever possible, such as comparing multiple models on a balanced dataset, iterating on prompts, and addressing formatting and classification issues. Researchers should aim to select the best-performing model for their intended use. “Best” should be defined in advance, as different metrics can favor different tasks. For example, for screening tasks, the priority is often Sensitivity, because missing true cases can be costly (e.g., screening for suicide risk, early psychosis, neurocognitive impairment). In contrast, when FP carries substantial consequences, priorities often shift toward PPV and Specificity (e.g., forensic evaluations of competency, criminal responsibility; classifications that could trigger restrictive interventions; high-stakes occupational clearance). Once priorities are specified, a pragmatic choice is to retain the model that performs best on the primary metric while meeting the predefined minimums on the others. On the contrary, the model with the highest performance may not be the best practical choice, because model selection also depends on external constraints such as cost, privacy, data handling, and provider governance. Researchers should weigh in those factors. Although a researcher may opt for a model that is not the “best” performing because it may be too expensive or does not meet the necessary ethical requirements, the subsequent validation steps remain fixed and are designed to test whether LLM-generated scores preserve the meaning of human-generated scores.

Reliability: Expectations and Evidence

Perfect agreement (e.g., ICC = 1.00 or r = 1.00) is not the standard that humans achieve in psychological or medical literature (Meyer, 2004). Ambiguity in responses, threshold cases, and non-exhaustive guidelines produces genuine assessor disagreements, so an LLM coder that matches high human-level agreement is already performing at a demanding benchmark. In our case study, the LLM coder achieved good response level agreement (kappa = .71–.74) and excellent protocol level agreement (ICCs = .94–.95) with human coders. The LLM coder demonstrated near-perfect consistency across independent runs (LLM-LLM ICC T1–T3 = .97–.99), indicating, to some extent, that the LLM coder agrees with itself more often than with a single human rater. This is not a surprising finding for a model with unchanged prompts and low temperature. Literature benchmarks for MOR (expert ICC = .90, Meyer et al., 2002; weighted mean ICC ≈ .87; Schneider et al., 2022) place the assessor-assessor agreement 8 to 11 points below the LLM-LLM consistency. Also, the LLM-assessor agreement was directly comparable to the assessor-assessor agreement at the protocol level.

A related consideration is the choice of the human reference standard. In our example, we used a single-rater score to parallel prior work, but developers can also use aggregate reference scoring, such as majority vote or adjudicated consensus at the response level and averaged totals at the protocol level, which typically reduces idiosyncratic rater error and increases reliability (McGraw & Wong, 1996; Shrout & Fleiss, 1979). However, aggregate criteria are not automatically “truer” because they can mask poor single-rater performance (Shrout & Fleiss, 1979), obscure genuine boundary uncertainty (AERA et al., 2014), and introduce shared or group-level biases. Ultimately, the most appropriate reference standard depends on the intended use and research question, including whether conclusions will be applied in settings where scoring is typically done by a single coder or by pooled ratings.

Validity: Preserving Construct Meaning

Evaluating validity against an external criterion is a logical extension of high interrater reliability, given that the human rater already demonstrated good validity. If two raters produce highly reliable, interchangeable scores, then each set of scores should correlate similarly with relevant external measures. In our study, we make this point concrete by first showing compelling LLM-assessor reliability and then showing parallel validity with an independent criterion. Practically, when reliability is high with a human coder, additional validity analysis will replicate the validity for humans and the LLM; when reliability is modest or low, criterion analyses become decisive, because the question shifts from “is the LLM as good as humans?” to “which coder is more valid?” In that circumstance, researchers should compare LLM and human coders with theory-relevant external criteria to determine which better preserves the construct.

In our example, we used MMPI indicators of dysphoria and negative affect as independent criteria, in a dataset where we knew those correlations were already high with human coders (see Meyer, 1997; Pimentel & Meyer, 2025). The LLM coder reproduced the expected pattern and showed slightly larger correlations while retaining the same rank ordering across scales. This pattern supports that the LLM-generated codes preserve the construct assessed by MOR rather than introducing noise, and it supports the argument that strong agreement between the LLM coder and the human coder preserves the validity of those scores.

Implications for Practice

We are not anticipating having a fully automated Rorschach coder yet. The Rorschach is time-intensive to learn, administer, and score, so a practical near-term goal is to develop a human-in-the-loop coding assistant that accelerates research and supports clinical workflows. This use of LLMs as assistants is also advocated by Guo et al. (2024), who concluded that professional oversight of LLMs could enhance health care efficiency. In the context of the Rorschach task, if multiple variables can be pre-coded reliably, studies could include larger samples and deploy performance-based tests more routinely. This matters because studies using performance-based measures typically have far fewer participants than studies using self-reports. Our case study demonstrates good to excellent reliability and validity for a single code; many other LLM coders would be required to build a complete LLM-based Rorschach coder.

For MOR, the slight overcoding from the LLM coder (raw score means ≈ 2.63–2.71 vs. assessor 2.40) is operationally useful because assessors can triage by prioritizing positive calls. Instead of scanning ~20 to 30 responses per protocol, an assessor can review ~2 to 4 LLM-flagged positive responses to confirm or reject them. This is preferable to an undercoding system that would force clinicians to hunt for missed positives across many negatives.

This triage workflow also clarifies how performance metrics should be prioritized. Although sensitivity and specificity tradeoffs are well described in the assessment and diagnostic literature (Hajian-Tilaki, 2013; Meehl & Rosen, 1955; Swets, 1988), LLM development enables us to go one step further through targeted error analysis. When some FPs are acceptable, the practical priority is to minimize FN, so clinically relevant content is rarely missed, and then to reduce FP to lower the assessor’s review burden. In practice, this means using error analysis to identify recurring sources of missed cases or false alarms, and updating prompts, with the expectation that refinement will gradually narrow the gap with human coding, likely surpassing it, given the high LLM-LLM stability. The high LLM-LLM stability also benefits longitudinal research as the model applies its decision rules consistently across time, protocols, and administrations, reducing rater-related noise and increasing the ability to detect true change. In those studies, the specific values for a score are not the focus, but rather the change over time. In those circumstances, LLM coding could be highly valuable to speed up processing time.

Future researchers might also consider a multiple-coder workflow, which, conceptually, is similar to ensemble models (Breiman, 2001; Mohammed & Kora, 2023) or crowdsourcing (Shapley & Grofman, 1984; Snow et al., 2008) in which multiple independent processes are combined into a single decision rule. Although our three runs were used to evaluate output stability rather than to aggregate decisions, one could combine them using a majority vote or a more inclusive any-positive rule in which a code is assigned if at least one session endorses it. These aggregation rules would shift error tradeoffs, with the any-positive rule tending to reduce FN at the cost of more FP, and majority vote, the opposite. If an ensemble rule is used, it should be evaluated as a new coder, including repeating the ensemble procedure across multiple runs to test stability. This approach is slower and more costly, but it may be valuable depending on the context (when FP and FN priorities are clear). However, at low temperatures, we anticipate that ensemble workflows will not deviate much from single runs, as outputs will be more stable. Increasing the temperature would, in principle, result in larger differences across runs.

Disagreements as Signals and Future Developments

Beyond assisting with scoring established variables, the use of an LLM coder can also support test development and refinement. During the development of guidelines for clinicians, an LLM coder can serve as a scalable stress test of scoring rules by processing large datasets, surfacing recurring ambiguities and boundary cases where rules or examples are underspecified. Error analysis can then guide revisions to definitions, decision thresholds, and illustrative examples. Also, if an LLM coder shows greater validity against a specific external criterion than humans, disagreements that appear to be errors might be systematic signals rather than noise. These signals are response patterns that humans miss. In such cases, error analysis can help identify candidate additions to manuals and guidelines that would improve human scoring. This type of analysis must be conducted prudently, because apparent gains in validity could also reflect other issues (e.g., overfitting, criterion contamination, or construct drift). Disagreements should be evaluated by experts, documented, and tested in independent validation samples before being incorporated into scoring guidance.

In our example, the LLM-criterion correlations were slightly higher than assessor-criterion correlations while preserving the same profile. As such, some LLM-flagged responses that disagree with the assessor’s coding may be relevant for improving coding relative to this specific criterion. However, if we change the prompts based on mistakes we see in the validation dataset, we risk making a prompt that works well only for that dataset. Hence, we did not review the LLM-assessor disagreements in the Chicago validation dataset (i.e., we did not conduct error analysis). We kept this dataset untouched so it can remain a clean test of performance and be reused later to check stability over time. Instead, we used a separate dataset (i.e., Norms) to refine the prompts with error analysis. This separation helps reduce overfitting and makes it more likely that any improvements will generalize to new data.

Our results show a small, consistent numerical edge for LLM-criterion correlations relative to assessor-criterion correlations on MMPI-2 negative affect markers. If one were to treat these MMPI-2 scales as the defining criterion for MOR, the implication would be straightforward: MOR guidelines should be updated to code more like the LLM coder because the LLM coder is both more consistent (LLM-LLM ICC ≈ 0.98) and slightly more predictive of that criterion than individual human coders. However, MMPI negative affect is not the construct definition of MOR; it is only a plausible external criterion with established, historical utility (e.g., Meyer, 1997) that we deliberately used to check whether the LLM coder preserves known relations. Consequently, stronger LLM-MMPI correlations are evidence of preserved validity with respect to that criterion, not a mandate to revise MOR guidelines only because of that. An alternative path is to treat LLM-assessor disagreements as potential room for revisiting and updating guidelines rather than a mandate for guideline changes.

Although responding to structured self-report items likely engages different psychological processes than producing unstructured descriptions (Bornstein, 2011; Meyer, 1997; Mihura, 2012), some researchers argue that LLMs will change mental health assessment by replacing self-reports with natural-language communication (Kjell et al., 2024). In the context of performance-based tasks, we believe they will not be replaced, but rather complemented and enhanced, making screening and coding less time-consuming and enabling busy clinicians and researchers to use these tasks more often and effectively.

Another advantage of using LLMs for coding is that they can process information in several languages (e.g., Rathje et al., 2024; Rosoł et al., 2023). Although we did not test it systematically, we had initial optimistic results testing this LLM coder in Portuguese, Spanish, and Italian. A future study should focus on testing these languages on datasets with strong criteria for each language. In this study, we presented results for the LLM MOR coder; however, we are currently working on many other LLM coders. We plan to make these available on the R-PAS website for assessors when ready.

High-Stakes Use, Ethics, Standards, and Governance

The use of LLMs in psychological assessment raises ethical and governance issues that extend beyond reliability and validity (Fareed et al., 2025; Putica et al., 2025) and brings additional technical and ethical responsibilities as society tends to tolerate human error more than machine error, and often shows aversion to automated decisions (Bigman & Gray, 2018; Dietvorst et al., 2015; Mayer et al., 2023). Hence, developers should strive to meet or exceed human benchmarks. Our framework operationalizes this by comparing model-model and model-human agreement to assessor-assessor benchmarks and providing evidence of validity against independent criteria also used with human scores.

Test standards prioritize governance and transparency with clear documentation of the theoretical and procedural basis of automated logic for expert evaluation (AERA et al., 2014; International Test Commission [ITC] & Association of Test Publishers [ATP], 2025). Because LLM processes are often opaque (also called black-box), developers should prioritize transparency in how outputs are generated and reviewed (Zhui et al., 2024). Our example supports this by providing outputs with a fixed structure and requiring brief, evidence-based explanations that point to the words or phrases that trigger each decision. Although it does not open the black-box, it makes outputs easier to inspect, debug, and evaluate systematically. In addition, responsible use requires ongoing monitoring rather than one-time validation (AERA et al., 2014; ITC & ATP, 2025), so we recommend periodic calibration checks, including repeated runs and intrarater and interrater comparisons against human-scored samples, to detect drift and verify that the models have not changed their outcomes.

Bias and fairness are also an ethical concern because LLMs may reflect training data biases and may misclassify culturally specific content (Fareed et al., 2025; Putica et al., 2025; Zhui et al., 2024). Although developers cannot control proprietary training data, emerging work suggests that prompt structure, constraints, and multi-step procedures can reduce biased outputs (Li et al., 2025; Morales & Raman, 2025). In our example, we added inclusion and exclusion rules to reduce false positives when cultural, artistic, or normative contexts were present.

Safety and clinical risk also require attention because LLMs can produce confident errors and overgeneralize from salient cue words (Fareed et al., 2025; Zhui et al., 2024). Testing standards emphasize governance for boundary cases, quality control, and human review, particularly when responses are unusual or ambiguous (AERA et al., 2014; ITC & ATP, 2025). Accordingly, in our MOR example, we used error analysis to identify failure patterns to add minimum thresholds and constraints (e.g., prompt changes for “fire” and “blood”).

Also, privacy and accountability are essential (Fareed et al., 2025; Zhui et al., 2024), as psychological tests can contain sensitive personal information. LLM-based scoring should follow institutional governance and applicable privacy standards (e.g., AERA et al., 2014; ITC & ATP, 2025), including data minimization, restricted access, and secure processing (Fareed et al., 2025; Zhui et al., 2024). In our example, we minimized risk by sending only the Response and Clarification Phase text without the respondent’s identifiers. Also, we note that provider safeguards can still assist by blocking content that contains personal identification. In our example, one response was automatically blocked because the respondent mentioned a person’s name, and OpenAI’s safeguards were triggered.

Limitations and Strengths

This study has limitations that should guide interpretation and inform future work. First, findings are limited to a single Rorschach variable (MOR) and are not generalized to other codes with different coding rules. Second, validation was conducted with a single external dataset, which is clinically focused. Broader replication across sites, languages, and populations will be valuable. Third, external validity analyses were restricted to MMPI-2 negative affect scales. Although correlations were in the expected range, additional validity evidence from known-groups, discriminant measures, and real-world outcomes would be informative. Fourth, although the framework addresses low-base-rate coding, the rarity of the MOR code led to just moderate positive predictive values (~.69–.71), meaning that about 30% of the LLM-flagged positives did not match the assessor codes. This underscores the importance of human review, especially for FP, and suggests opportunities for prompt refinement (e.g., using error analysis). Finally, the workflow treated each response independently, which maximizes psychometric clarity but may underuse cross-response contextual cues that human coders sometimes rely on.

This article also offers several strengths. First, it presents a clear two-stage framework that distinguishes flexible pre-validation from fixed psychometric evaluation procedures, aligning with reporting standards such as TRIPOD+AI, AERA, and ITC. Second, it emphasizes reliability and validity metrics suited to low-base-rate variables, improving comparability and transparency in evaluation. Third, validation was performed on an independent dataset with natural base rates. Fourth, the model demonstrated good to excellent performance relative to the human coders, with good response level agreement (κ ≈ .72), excellent protocol level ICCs (.94–.95), and near-perfect stability across independent runs (.97–.99). Fifth, construct-relevant external validity was preserved and slightly enhanced. Sixth, the manuscript explicitly highlights the pitfalls of accuracy and specificity in rare-event contexts and centers evaluation on kappa, PPV, and Sensitivity, providing a methodological contribution to the field. Seventh, operationally, the LLM coder’s slight overcoding at the protocol level favors clinical triage by reducing the likelihood of missed positives. Eighth, the study also compared multiple LLMs, documented prompt design decisions, and manually reviewed disagreements (i.e., error analysis) to target ambiguous thresholds. Finally, the analytic pipeline was deliberately structured for reproducibility so that other researchers can mirror and extend the approach.

Conclusion

We provided a replicable framework for evaluating LLM-based coders in psychological assessment, with a focus on evaluating psychometric rigor. The steps outlined can be generalized to other coding and classification tasks and other domains. This framework supports the responsible integration of LLM tools into clinical and research settings, empowering psychologists to assess and adopt these emerging technologies critically. The MOR case demonstrates that such LLM coders can achieve excellent protocol-level agreement, near-perfect stability, and external validity relations, supporting their suitability for research and careful clinical use.

Supplemental Material

sj-docx-1-asm-10.1177_10731911261455137 – Supplemental material for Evaluating LLM-Based Coders in Psychological Assessment: A Validation Framework With Application to the Rorschach Morbid Content Variable

Supplemental material, sj-docx-1-asm-10.1177_10731911261455137 for Evaluating LLM-Based Coders in Psychological Assessment: A Validation Framework With Application to the Rorschach Morbid Content Variable by Ruam P. F. A. Pimentel and Gregory J Meyer in Assessment

Supplemental Material

sj-docx-2-asm-10.1177_10731911261455137 – Supplemental material for Evaluating LLM-Based Coders in Psychological Assessment: A Validation Framework With Application to the Rorschach Morbid Content Variable

Supplemental material, sj-docx-2-asm-10.1177_10731911261455137 for Evaluating LLM-Based Coders in Psychological Assessment: A Validation Framework With Application to the Rorschach Morbid Content Variable by Ruam P. F. A. Pimentel and Gregory J Meyer in Assessment

Footnotes

ORCID iDs

Ruam P. F. A. Pimentel

Gregory J Meyer

Ethical Considerations

Ethical approval was not required for this article.

Consent to Participate

There are no human participants in this article, and informed consent is not required.

Consent for Publication

Not applicable.

Author Contributions

Ruam Pimentel, conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, software, visualization, writing – original draft, writing – review & editing; Gregory Meyer, conceptualization, methodology, resources, supervision, validation, writing – review & editing.

All authors approved the final version of the article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The first author is a postdoctoral fellow at the Rorschach Performance Assessment System® (R-PAS®). The second author is a member of the company that sells the manual for R-PAS® and associated products.

Data Availability Statement

The data are not publicly available but can be obtained by request.

AI Assistance

AI tools were used to enhance language, clarity, and conciseness. Specifically, we used Grammarly as the primary grammar checker and its AI writing assistance tool, which operates GPT models in the background. We also used Copilot, available in Microsoft Word, in other instances. Finally, we used Apple Intelligence in a few cases. All AI usage focused on improving language; it was not used to generate new content, to define concepts, or for other improper uses of AI in research. Most uses of AI did not require prompts; instead, we checked the AI tool’s automatic suggestions. The authors reviewed and edited all final outputs to maintain the original intent. The authors generated all content in this article, and none was generated by AI.

Supplemental Material

Supplemental material for this article is available online.

Notes

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Report and recommendations for the reauthorization of the institute of education sciences.

Bigman

Y. E.

Gray

(2018). People are averse to machines making moral decisions. Cognition, 181, 21–34. https://doi.org/10.1016/j.cognition.2018.08.003

Bornstein

R. F.

(2011). Toward a process-focused model of test score validity: Improving psychological assessment in science and practice. Psychological Assessment, 23(2), 532–544. https://doi.org/10.1037/a0022402

Breiman

(2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324

Brickman

Gupta

Oltmanns

J. R.

(2025). Large language models for psychological assessment: A comprehensive overview. Advances in Methods and Practices in Psychological Science, 8(3). https://doi.org/10.1177/25152459251343582

Brown

Mann

Ryder

Subbiah

Kaplan

J. D.

Dhariwal

Neelakantan

Shyam

Sastry

Askell

Agarwal

Herbert-Voss

Krueger

Henighan

Child

Ramesh

Ziegler

Winter

Amodei

(2020). Language models are few-shot learners. In Larochelle

Ranzato

Hadsell

Balcan

M. F.

Lin

(Eds.), Advances in neural information processing systems (Vol. 33, pp. 1877–1901). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284

Collins

G. S.

Moons

K. G. M.

Dhiman

Riley

R. D.

Beam

A. L.

Van Calster

Ghassemi

Liu

Reitsma

J. B.

Van Smeden

Boulesteix

A.-L.

Camaradou

J. C.

Celi

L. A.

Denaxas

Denniston

A. K.

Glocker

Golub

R. M.

Harvey

Heinze

. . .Logullo

(2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. British Medical Journal, 385, Article e078378. https://doi.org/10.1136/bmj-2023-078378

Cumming

Finch

(2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60(2), 170–180. https://doi.org/10.1037/0003-066X.60.2.170

10.

Dauphin

Siefert

(2025). From Llama to language: Prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists. Frontiers in Artificial Intelligence, 8, Article 1398885. https://doi.org/10.3389/frai.2025.1398885

11.

de Almeida Schneider

A. M.

Bandeira

D. R.

Meyer

G. J.

(2022). Rorschach Performance Assessment System (R-PAS) interrater reliability in a Brazilian adolescent sample and comparisons with three other studies. Assessment, 29(5), 859–871. https://doi.org/10.1177/1073191120973075

12.

Demszky

Yang

Yeager

D. S.

Bryan

C. J.

Clapper

Chandhok

Eichstaedt

J. C.

Hecht

Jamieson

Johnson

Jones

Krettek-Cobb

Lai

JonesMitchell

Ong

D. C.

Dweck

C. S.

Gross

J. J.

Pennebaker

J. W.

(2023). Using large language models in psychology. Nature Reviews Psychology, 2, 688–701. https://doi.org/10.1038/s44159-023-00241-5

13.

Devine

R. T.

Kovatchev

Grumley Traynor

Smith

Lee

(2023). Machine learning and deep learning systems for automated measurement of “advanced” theory of mind: Reliability and validity in children and adolescents. Psychological Assessment, 35(2), 165–177. https://doi.org/10.1037/pas0001186

14.

Dietvorst

B. J.

Simmons

J. P.

Massey

(2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. https://doi.org/10.1037/xge0000033

15.

Eberhardt

S. T.

Vehlen

Schaffrath

Schwartz

Baur

Schiller

Hallmen

André

Lutz

(2025). Development and validation of large language model rating scales for automatically transcribed psychological therapy sessions. Scientific Reports, 15(1), 29541. https://doi.org/10.1038/s41598-025-14923-y

16.

Ewbank

M. P.

Cummins

Tablan

Catarino

Buchholz

Blackwell

A. D.

(2021). Understanding the relationship between patient language and outcomes in internet-enabled cognitive behavioural therapy: A deep learning approach to automatic coding of session transcripts. Psychotherapy Research, 31(3), 300–312. https://doi.org/10.1080/10503307.2020.1788740

17.

Exner

J. E.

(2003). The rorschach: A comprehensive system (4th ed., Vol. 1). John Wiley & Sons.

18.

Fareed

Fatima

Uddin

Ahmed

Sattar

M. A.

(2025). A systematic review of ethical considerations of large language models in healthcare and medicine. Frontiers in Digital Health, 7, Article 1653631. https://doi.org/10.3389/fdgth.2025.1653631

19.

Furr

R. M.

(2022). Psychometrics: An introduction (4th ed.). Sage.

20.

Grove

W. M.

Andreasen

N. C.

McDonald-Scott

Keller

M. B.

Shapiro

R. W.

(1981). Reliability studies of psychiatric diagnosis: Theory and practice. Archives of General Psychiatry, 38(4), 408–413. https://doi.org/10.1001/archpsyc.1981.01780290042004

21.

Guo

Lai

Thygesen

J. H.

Farrington

Keen

(2024). Large language models for mental health applications: Systematic review. JMIR Mental Health, 11, Article e57400. https://doi.org/10.2196/57400

22.

Hajian-Tilaki

(2013). Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian Journal of Internal Medicine, 4(2), 627–635.

23.

Garcia

E. A.

(2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263–1284. https://doi.org/10.1109/TKDE.2008.239

24.

International Test Commission & Association of Test Publishers. (2025). Guidelines for technology-based assessment. https://www.intestcom.org/page/28

25.

Jeon

Yoo

Lee

Son

Kim

Han

(2024, March 21–22). A dual-prompting for interpretable mental health language models. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024) (pp. 247–255). Association for Computational Linguistics. https://doi.org/10.48550/ARXIV.2402.14854

26.

Kjell

O. N. E.

Kjell

Schwartz

H. A.

(2024). Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessment. Psychiatry Research, 333, 115667. https://doi.org/10.1016/j.psychres.2023.115667

27.

Kottner

Audigé

Brorson

Donner

Gajewski

B. J.

Hróbjartsson

Roberts

Shoukri

Streiner

D. L.

(2011). Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1), 96–106. https://doi.org/10.1016/j.jclinepi.2010.03.002

28.

Tang

Liu

Spirtes

Zhang

Leqi

Liu

(2025). Prompting fairness: Integrating causality to debias large language models. The 13th International Conference on Learning Representations (ICLR 2025), Singapore.

29.

Liu

Yuan

Jiang

Hayashi

Neubig

(2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35. https://doi.org/10.1145/3560815

30.

Mayer

M. M.

Buchner

Bell

(2023). Humans, machines, and double standards? The moral evaluation of the actions of autonomous vehicles, anthropomorphized autonomous vehicles, and human drivers in road-accident dilemmas. Frontiers in Psychology, 13, Article 1052729. https://doi.org/10.3389/fpsyg.2022.1052729

31.

McGrath

R. E.

Pogge

D. L.

Stokes

J. M.

Cragnolino

Zaccario

Hayman

Piacentini

Wayland-Smith

(2005). Field reliability of Comprehensive System scoring in an adolescent inpatient sample. Assessment, 12(2), 199–209. https://doi.org/10.1177/1073191104273384

32.

McGraw

K. O.

Wong

S. P.

(1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30

33.

Meehl

P. E.

Rosen

(1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52(3), 194–216. https://doi.org/10.1037/h0048070

34.

Meyer

G. J.

(1997). On the integration of personality assessment methods: The Rorschach and MMPI. Journal of Personality Assessment, 68(2), 297–330. https://doi.org/10.1207/s15327752jpa6802_5

35.

Meyer

G. J.

(1999). The convergent validity of MMPI and Rorschach Scales: An extension using profile scores to define response and character styles on both methods and a reexamination of simple Rorschach response frequency. Journal of Personality Assessment, 72(1), 1–35. https://doi.org/10.1207/s15327752jpa7201_1

36.

Meyer

G. J.

(2004). The reliability and validity of the Rorschach and TAT compared to other psychological and medical procedures: An analysis of systematically gathered evidence. In Hilsenroth

Segal

(Eds.), Personality assessment (Vol. 2, pp. 315–342). Wiley.

37.

Meyer

G. J.

Hilsenroth

M. J.

Baxter

Exner

J. E.

Fowler

J. C.

Piers

C. C.

Resnick

(2002). An examination of interrater reliability for scoring the Rorschach Comprehensive System in eight data sets. Journal of Personality Assessment, 78(2), 219–274. https://doi.org/10.1207/S15327752JPA7802_03

38.

Meyer

G. J.

Riethmiller

R. J.

Brooks

R. D.

Benoit

W. A.

Handler

(2000). A replication of Rorschach and MMPI-2 convergent validity. Journal of Personality Assessment, 74(2), 175–215. https://doi.org/10.1207/S15327752JPA7402_3

39.

Meyer

G. J.

Viglione

D. J.

Mihura

J. L.

Erard

R. E.

Erdberg

(2011). Rorschach performance assessment system: Administration, coding, interpretation, and technical manual. Rorschach Performance Assessment System LLC.

40.

Mihura

J. L.

(2012). The necessity of multiple test methods in conducting assessments: The role of the Rorschach and self-report. Psychological Injury and Law, 5(2), 97–106. https://doi.org/10.1007/s12207-012-9132-9

41.

Mihura

J. L.

Meyer

G. J.

Dumitrascu

Bombel

(2013). The validity of individual Rorschach variables: Systematic reviews and meta-analyses of the Comprehensive System. Psychological Bulletin, 139(3), 548–605. https://doi.org/10.1037/a0029406

42.

Mohammed

Kora

(2023). A comprehensive review on ensemble deep learning: Opportunities and challenges. Journal of King Saud University–Computer and Information Sciences, 35(2), 757–774. https://doi.org/10.1016/j.jksuci.2023.01.014

43.

Morales

Raman

(2025, February 26–March 1). Prompt-engineering strategies for minimizing bias in large language model outputs: Applications in computing education. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education (Vol. 2, 1743–1743). Association for Computing Machinery. https://doi.org/10.1145/3641555.3705080

44.

Organisciak

Acar

Dumas

Berthiaume

(2023). Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity, 49, 101356. https://doi.org/10.1016/j.tsc.2023.101356

45.

Park

J. Y.

Seo

E. H.

Yoon

H.-J.

Won

Lee

K. H.

(2023). Automating Rey Complex Figure Test scoring using a deep learning-based approach: A potential large-scale screening tool for cognitive decline. Alzheimer’s Research & Therapy, 15(1), 145. https://doi.org/10.1186/s13195-023-01283-w

46.

Pimentel

R. P. F. A.

Meyer

G. J.

(2025, May 8–10). Assessing manifestations of depressive behavior during the Rorschach task: Study 3 [Coordinated Session]. 2nd Annual Rorschach Performance Assessment System (R-PAS) in Multimethod Assessment Conference, Portland, OR, United States.

47.

Priyadarshana

Y. H. P. P.

Senanayake

Liang

Piumarta

(2024). Prompt engineering for digital mental health: A short review. Frontiers in Digital Health, 6, Article 1410947. https://doi.org/10.3389/fdgth.2024.1410947

48.

Putica

Khanna

Bosl

Saraf

Edgcomb

(2025). Ethical decision-making for AI in mental health: The integrated ethical approach for computational psychiatry (IEACP) framework. Psychological Medicine, 55, Article e213. https://doi.org/10.1017/S0033291725101311

49.

Qiu

Sun

Shao

Dai

Huang

(2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10), 1872–1897. https://doi.org/10.1007/s11431-020-1647-3

50.

Rathje

Mirea

D.-M.

Sucholutsky

Marjieh

Robertson

C. E.

Van Bavel

J. J.

(2024). GPT is an effective tool for multilingual psychological text analysis. Proceedings of the National Academy of Sciences, 121(34), Article e2308950121. https://doi.org/10.1073/pnas.2308950121

51.

R Core Team. (2025). R: A language and environment for statistical computing [Manual]. R Foundation for Statistical Computing. https://www.R-project.org/

52.

Rosoł

Gąsior

J. S.

Łaba

Korzeniewski

Młyńczak

(2023). Evaluation of the performance of GPT-3.5 and GPT-4 on the polish medical final examination. Scientific Reports, 13(1), 20512. https://doi.org/10.1038/s41598-023-46995-z

53.

Saretzki

Knopf

Forthmann

Goecke

Jaggy

A.-K.

Benedek

Weiss

(2025). Scoring German alternate uses items applying large language models. Journal of Intelligence, 13(6), 64. https://doi.org/10.3390/jintelligence13060064

54.

Schmidt

F. L.

Hunter

J. E.

(2015). Methods of meta-analysis: Correcting error and bias in research findings. Sage. https://doi.org/10.4135/9781483398105

55.

Shapley

Grofman

(1984). Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice, 43(3), 329–343.

56.

Shen

Houser

Smith

D. V.

Murty

V. P.

(2023). Machine-learning as a validated tool to characterize individual differences in free recall of naturalistic events. Psychonomic Bulletin & Review, 30(1), 308–316. https://doi.org/10.3758/s13423-022-02171-4

57.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

58.

Snow

O’Connor

Jurafsky

(2008, October 25–27). Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks. In Lapata

H. T.

(Eds.), Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (pp. 254–263). Association for Computational Linguistics. https://aclanthology.org/D08-1027/

59.

Swets

J. A.

(1988). Measuring the accuracy of diagnostic systems. Science, 240(4857), 1285–1293. https://doi.org/10.1126/science.3287615

60.

Tanana

Hallgren

K. A.

Imel

Z. E.

Atkins

D. C.

Srikumar

(2016). A comparison of natural language processing methods for automated coding of motivational interviewing. Journal of Substance Abuse Treatment, 65, 43–50. https://doi.org/10.1016/j.jsat.2016.01.006

61.

Viglione

D. J.

Blume-Marcovici

A. C.

Miller

H. L.

Giromini

Meyer

G. J.

(2012). An inter-rater reliability study for the Rorschach Performance Assessment System. Journal of Personality Assessment, 94(6), 607–612. https://doi.org/10.1080/00223891.2012.684118

62.

Viglione

D. J.

Meyer

G. J.

(2008). An overview of Rorschach psychometrics for forensic practice. In Gacono

C. B.

Evans

F. B.

Kaser-Boyd

Gacono

L. A.

(Eds.), Handbook of forensic Rorschach psychology (pp. 21–53). Erlbaum.

63.

Wang

Dsouza

Lee

Apperly

Devine

Van Der Kleij

Lee

(2025, May 3). Automatic scoring of an open-response measure of advanced mind-reading using large language models. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025) (pp. 79–89). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.clpsych-1.7

64.

Wei

Wang

Schuurmans

Bosma

Ichter

Xia

Chi

E. H.

Q. V.

Zhou

(2022, November 28–December 9). Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Nips ’22 (pp. 24824–24837). Curran Associates, Inc.

65.

Zhui

Fenghe

Xuehu

Qining

Wei

(2024). Ethical considerations and fundamental principles of large language models in medical education: Viewpoint. Journal of Medical Internet Research, 26, Article e60083. https://doi.org/10.2196/60083

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.02 MB

0.03 MB

0.00 MB