Leveraging Artificial Intelligence to Linguistically Compare Test Translations: A Methodological Introduction and Demonstration

Abstract

The validity and utility of translated instruments (psychological measures) depend on the quality of their translation, and differences in key linguistic characteristics could introduce bias. Likewise, linguistic differences between instruments designed to measure analogous constructs might contribute to similar instruments possessing dissimilar psychometrics. This article introduces and demonstrates the use of natural language processing (NLP), a subfield of artificial intelligence, to linguistically analyze 13 translations of two psychological measures previously translated into numerous languages. NLP was used to generate estimates reflecting specific linguistic characteristics of test items (emotional tone/intensity, sentiment, valence, arousal, and dominance), which were then compared across translations at both the test- and item-level, as well as between the two instruments. Results revealed that key linguistic characteristics can profoundly vary both within and between tests. Following a discussion of results, the current limitations of this approach are summarized and strategies for advancing this methodology are proposed.

Keywords

artificial intelligence psychometrics natural language processing linguistics test translation

The vast majority of self-report measurement instruments (i.e., tests) are constructed using a single language, requiring instrument translation before these can be used in research and practice with populations who communicate in other languages. The psychometric properties of these instruments across languages depend on accurate and culturally applicable translations, with translation quality conventionally demonstrated with traditional psychometric studies (e.g., criterion-related validity, measurement invariance). However, advances in the accessibility and validity of psychological assessment across languages and cultures require an increasingly better understanding of the similarities and differences between instrument translations because characteristics of test item wording, format, and context influence the way individuals respond on self-reports (Schwarz, 1999). One potentially important feature to examine is the degree of variability in key linguistic characteristics across translations. Most importantly, attention should be given to semantic and pragmatic differences between translations of the same measure that might influence how an individual responds to test items. Yet, the importance of understanding semantic and pragmatic differences between tests likely extends beyond the context of instrument translation, as these linguistic differences might also contribute to the divergent psychometric properties observed between instruments designed to measure the same latent construct.

Linguistic differences between instruments and test translations can feasibly introduce unintended measurement bias through multiple mechanisms. First, semantics and pragmatics are important components of language for the accurate communication of a fact or sentiment (Crystal, 1981). Consider the statements, “I’m sad” and “I’m depressed,” both of which generally communicate a negative emotional experience. However, the stronger negative sentiment transmitted by the second sentence—in this case, through the use of the word “depressed” instead of “sad”—changes the intensity of this statement and, consequently, changes what it would mean for a test responder to endorse this item. Most psychologists and psychometricians would agree that replacing the word “sad” with the word “depressed” is problematic because these words are not equivalent, and their exchange would change the latent meaning and intensity of the test item. However, determining the equivalence of words and multi-word items across languages is not as simple as the offered example (see Regmi et al., 2010). As such, translated items that are presumably equivalent might not convey the same meaning across languages and cultures.

Priming effects are a second mechanism through which linguistic differences might introduce measurement bias. Priming occurs when stimuli function as subtle contextual cues (primes) that increase the accessibility of mental concepts associated with the presented cues (Bargh & Chartrand, 2000). Studies repeatedly demonstrate the effects of lexical and semantic priming (Meyer & Schvaneveldt, 1971) on participants’ responses to psychological measures (e.g., Bartz & Lydon, 2004; Stapel & Blanton, 2004; Stapel & Koomen, 2000, 2001). Pertinent for this article, the sentiment and emotional tone/intensity of an item conceivably functions as a subtle contextual cue that influences the way an individual responds to the test item, and possibly successive items. These priming stimuli, which operate through multiple channels (e.g., cognition, perception, affect, motivation; Elgendi et al., 2018; Molden, 2014), can generate assimilating or contrasting effects: A test response can be defined as the product of an assimilating effect when the response distortion aligns with the priming stimulus (e.g., a test item with a sad emotional tone results in an individual rating themselves as more depressed than would be the case in the absence of the priming effect), whereas a contrasting effect induces a response distortion that diverges from the priming stimulus (e.g., when an individual denies ordinary experiences of sadness to distance themselves from an item asking about depression that has an overall negative sentiment).

Regardless of the mechanism(s) through which linguistic differences might be introducing measurement bias, the fact of the matter is that certain linguistic differences might, at least partially, be responsible for noninvariance seen across translations of psychological measures (e.g., Le Corff et al., 2022; Natoli et al., 2022; Sorrel et al., 2021). As noted above, the importance of understanding linguistic differences between tests might also underlie the divergent psychometrics observed when comparing tests that are expected to be psychometrically similar for other reasons (e.g., tests intended to measure the same latent construct) (Podsakoff et al., 2003). Before this potential issue can be addressed, it’s first necessary to understand the nature of variability in linguistic characteristics across instrument translations and between measures assessing analogous constructs and optimize a methodology appropriate for these lines of inquiry. Natural language processing (NLP), a subfield of artificial intelligence dedicated to helping machines learn to interpret, manipulate, and understand human language, offers a promising technology for meeting this objective.

Natural Language Processing

With psychological tests predominantly operating through written and spoken language, NLP is a uniquely capable technology for analyzing and comparing instrument translations. The utility of NLP can be categorized into two areas: natural language understanding and natural language generation, the first of which is more applicable to the current study. Paralleling the field of linguistics, natural language understanding analyzes the phonology, morphology, syntax, semantics, and pragmatics of text data, enabling machines to understand and analyze (parts of) language. The present study focuses on understanding the semantic and pragmatic levels of language, using NLP to recognize the “meaning” of a given test item and quantify its sentiment (i.e., the positivity or negativity expressed) and emotional tone/intensity. This is done by deconstructing each test item into its parts (i.e., words, called tokens), assigning each token a value representing the sentiment or emotion(s) represented and their intensity, and then aggregating these at the sentence (item) level. The resulting output delivers a vector of values for each test item representing sentiment, the presence and degree of different emotions, and their corresponding valence defined by validated lexicons. These results can then be analyzed and compared across items, tests, and translations.

The Current Study

This article introduces and demonstrates the use of NLP to linguistically analyze two measures of personality dysfunction that have been translated into numerous languages and validated: the Level of Personality Functioning Scale—Brief Form 2.0 (LPFS-BF 2.0; Weekers et al., 2019) and the Personality Disorder Severity ICD-11 Scale (PDS-ICD-11; Bach et al., 2021). To do so, test items from 13 translations of these instruments were subjected to emotion recognition and sentiment analysis using multilingual NRC Word-Emotion Association (NRC WEA; Mohammad & Turney, 2013), NRC Emotion Intensity (NRC EI; Mohammad, 2018a), and NRC Valence, Arousal, and Dominance (NRC VAD; Mohammad, 2018b) lexicons. This application of NLP was used to generate values reflecting specific linguistic characteristics of each test item (i.e., emotional tone/intensity, sentiment, valence, arousal, and dominance), which were then compared across translations at both the test- and item-level. As a secondary objective, these same linguistic characteristics were compared across the two test instruments to quantify (dis)similarity in the emotional tone/intensity, sentiment, valence, arousal, and dominance of different measures intended to assess equivalent latent constructs. As a whole, this study aimed to introduce and demonstrate a methodological approach for assessing variability in linguistic characteristics across instruments and across instrument translations.

Method

Test Instruments

Level of Personality Functioning Scale—Brief Form 2.0

The LPFS-BF 2.0 (Weekers et al., 2019) is a 12-item self-report measure of self-functioning and interpersonal functioning (i.e., personality functioning) impairment. Items 1 to 6 reflect aspects of self-functioning and items 7 to 12 reflect aspects of interpersonal functioning. Individual self-functioning impairment and interpersonal functioning impairment scores are calculated by aggregating the six items comprising each scale, and a total personality functioning impairment score is calculated by summing all 12 items. The LPFS-BF 2.0 has demonstrated strong psychometric properties when administered in several languages, including Danish, Dutch, English, Estonian, French, German, Italian, and Spanish (Combaluzier et al., 2023; Cottin et al., in preparation; Gritti et al., in preparation; Le Corff et al., 2022; Natoli et al., 2022; Oitsalu et al., 2022; Spitzer et al., 2021; Weekers et al., 2019, 2023; Zimmermann et al., 2020).

Personality Disorder Severity ICD-11 Scale

The PDS-ICD-11 (Bach et al., 2021) is a 14-item self-report measure designed to assess global personality disorder severity as operationalized in the chapter on personality disorders and related traits in ICD-11. Test items reflect the literal concepts and wordings of the ICD-11’s framework for personality disorder severity determination and can be summed to produce an overall severity index. The PDS-ICD-11 has demonstrated strong psychometric properties when administered in multiple different languages, including Danish, English, and Spanish (Bach et al., 2021; Gutiérrez et al., 2023; Natoli & Rodriguez, 2024).

At the time of data analysis, the LPFS-BF 2.0 and PDS-ICD-11 had both been translated into the following languages that use Latin character sets¹: Czech, Danish, Dutch, English, Estonian, French, German, Italian, Norwegian, Polish, Portuguese, Spanish, and Turkish.

Lexicons

Three existing manually curated multilingual lexicons that are publicly available were used for the current study. The NRC EI (Mohammad, 2018a) lexicon consists of over 10,000 terms associated with emotions to varying degrees. Each term in the lexicon has eight corresponding intensity scores for eight basic emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, and trust), which vary from 0 (conveys the lowest amount of a given emotion) to 1 (conveys the highest amount of a given emotion). For instance, the term “disagreement” would be assigned an anger intensity score of .348, a sadness intensity score of .438, and an intensity score of .000 for the remaining six emotions. The lexicon’s intensity scores were derived by manual annotators following a standardized annotation scheme and best–worst scaling procedure (see Mohammad, 2018a). The NRC WEA (Mohammad & Turney, 2013), containing approximately 14,000 terms, and the NRC VAD (Mohammad, 2018b), containing over 20,000 terms, were created following similar procedures as the NRC EI. These two lexicons were used in the current study to generate positive/negative sentiment values and valence, arousal, and dominance values, respectively. Positive and negative sentiment is assigned to each term in a test item dichotomously (1 = present, 0 = not present) whereas assigned values of valence (positive—negative), arousal (active—passive), and dominance (full control—lacks control) reflect a term’s intensity in a given domain, ranging from 0 to 1, with greater values reflecting more positive valence, stronger arousal, or greater dominance. The NRC EI, NRC WEA, and NRC VAD lexicons have been translated by their developers into over 100 languages and, although some cultural differences exist, their stability has been shown across languages (Mohammad, 2020; https://www.saifmohammad.com/).

Procedures and Analysis

Data Preprocessing

All test items were preprocessed prior to linguistic analysis. First, test item text was standardized to make items easier to process by converting all letters to lower-case, removing punctuation, and converting all numerals to words. As the tasks of this study were mostly aligned with traditional sentiment classification and analysis at the sentence (test item) level, no tokenization was employed during preprocessing. Stop words were not removed to avoid the unintentional removal of their potential contribution to the sentiment of a given test item. Likewise, no stemming technique or lemmatization was used. Given the unique structure of the PDS-ICD-11, wherein differentiating content is included in the response options of test items rather than in the test item statement itself, linguistic analysis was performed on the individual response options and then aggregated for each test item.

Linguistic Analysis

The Syuzhet (Jockers, 2015) package in R was used with multilingual NRC WEA, NRC VAD, and NRC EI lexicons to generate estimates reflecting emotional tone/intensity, sentiment, valence, arousal, and dominance for each test item within each language. Estimates were aggregated across all items within a given instrument to derive test-level estimates for each linguistic characteristic for each translation. A second set of test-level estimates were derived in a similar way after weighting the emotional tone/intensity, sentiment, valence, arousal, and dominance values by the total word count of a given translation. Descriptive data were then compared across translations at both the test- and item-level, and cross-translation estimates weighted for word count and their accompanying 95% confidence intervals (CIs) were used to compare the linguistic characteristics of the LPFS-BF 2.0 to those of the PDS-ICD-11.

Percentage agreement can be an inflated estimate of uniformity due to chance agreement. As such, consistency of each linguistic characteristic estimate across translations at the overall test level was also evaluated by calculating intraclass correlation coefficients (ICC; two-way mixed effects consistency of a single measurement [3,1]) and Krippendorff’s alpha (α) as additional measures of uniformity with corresponding 95% confidence intervals computed for each ICC and Krippendorff’s α. ICCs and Krippendorff’s α’s were computed using the psych (Revelle, 2024) and kripp.boot (Proutskova & Gruszczynski, 2007) packages in R, respectively. Code for linguistic analysis and computation of consistency metrics is available at https://osf.io/urghw.

Results

Test Level Comparisons

Overall levels of emotional tone/intensity, sentiment, valence, arousal, and dominance for the LPFS-BF 2.0 and PDS-ICD-11 and their associated means, standard deviations, and 95% CIs are reported in Table 1. Parallel estimates weighted by word count are reported in Table 2. As illustrated in Figure 1, most LPFS-BF 2.0 translations possess a relatively positive overall sentiment (upward slope), whereas all but two translations of the PDS-ICD-11 possessed a relatively negative overall sentiment (downward slope). Overall sentiment of LPFS-BF 2.0 translations ranged from −9 (Polish) to +8 (Spanish), with an average overall sentiment across translations (M = +0.77, SD = 5.10) that did not significantly differ from zero based on 95% CIs. The sentiment of each LPFS-BF 2.0 translation fell within ±1.96 SDs of the mean. The overall sentiment of PDS-ICD-11 translations (M = −18.69, SD = 17.30) was found to be significantly negative based on 95% CIs and ranged from −61 (Estonian) to +11 (Czech). After weighting estimates by word count, the average overall sentiment of the LPFS-BF 2.0 (M = 0.007, SD = 0.042) was found to be more positive than the overall sentiment of the PDS-ICD-11 (M = -0.023, SD = 0.024), which remained significantly less than zero; however, the difference in overall sentiment between the two instruments was not statistically significant based on 95% CIs.

Table 1

Overall Levels of Emotional Intensity, Sentiment, Valence, Arousal, and Dominance for the LPFS-BF 2.0 and PDS-ICD-11.

	Language															95% CIs
Characteristic	1	2	3	4	5	6	7	8	9	10	11	12	13	Mean	SD	LL	UL
LPFS-BF 2.0
Anger	0.379	0.294	0.156	0.000	0.000	0.000	0.000	0.000	0.453	0.000	1.052	0.379	0.988	0.285	0.354	0.092	0.477
Anticipation	1.039	1.516	1.121	0.906	0.461	0.000	0.000	2.617	1.644	0.000	1.071	1.562	4.122	1.235	1.109	0.633	1.838
Disgust	0.000	0.282	0.304	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.422	0.078	0.145	−0.001	0.156
Fear	0.432	0.656	0.000	0.537	0.718	0.000	0.000	0.439	0.718	0.000	0.866	0.234	0.945	0.427	0.335	0.244	0.609
Joy	0.954	1.339	3.654	2.035	0.531	0.000	0.394	1.078	5.065	2.797	0.695	1.091	5.102	1.903	1.662	0.999	2.806
Sadness	0.766	0.344	0.000	1.056	0.455	0.469	0.455	0.500	0.757	0.719	1.976	0.500	1.009	0.693	0.458	0.444	0.942
Surprise	0.375	0.890	0.719	0.930	0.000	0.000	0.000	0.000	0.961	0.000	0.000	0.000	1.906	0.445	0.575	0.132	0.757
Trust	4.720	3.459	4.652	3.296	1.766	2.039	1.391	2.937	4.417	1.289	2.461	2.992	2.754	2.936	1.118	2.328	3.544
Negative	1.000	4.000	1.000	0.000	8.000	0.000	7.000	1.000	7.000	9.000	8.000	2.000	6.000	4.154	3.301	2.359	5.948
Positive	5.000	6.000	4.000	4.000	1.000	4.000	2.000	4.000	11.000	0.000	3.000	10.000	10.000	4.923	3.339	3.108	6.738
Sentiment	4.000	2.000	3.000	4.000	−7.000	4.000	−5.000	3.000	4.000	−9.000	−5.000	8.000	4.000	0.769	5.101	−2.004	3.542
Valence	27.864	18.412	48.963	47.580	29.025	28.564	30.284	36.371	47.86	33.879	28.276	23.102	40.946	33.933	9.424	28.810	39.056
Arousal	19.892	14.012	33.767	37.924	25.699	20.863	23.771	30.494	38.357	27.646	22.184	17.037	32.641	26.484	7.476	22.420	30.547
Dominance	24.749	17.319	44.579	45.703	29.627	23.478	29.275	36.294	47.268	23.4	27.169	21.063	38.041	31.382	9.607	26.160	36.604
PDS-ICD-11
Anger	7.196	1.166	8.053	8.358	1.644	6.515	6.557	10.656	10.809	0.578	10.465	4.136	6.996	6.395	3.405	4.544	8.245
Anticipation	7.589	2.220	3.986	4.062	0.000	3.420	1.446	8.474	6.564	0.391	3.317	5.890	9.791	4.396	2.962	2.786	6.006
Disgust	4.956	0.564	5.337	2.912	4.520	7.067	2.617	3.507	6.342	0.890	7.877	2.474	5.386	4.188	2.175	3.006	5.371
Fear	16.618	1.953	8.984	8.448	7.693	6.059	2.829	11.161	11.627	2.405	14.184	5.698	13.914	8.583	4.567	6.100	11.065
Joy	14.323	11.553	14.445	11.428	4.914	7.007	4.364	11.698	16.400	5.628	7.545	7.557	11.204	9.851	3.779	7.797	11.905
Sadness	10.070	1.157	13.529	11.688	3.553	11.235	8.301	8.515	11.257	4.528	15.875	7.620	9.462	8.984	3.925	6.850	11.117
Surprise	3.875	1.780	1.032	0.751	3.812	1.438	1.626	0.445	4.966	0.000	3.920	0.000	3.804	2.111	1.659	1.209	3.013
Trust	21.594	17.311	16.449	10.410	4.163	16.468	10.300	19.567	23.216	1.633	13.252	11.602	6.482	13.265	6.377	9.798	16.732
Negative	43.000	8.000	24.000	32.000	66.000	43.000	43.000	65.000	72.000	33.000	49.000	30.000	67.000	44.231	18.444	34.205	54.257
Positive	30.000	19.000	12.000	27.000	5.000	29.000	21.000	32.000	43.000	8.000	23.000	34.000	49.000	25.538	12.358	18.821	32.256
Sentiment	−13.000	11.000	−12.000	−5.000	−61.000	−14.000	−22.000	−33.000	−29.000	−25.000	−26.000	4.000	−18.000	−18.692	17.296	−28.094	−9.291
Valence	156.338	82.082	220.577	224.222	138.150	184.807	139.937	211.859	226.563	121.807	241.117	152.654	163.712	174.140	46.410	148.912	199.369
Arousal	125.741	51.450	194.919	199.103	128.350	156.001	121.690	213.299	223.221	97.067	201.124	120.137	154.327	152.802	49.324	125.990	179.615
Dominance	148.136	67.588	206.899	222.739	132.547	160.452	132.048	216.651	221.129	106.210	226.419	140.560	161.537	164.840	48.579	138.433	191.247

Note. 1 = English, 2 = Czech, 3 = Danish, 4 = Dutch, 5 = Estonian, 6 = French, 7 = German, 8 = Italian, 9 = Norwegian, 10 = Polish, 11 = Portuguese, 12 = Spanish, 13 = Turkish.

Table 2

Overall Levels of Emotional Intensity, Sentiment, Valence, Arousal, and Dominance for the LPFS-BF 2.0 and PDS-ICD-11 Weighted by Word Count.

	Language															95% CIs
Characteristic	1	2	3	4	5	6	7	8	9	10	11	12	13	Mean	SD	LL	UL
LPFS-BF 2.0
Anger	0.003	0.003	0.001	0.000	0.000	0.000	0.000	0.000	0.003	0.000	0.008	0.003	0.011	0.002	0.003	0.001	0.004
Anticipation	0.008	0.015	0.009	0.007	0.004	0.000	0.000	0.020	0.013	0.000	0.008	0.013	0.045	0.011	0.012	0.005	0.017
Disgust	0.000	0.003	0.002	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.005	0.001	0.001	0.000	0.002
Fear	0.003	0.006	0.000	0.004	0.006	0.000	0.000	0.003	0.006	0.000	0.006	0.002	0.010	0.004	0.003	0.002	0.005
Joy	0.008	0.013	0.029	0.016	0.005	0.000	0.003	0.008	0.039	0.023	0.005	0.009	0.056	0.016	0.016	0.008	0.025
Sadness	0.006	0.003	0.000	0.008	0.004	0.003	0.004	0.004	0.006	0.006	0.015	0.004	0.011	0.006	0.004	0.004	0.008
Surprise	0.003	0.009	0.006	0.007	0.000	0.000	0.000	0.000	0.007	0.000	0.000	0.000	0.021	0.004	0.006	0.001	0.007
Trust	0.037	0.034	0.036	0.026	0.015	0.015	0.011	0.022	0.034	0.011	0.018	0.025	0.030	0.024	0.009	0.019	0.029
Negative	0.008	0.039	0.008	0.000	0.068	0.000	0.055	0.008	0.054	0.074	0.060	0.016	0.066	0.035	0.028	0.020	0.050
Positive	0.039	0.058	0.031	0.031	0.009	0.029	0.016	0.030	0.085	0.000	0.022	0.082	0.110	0.042	0.031	0.025	0.059
Sentiment	0.031	0.019	0.023	0.031	−0.060	0.029	−0.039	0.023	0.031	−0.074	−0.037	0.066	0.044	0.007	0.042	−0.016	0.029
Valence	0.219	0.179	0.383	0.369	0.248	0.204	0.238	0.276	0.368	0.280	0.211	0.189	0.450	0.278	0.083	0.233	0.323
Arousal	0.157	0.136	0.264	0.294	0.220	0.149	0.187	0.231	0.295	0.228	0.166	0.140	0.359	0.217	0.067	0.181	0.254
Dominance	0.195	0.168	0.348	0.354	0.253	0.168	0.231	0.275	0.364	0.193	0.203	0.173	0.418	0.257	0.083	0.212	0.302
PDS-ICD-11
Anger	0.009	0.002	0.009	0.009	0.002	0.007	0.008	0.013	0.012	0.001	0.012	0.005	0.011	0.008	0.004	0.005	0.010
Anticipation	0.010	0.003	0.005	0.005	0.000	0.004	0.002	0.010	0.008	0.001	0.004	0.007	0.015	0.005	0.004	0.003	0.008
Disgust	0.006	0.001	0.006	0.003	0.006	0.008	0.003	0.004	0.007	0.001	0.009	0.003	0.008	0.005	0.003	0.004	0.006
Fear	0.021	0.003	0.010	0.009	0.011	0.006	0.003	0.013	0.013	0.003	0.016	0.007	0.021	0.011	0.006	0.007	0.014
Joy	0.018	0.016	0.017	0.013	0.007	0.007	0.005	0.014	0.019	0.007	0.009	0.009	0.017	0.012	0.005	0.010	0.015
Sadness	0.013	0.002	0.016	0.013	0.005	0.012	0.010	0.010	0.013	0.006	0.018	0.009	0.014	0.011	0.004	0.008	0.013
Surprise	0.005	0.002	0.001	0.001	0.005	0.002	0.002	0.001	0.006	0.000	0.004	0.000	0.006	0.003	0.002	0.001	0.004
Trust	0.028	0.024	0.019	0.012	0.006	0.018	0.012	0.023	0.027	0.002	0.015	0.014	0.010	0.016	0.008	0.012	0.020
Negative	0.055	0.011	0.028	0.036	0.093	0.046	0.050	0.077	0.083	0.042	0.055	0.035	0.101	0.055	0.026	0.041	0.069
Positive	0.038	0.026	0.014	0.030	0.007	0.031	0.024	0.038	0.049	0.010	0.026	0.040	0.074	0.031	0.017	0.022	0.041
Sentiment	−0.017	0.015	−0.014	−0.006	−0.086	−0.015	−0.025	−0.039	−0.033	−0.032	−0.029	0.005	−0.027	−0.023	0.024	−0.036	−0.011
Valence	0.200	0.114	0.254	0.251	0.195	0.197	0.162	0.251	0.261	0.156	0.272	0.180	0.248	0.211	0.047	0.185	0.236
Arousal	0.161	0.071	0.225	0.222	0.182	0.166	0.141	0.253	0.257	0.124	0.227	0.141	0.233	0.185	0.054	0.155	0.214
Dominance	0.190	0.094	0.238	0.249	0.187	0.171	0.153	0.257	0.254	0.136	0.255	0.165	0.244	0.200	0.052	0.171	0.228

Note. 1 = English, 2 = Czech, 3 = Danish, 4 = Dutch, 5 = Estonian, 6 = French, 7 = German, 8 = Italian, 9 = Norwegian, 10 = Polish, 11 = Portuguese, 12 = Spanish, 13 = Turkish.

Plotting the overall valence, arousal, and dominance of the LPFS-BF 2.0 and PDS-ICD-11 translations (Figure 2) revealed similar patterns across translations and for both instruments (i.e., relatively lower arousal and higher valence and dominance). Each LPFS-BF 2.0 translation was each within ±1.96 SDs of the respective mean for overall valence (M = 33.93, SD = 9.42), arousal (M = 26.48, SD = 7.48), and dominance (M = 31.38, SD = 9.61). Regarding the valence (M = 174.14, SD = 46.41), arousal (M = 152.80, SD = 49.32), and dominance (M = 164.84, SD = 48.58) of the PDS-ICD-11 translations, the Czech translation was 1.98, 2.05, and 2.00 standard deviations below the cross-translation mean, respectively, while all other translations were within ±1.96 SDs of the respective mean. When weighted by word count, average arousal and dominance of the LPFS-BF 2.0 and PDS-ICD-11 test items were similar, whereas the LPFS-BF 2.0 was again found to have a more positive valence (see Table 2).

Similar patterns across translations of the LPFS-BF 2.0 and PDS-ICD-11 were seen for the overall emotional tone and intensity of these instruments (see Figure 3), and all estimates (aside from disgust for the LPFS-BF 2.0) were significantly different from zero based on their corresponding 95% CIs. Both instruments demonstrated relatively greater elevations in joy and trust across most languages, while relatively greater elevations in anticipation were observed across the majority of LPFS-BF 2.0 translations and relatively greater elevations in fear, sadness, and to a slightly lesser degree, anger were seen across most translations of the PDS-ICD-11. Of the LPFS-BF 2.0 translations, the emotional intensity of the Turkish LPFS-BF 2.0 was found to be notably stronger in anger (+1.98 SDs), anticipation (+2.61 SDs), disgust (+2.38 SDs), and surprise (+2.54 SDs) relative to corresponding means across translations, and the emotional tone of the Portuguese LPFS-BF 2.0 was notably stronger with regard to anger (+2.17 SDs) and sadness (+2.80 SDs). The emotional intensity of PDS-ICD-11 translations was predominantly within ±1.96 SDs of their respective means; however, the sadness of the Czech PDS-ICD-11 was found to 1.99 standard deviations below the cross-translation mean. Comparisons of cross-translation emotional tone and intensity estimates for the LPFS-BF 2.0 and PDS-ICD-11 weighted for word count revealed the emotional intensity of the PDS-ICD-11 was significantly stronger with regard to anger, disgust, fear, and sadness while the emotional intensity of the LPFS-BF 2.0 was significantly stronger with regard to joy (see Table 2).

Percentage agreement, ICCs, and Krippendorff’s α’s reflecting overall consistency of each linguistic characteristic across translations of the LPFS-BF 2.0 and PDS-ICD-11 are reported in Table 3. Percentage agreement (exact agreement) ranged from 0% to 66.67% (Disgust) across LPFS-BF 2.0 translations and from 0% to 63.75% (Surprise) across PDS-ICD-11 translations, with no estimate crossing the common threshold for acceptability (typically >70% agreement). Poor consistency across translations was also illustrated by the computed ICCs and Krippendorff’s α’s in Table 3, none of which reached traditionally acceptable levels of agreement (ICC > 0.5, Koo & Li, 2016; α > .67, Krippendorff, 2019).

Table 3

Overall Consistency of Linguistic Characteristic Across LPFS-BF 2.0 and PDS-ICD-11 Translations.

		Percentage Agreement	ICC (3,1)			Krippendorff’s α
Characteristic		Exact (%)	ICC	LL	UL	α	LL	UL
Negative^a	LPFS-BF 2.0	8.33	0.113	0.020	0.343	.048	−.057	.158
	PDS-ICD-11	16.25	0.229	0.166	0.312	.205	.134	.270
Positive^a	LPFS-BF 2.0	8.33	0.352	0.180	0.636	.251	.164	.346
	PDS-ICD-11	18.75	0.159	0.107	0.231	.173	.091	.257
Anger	LPFS-BF 2.0	50.00	0.111	0.019	0.341	.097	−.174	.380
	PDS-ICD-11	30.00	0.141	0.091	0.209	.131	.079	.179
Anticipation	LPFS-BF 2.0	25.00	0.227	0.091	0.503	.181	.069	.289
	PDS-ICD-11	42.50	0.178	0.123	0.253	.163	.067	.249
Disgust	LPFS-BF 2.0	66.67	0.000	−0.042	0.130	−.011	−.924	.702
	PDS-ICD-11	50.00	0.192	0.134	0.269	.185	.107	.256
Fear	LPFS-BF 2.0	41.67	0.268	0.119	0.550	.244	.092	.378
	PDS-ICD-11	27.50	0.363	0.286	0.457	.341	.295	.384
Joy	LPFS-BF 2.0	8.33	0.235	0.096	0.512	.179	.056	.308
	PDS-ICD-11	20.00	0.223	0.161	0.305	.212	.169	.251
Sadness	LPFS-BF 2.0	25.00	0.190	0.067	0.456	.176	.049	.296
	PDS-ICD-11	26.25	0.235	0.172	0.319	.222	.183	.261
Surprise	LPFS-BF 2.0	50.00	0.107	0.017	0.334	.085	−.076	.235
	PDS-ICD-11	63.75	0.103	0.061	0.163	.099	−.031	.229
Trust	LPFS-BF 2.0	0.00	0.425	0.238	0.699	.403	.323	.477
	PDS-ICD-11	17.50	0.224	0.162	0.307	.207	.164	.251
Valence	LPFS-BF 2.0	0.00	0.275	0.124	0.558	.239	.129	.337
	PDS-ICD-11	0.00	0.454	0.373	0.548	.214	.170	.248
Arousal	LPFS-BF 2.0	0.00	0.201	0.074	0.470	.171	.046	.291
	PDS-ICD-11	0.00	0.398	0.318	0.492	.195	.154	.236
Dominance	LPFS-BF 2.0	0.00	0.320	0.156	0.605	.257	.160	.347
	PDS-ICD-11	0.00	0.449	0.367	0.543	.201	.156	.233

Note.^aKrippendorff’s α computed using ordinal specification due to data type. Otherwise, Krippendorff’s α computed using ratio specification.

Item Level Comparisons

Levels of emotional tone/intensity, sentiment, valence, arousal, and dominance for each test item, averaged across translations, are reported in Tables 4 and 5 (see Tables S1 and S2 in the Supplemental Materials for parallel tables reporting item-level estimates weighted by word count). The emotional tone/intensity, sentiment, valence, arousal, and dominance for each test item within each language are presented as heatmap tables in the Supplemental Materials for the LPFS-BF 2.0 and PDS-ICD-11 (Supplemental Tables S3 and S4, respectively; weighted by word count in Supplemental Tables S5 and S6, respectively). The average sentiment of individual LPFS-BF 2.0 items (M = 0.064, SD = 0.600) was closely split, with items 4, 6, 9, 10, and 12 demonstrating relatively positive sentiment, items 2, 3, 5, 7, and 8 demonstrating relatively negative sentiment, and items 1 and 11 producing a balance between negative and positive sentiment. The estimated sentiment of item 12, averaged across LPFS-BF 2.0 translations, was 2.71 standard deviations above the mean (positive sentiment). Regarding average sentiment of individual PDS-ICD-11 items across translations (M = −1.34, SD = 3.06), eight of the 14 items possessed relatively negative sentiment (items 1, 4, 5, 7, 8, 11, 12, 13), while the remaining six items possessed relatively positive sentiment (items 2, 3, 6, 9, 10, 14). The estimated sentiment of item 13 was 2.53 standard deviations below the average sentiment for PDS-ICD-11 items across all translations (negative sentiment).

Table 4

Emotional Intensity, Sentiment, Valence, Arousal, and Dominance for Individual Items of the LPFS-BF 2.0 Averaged Across Translations.

Item	Anger	Anticipation	Disgust	Fear	Joy	Sadness	Surprise	Trust	Negative	Positive	OverallSentiment	Valence	Arousal	Dominance
(1) I often do not know who I really am
	0.000	0.022	0.000	0.000	0.054	0.000	0.000	0.049	0.154	0.154	0.000	2.093	1.661	1.670
(2) I often think very negatively about myself
	0.047	0.000	0.032	0.027	0.000	0.220	0.000	0.042	0.692	0.154	−0.538	2.185	1.926	2.168
(3) My emotions change without me having a grip on them
	0.000	0.000	0.000	0.015	0.046	0.011	0.000	0.018	0.231	0.000	−0.231	0.939	0.696	0.766
(4) I have no sense of where I want to go in my life
	0.000	0.171	0.000	0.000	0.390	0.000	0.000	0.099	0.154	0.308	0.154	3.233	2.525	2.576
(5) I often do not understand my own thoughts and feelings
	0.011	0.127	0.011	0.025	0.103	0.013	0.075	0.080	0.308	0.154	−0.154	2.259	1.568	2.016
(6) I often make unrealistic demands on myself
	0.000	0.040	0.000	0.000	0.053	0.020	0.000	0.082	0.000	0.077	0.077	1.923	1.483	1.820
(7) I often have difficulty understanding the thoughts and feelings of others
	0.128	0.082	0.011	0.152	0.133	0.198	0.075	0.182	0.769	0.692	−0.077	3.158	2.554	3.183
(8) I often find it hard to stand it when others have a different opinion
	0.072	0.040	0.000	0.162	0.055	0.129	0.032	0.029	1.000	0.154	−0.846	3.068	3.262	3.495
(9) I often do not fully understand why my behavior has a certain effect on others
	0.000	0.202	0.000	0.000	0.174	0.036	0.070	0.407	0.308	0.923	0.615	4.291	2.969	4.163
(10) My relationships and friendships never last long
	0.000	0.147	0.000	0.000	0.170	0.000	0.036	0.900	0.154	0.231	0.077	2.414	1.890	2.307
(11) I often feel very vulnerable when relations become more personal
	0.014	0.000	0.000	0.031	0.141	0.039	0.000	0.626	0.231	0.231	0.000	2.525	2.094	2.249
(12) I often do not succeed in cooperating with others in a mutually satisfactory way
	0.012	0.404	0.023	0.014	0.583	0.026	0.157	0.422	0.154	1.846	1.692	5.845	3.855	4.969
	Anger	Anticipation	Disgust	Fear	Joy	Sadness	Surprise	Trust	Negative	Positive	OverallSentiment	Valence	Arousal	Dominance
Mean	0.024	0.103	0.006	0.036	0.159	0.058	0.037	0.245	0.346	0.410	0.064	2.828	2.207	2.615
SD	−0.038	−0.113	−0.011	−0.056	−0.160	−0.075	−0.047	−0.272	−0.292	−0.502	−0.600	−1.208	−0.834	−1.109

Table 5

Emotional Intensity, Sentiment, Valence, Arousal, and Dominance for Individual Items of the PDS-ICD-11 Averaged Across Translations.

Item	Anger	Anticipation	Disgust	Fear	Joy	Sadness	Surprise	Trust	Negative	Positive	OverallSentiment	Valence	Arousal	Dominance
(1) Identity
	0.198	0.203	0.114	0.172	0.548	0.051	0.294	1.096	2.000	1.692	−0.308	15.255	12.563	14.381
(2) Self-worth
	0.240	0.791	0.264	0.333	1.590	0.343	0.346	1.809	2.538	4.769	2.231	18.088	12.813	15.584
(3) Self-perception
	0.096	0.236	0.000	0.087	0.452	0.000	0.084	0.783	0.692	1.000	0.308	4.505	3.166	3.976
(4) Goals
	0.373	0.317	0.090	0.829	1.994	0.748	0.278	1.282	3.462	1.846	−1.615	15.064	15.123	16.848
(5) Interest in relationships
	0.264	0.261	0.142	0.162	0.667	0.949	0.051	0.560	3.000	2.462	−0.538	12.199	9.429	10.353
(6) Perspective-taking
	0.063	0.608	0.011	0.114	0.744	0.089	0.287	0.577	0.692	2.308	1.615	14.602	8.988	11.812
(7) Mutuality in relationships
	0.830	0.062	0.725	0.246	0.370	0.728	0.077	2.149	3.462	1.000	−2.462	11.699	10.432	10.910
(8) Disagreement management
	0.852	0.267	0.255	0.406	0.678	0.870	0.044	1.246	2.385	1.923	−0.462	10.953	10.682	10.730
(9) Emotional control and expression
	0.486	0.183	0.152	0.223	0.375	0.704	0.000	0.559	1.615	1.769	0.154	15.228	13.100	14.496
(10) Behavioral control
	0.172	0.513	0.046	0.260	0.985	0.352	0.017	0.884	1.615	1.769	0.154	20.037	16.443	18.875
(11) Experience of reality during stress
	0.669	0.793	0.514	2.376	0.576	1.199	0.353	0.942	5.154	2.692	−2.462	17.537	16.824	16.628
(12) Harm to self
	0.894	0.022	0.823	1.527	0.035	1.165	0.077	0.088	7.154	0.077	−7.077	4.065	6.710	4.970
(13) Harm to others (intentional or unintentional)
	1.178	0.044	1.009	1.747	0.274	1.449	0.162	0.375	9.846	0.769	−9.077	5.160	9.222	6.675
(14) In thinking about your answers to the above, how much do they cause problems in important areas of your life (e.g., personal, family, social, education, work)?
	0.080	0.095	0.044	0.102	0.563	0.337	0.043	0.916	0.615	1.462	0.846	9.748	7.307	8.602
	Anger	Anticipation	Disgust	Fear	Joy	Sadness	Surprise	Trust	Negative	Positive	OverallSentiment	Valence	Arousal	Dominance
Mean	0.457	0.314	0.299	0.613	0.704	0.642	0.151	0.948	3.159	1.824	−1.335	12.439	10.914	11.774
SD	0.350	0.253	0.320	0.707	0.501	0.442	0.126	0.531	2.544	1.061	3.062	4.921	3.723	4.436

The greatest levels of arousal on the LPFS-BF 2.0 were seen for items 12 (+1.98 SDs) and 8 (+1.27 SDs), while item 3 generated the lowest arousal (−1.81 SDs). LPFS-BF 2.0 items 12 (+2.12 SDs) and 9 (+1.40 SDs) produced the highest levels of dominance, with item 3 (−1.67 SDs) producing the lowest levels. The most arousing items on the PDS-ICD-11, averaged across translations, were items 11 (+1.59 SDs), 10 (+1.49 SDs), and 4 (+1.13 SDs), while items 3 (−2.08 SDs) and 12 (−1.13 SDs) communicated the lowest levels of arousal. Similarly, items 10 (+1.60 SDs), 4 (+1.14 SDs), and 11 (+1.09 SDs) were found to be the items with the greatest dominance while items 3 (−1.76 SDs), 12 (−1.53 SDs), and 13 (−1.15 SDs) appeared to be the lowest in dominance.

Regarding emotional tone and intensity of LPFS-BF 2.0 items averaged across translations, item 2 evidenced the greatest levels of disgust (+2.41 SDs) and sadness (+2.16 SDs), item 7 was characterized by the greatest level of anger (+2.73 SDs), item 8 produced the strongest level of fear (+2.25 SDs), item 10 generated the greatest trust (+2.41 SDs), and item 12 was found to have the greatest levels of anticipation (+2.66 SDs), joy (+2.65 SDs), and surprise (+2.55 SDs). On the PDS-ICD-11, item 4 demonstrated the highest level of joy (+2.57 SDs), item 7 showed the greatest levels of trust (+2.26 SDs), item 11 was found to have the greatest levels of anticipation (+1.89 SDs), fear (+2.49 SDs), and surprise (+1.60 SDs), and item 13 was characterized by the greatest levels of anger (+2.06 SDs), disgust (+2.22 SDs), and sadness (+1.83 SDs).

Discussion

The objective of this article was to introduce and demonstrate a methodological approach that leverages NLP to quantitatively assess variability in linguistic characteristics across instrument translations and, as a secondary objective, between instruments purportedly measuring analogous latent constructs. As demonstrated by linguistically analyzing 13 translations of two measures of personality dysfunction (LPFS-BF 2.0 and PDS-ICD-11), this methodological approach appears feasible and informative, revealing multiple insights.

First, both the overall sentiment and emotional tone/intensity of instrument translations and different measures previously demonstrated to assess analogous constructs can diverge. The implications of these differences are yet to be explored but could extend to helping explain noninvariance of test translations and why ostensibly similar instruments possess dissimilar psychometrics. The introduced methodology offers an avenue for investigating these and other pertinent questions, such as which linguistic characteristics are ideal for a psychological instrument. Should test developers strive for linguistic neutrality or do tests perform better when their linguistic characteristics match their content (e.g., a hopefulness measure comprised only of items with positive overall sentiment)? Answers to these questions stand to inform test development practices and benefit both researchers and clinicians deciding between instruments.

Second, the variability in linguistic characteristics between the LPFS-BF 2.0 and PDS-ICD-11 revealed by the demonstrated methodology inspires another interesting question. These two tests make use of dissimilar structures despite assessing comparable constructs: the LPFS-BF 2.0 uses a traditional format, with each item rated using the same Likert scale, whereas the PDS-ICD-11 employs item-specific bipolar responses. Thus, at least a proportion of the linguistic differences observed between these two instruments might be due to different test structures. Past researchers have argued that a scale’s structure is a potential source of bias and can distort psychometrics (e.g., produce artificially inflated covariation within- and between-tests using similar Likert scales and anchors; Podsakoff et al., 2003). It’s reasonable to provisionally extend this argument to an instrument’s linguistic characteristics, begging the question, what role does test structure play when comparing the linguistic characteristics of different instruments and does the interaction between structure and linguistics impact a test’s psychometrics? The methodology introduced herein offers a strategy for empirically studying these questions, the answers to which might advise test developers or translators to consider different linguistic characteristics depending on an instrument’s structure.

Finally, study results showed linguistic characteristics can also profoundly vary across test items within a single instrument. Thus, the insights and questions posed above also appear applicable at the item level. Other item-specific research questions are also salient. For instance, the demonstrated methodology could be combined with item response theory methods to compare the difficulty and discrimination of semantically neutral test items, such as items 1 and 11 on the LPFS-BF 2.0, to the difficulty and discrimination of items whose sentiment matches their content, such as item 2 (overall negative sentiment; “I often think very negatively about myself.”). Significant differences would offer evidence supporting the importance of linguistic heterogeneity of test items. To this end, future studies are needed to not only illuminate the implications of item-level differences but also determine the utility—or lack thereof—in understanding these differences from both psychometric and clinical perspectives.

Beyond insights gained by examining the descriptive data derived via NLP, the current study also revealed that the consistency of linguistic characteristics across instrument translations, including translations that have been previously validated, can be exceptionally poor by conventional standards (Portney & Watkins, 2000). The relatively small number of test items examined likely lowered estimates (Lee et al., 2012), but this cannot discount the fact that the majority of estimates were remarkably low based on currently available interpretive guidelines for consistency reliability. Future efforts to establish consistency, reliability, interpretive guidelines specifically for the comparison of linguistic characteristics of instrument translations could be helpful. However, the poor uniformity across translations observed in the current study points to another need of higher priority. Multiple limitations exist when employing a single lexicon in NLP (Czarnek & Stillwell, 2022). Unfortunately, few multilingual lexicons are currently (publicly) available, limiting the utility and potential of the introduced methodology for linguistically analyzing instrument translations. As such, the development of additional robust multilingual lexicons for use in psychometric research would be an advantageous undertaking, and many approaches for doing so are available (Ghali et al., 2023).

The presented results reveal another major consideration that must be addressed before the potential of the introduced methodology can be truly realized. Negations, expressions that reverse the meaning of a statement (e.g., “not sad”), are a longstanding challenge for NLP (Morante & Blanco, 2021). The presence of negations on psychological measures has also been a longstanding psychometric challenge (Kam, 2018). Negations add complexity to a test’s linguistic characteristics and seemingly played a role in the current study (e.g., the overall positive sentiment of LPFS-BF 2.0 item 12). Different strategies for handling negation in NLP have been offered over the years (Mahany et al., 2022; Mukherjee et al., 2021), with some approaches created for specific applications (e.g., medical documents; Argüello-González et al., 2023). Although problematic in most endeavors, the nature and extent of issues created by negations for the introduced methodology requires further investigation. Of particular interest would be studies investigating whether negations in test items are deleterious, compounding, or benign to a test item’s priming effect(s). That is, does the priming effect of an item with a negation (e.g., “I am not sad”) differ from an analogous item without the negation (e.g., “I am sad”) or is the priming phenomenon evoked by individual terms regardless of context and qualifiers? Assuming the former, efforts to develop negation recognition and adjustment of algorithms specific to psychological tests would be vital.

Conclusion

The methodology introduced in this article offers a promising strategy for leveraging NLP to linguistically compare measures designed to assess analogous constructs and analyze test instrument translations to better understand translation quality. However, certain limitations remain and must be addressed before the presented methodology can be implemented with greater validity, utility, and confidence. Based on initial findings, the proposed strategies for improving the introduced methodology seem to be a worthwhile undertaking.

Supplemental Material

sj-docx-1-asm-10.1177_10731911251326371 – Supplemental material for Leveraging Artificial Intelligence to Linguistically Compare Test Translations

Supplemental material, sj-docx-1-asm-10.1177_10731911251326371 for Leveraging Artificial Intelligence to Linguistically Compare Test Translations by Adam P. Natoli in Assessment

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Preregistered

This study was not preregistered.

ORCID iD

Adam P. Natoli

Data Availability Statement

Data and analytic code are available at

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Argüello-González

Aquino-Esperanza

Salvador

Bretón-Romero

Del Río-Bermudez

Tello

Menke

(2023). Negation recognition in clinical natural language processing using a combination of the NegEx algorithm and a convolutional neural network. BMC Medical Informatics and Decision Making, 23, 216.

Bach

Brown

T. A.

Mulder

R. T.

Newton-Howes

Simonsen

Sellbom

(2021). Development and initial evaluation of the ICD-11 personality disorder severity scale: PDS-ICD-11. Personality and Mental Health, 15(3), 223–236.

Bargh

J. A.

Chartrand

T. L.

(2000). The mind in the middle: A practical guide to priming and automaticity research. In Reis

H. T.

Judd

C. M.

(Eds.), Handbook of research methods in social and personality psychology (pp. 253–285). Cambridge University Press.

Bartz

J. A.

Lydon

J. E.

(2004). Close relationships and the working self-concept: Implicit and explicit effects of priming attachment on agency and communion. Personality and Social Psychology Bulletin, 2004, 1389–1401.

Combaluzier

Gouvernet

Auvage

Bourgoise

Murphy

(2023). Le modèle alternatif des troubles de la personnalité en population française : évaluation avec des outils brefs. [The alternative model of personality disorders among the French population: Assessment with brief tools]. L’Encephale, 49(5), 496–503.

Cottin

Muzard

Hernández

Mundt

Behn

. (in preparation). Validation of the Spanish translation of the LPFS-BF 2.0 in a Chilean community and clinical sample.

Crystal

(1981). Clinical linguistics. Disorders of human communication (Vol. 3). Springer.

Czarnek

Stillwell

(2022). Two is better than one: Using a single emotion lexicon can lead to unreliable conclusions. PLoS One, 17(10), e0275910.

Elgendi

Kumar

Barbic

Howard

Abbott

Cichocki

(2018). Subliminal Priming-State of the Art and Future Perspectives. Behavioral Sciences, 8(6), 54.

10.

Ghali

J. P. E.

Inuzuka

Shima

Moriyama

Mutoh

(2023). Automating lexicon generation: A comprehensive review of alternative approaches. Procedia Computer Science, 225, 1142–1150. https://doi.org/10.1016/j.procs.2023.10.102

11.

Gritti

E. S.

De Carli

Hutsebaut

Simonelli

Zimmermann

. (in preparation). Cross-cultural adaptation of the Personality Functioning Scale- Brief Form 2.0 (LPFS-BF 2.0): Temporal stability and construct validity of the Italian version.

12.

Gutiérrez

Aluja

Rodríguez

Gárriz

Peri

J. M.

Gallart

Calvo

Ferrer

Gutiérrez-Zotes

Soler

Pascual

J. C.

(2023). Severity in the ICD-11 personality disorder model: Evaluation in a Spanish mixed sample. Frontiers in Psychiatry, 13, 1015489.

13.

Jockers

M. L.

(2015). Syuzhet: Extract sentiment and plot arcs from text. https://github.com/mjockers/syuzhet

14.

Kam

C. C. S.

(2018). Why do we still have an impoverished understanding of the item wording effect? An empirical examination. Sociological Methods & Research, 47(3), 574–597.

15.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

16.

Krippendorff

(2019). Content analysis. Sage. https://doi.org/10.4135/9781071878781

17.

Le Corff

Aluja

Rossi

Lapalme

Forget

García

L. F.

Rolland

J. P

. (2022). Construct Validity of the Dutch, English, French, and Spanish LPFS-BF 2.0: Measurement Invariance Across Language and Gender and Criterion Validity. Journal of personality disorders, 36(6), 662–679. https://doi.org/10.1521/pedi.2022.36.6.662

18.

Lee

K. M.

Lee

Chung

C. Y.

Ahn

Sung

K. H.

Kim

T. W.

Lee

H. J.

Park

M. S.

(2012). Pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research. Clinics in Orthopedic Surgery, 4(2), 149–155.

19.

Mahany

Khaled

Elmitwally

Aljohani

Ghoniemy

(2022). Negation and Speculation in NLP: A Survey, Corpora, Methods, and Applications. Applied Sciences, 12(10), 5209. https://doi.org/10.3390/app12105209

20.

Meyer

D. E.

Schvaneveldt

R. W.

(1971). Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227.

21.

Mohammad

S. M.

(2018a). Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.

22.

Mohammad

S. M.

(2018b). Word Affect Intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan.

23.

Mohammad

S. M.

(2020). Practical and ethical considerations in the effective use of emotion and sentiment lexicons. arXiv preprint. https://doi.org/10.48550/arXiv.2011.03492.

24.

Mohammad

S. M.

Turney

P. D.

(2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465.

25.

Molden

D. C.

(2014). Understanding priming effects in social psychology: An overview and integration. Social Cognition, 32(Supplement), 243–249.

26.

Morante

Blanco

(2021). Recent advances in processing negation. Natural Language Engineering, 27(2), 121–130. https://doi.org/10.1017/S1351324920000534

27.

Mukherjee

Badr

Doppalapudi

Srinivasan

S. M.

Sangwan

R. S.

Sharma

(2021). Effect of negation in sentences on sentiment analysis and polarity detection. Procedia Computer Science, 185, 370–379. https://doi.org/10.1016/j.procs.2021.05.038

28.

Natoli

A. P.

Bach

Behn

Cottin Arredondo

Gritti

E. S.

Hutsebaut

Lamba

Le Corff

Zimmermann

Lapalme

(2022). Multi-national evaluation of the measurement invariance of the Level of Personality Functioning Scale—Brief Form 2.0: Comparison of student and community samples across seven countries. Psychological Assessment, 34(12), 1112–1125. https://doi.org/10.1037/pas0001176

29.

Natoli

A. P.

Rodriguez

C. M.

(2024). A new performance-based measure of personality functioning impairment: Development and preliminary evaluation of reliability and validity. Discover Mental Health, 4(1), 6. https://doi.org/10.1007/s44192-024-00059-4

30.

Oitsalu

M. L.

Kreegipuu

Hutsebaut

(2022). Psychometric evaluation of the Estonian version of the semi-structured interview for personality functioning DSM-5 (STiP-5.1). Borderline Personality Disorder and Emotion Dysregulation, 9(1), 28.

31.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879

32.

Portney

L.G.

Watkins

M.P.

(2000). Foundations of clinical research: Applications to practice (2nd ed.). Prentice Hall Health.

33.

Proutskova

Gruszczynski

(2007). Kripp.boot: An R Package for Performing Bootstrap Replicates of Krippendorff’s alpha on Intercoder Reliability Data. R package version 1.0.0. https://github.com/MikeGruz/kripp.boot

34.

Regmi

Naidoo

Pilkington

(2010). Understanding the Processes of Translation and Transliteration in Qualitative Research. International Journal of Qualitative Methods, 9, 16.

35.

Revelle

(2024). psych: Procedures for psychological, psychometric, and personality research. R package version 2.4.6, https://CRAN.R-project.org/package=psych

36.

Schwarz

(1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93

37.

Sorrel

M. A.

García

L. F.

Aluja

Rolland

J. P.

Rossier

Roskam

Abad

F. J.

(2021). Cross-cultural measurement invariance in the Personality Inventory for DSM-5. Psychiatry Research, 304, 114134. https://doi.org/10.1016/j.psychres.2021.114134

38.

Spitzer

Müller

Kerber

Hutsebaut

Brähler

Zimmermann

(2021). Die deutsche Version der Level of Personality Functioning Scale-Brief Form 2.0 (LPFS-BF): Faktorenstruktur, konvergente Validität und Normwerte in der Allgemeinbevölkerung. [The German version of the Level of Personality Functioning Scale-Brief Form 2.0 (LPFS-BF): Latent structure, convergent validity and norm values in the general population]. Psychotherapie, Psychosomatik, Medizinische Psychologie, 71(7), 284–293.

39.

Stapel

D. A.

Blanton

(2004). From seeing to being: Subliminal social comparisons affect implicit and explicit self-evaluations. Journal of Personality and Social Psychology, 87, 468.

40.

Stapel

D. A.

Koomen

(2000). Distinctness of others, mutability of selves: Their impact on self-evaluations. Journal of Personality and Social Psychology, 79, 1068–1087.

41.

Stapel

D. A.

Koomen

(2001). I, we, and the effects of others on me: How self-construal level moderates social comparison effects. Journal of Personality and Social Psychology, 80, 766–781.

42.

Weekers

L. C.

Hutsebaut

Kamphuis

J. H.

(2019). The level of personality functioning scale-brief form 2.0: Update of a brief instrument for assessing level of personality functioning. Personality and Mental Health, 13(1), 3–14. https://doi.org/10.1002/pmh.1434

43.

Weekers

L. C.

Sellbom

Hutsebaut

Simonsen

Bach

(2023). Normative data for the LPFS-BF 2.0 derived from the Danish general population and relationship with psychosocial impairment. Personality and Mental Health, 17(2), 157–164.

44.

Zimmermann

Müller

Bach

Hutsebaut

Hummelen

Fischer

(2020). A common metric for self-reported severity of personality disorder. Psychopathology, 53(3), 161–171. https://doi.org/10.1159/000507377

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB