A Multimodal Item Response Modeling for Personality Assessment in Organizational Research

Multimodal Process Data

As one of the most accessible and widely used sources of process data, response times (RTs) serve as a valuable indicator for detecting aberrant responding (Ulitzsch et al., 2020a, 2020b, 2022), identifying response styles (Henninger & Plieninger, 2021), and predicting preference strength (Konovalov & Krajbich, 2019). Specifically, RTs have been shown to be strongly associated with response uncertainty (Meng et al., 2013; Uggeldahl et al., 2016). The time required to respond is influenced by the individual's level of uncertainty: greater hesitation leads to longer latencies, whereas greater decisiveness leads to shorter latencies. Whereas response scores indicate what was selected, response time reflects how decisively the choice was made. Analysis of RT data can improve the differentiation of individuals across trait levels (Guo et al., 2024). A substantial body of research further indicates that jointly modeling response content and response times significantly improves the accuracy of latent trait estimation (Man et al., 2022; Tian et al., 2023).

Eye-tracking data is among the most widely used and important measures in psychological research. Although eye-tracking data collection remains predominantly lab-based, technological advances are increasing its accessibility and practicality. Apple's ARKit provides built-in eye-tracking capabilities on iOS devices, enabling researchers to use native eye-tracking functions on iPads without requiring additional specialized hardware (Anisimov et al., 2021; Taore et al., 2025). Scalable online research platforms, such as jsPsych internationally (Anwyl-Irvine et al., 2021) and Credamo in China (Zhang et al., 2023), now allow researchers to conduct remote eye-tracking studies using participants’ standard webcams. These platforms employ computer vision models to estimate gaze points, thereby making large-scale, cost-effective behavioral data collection more feasible. Meißner and Oll (2019) also emphasize the timeliness and potential of eye-tracking in organizational science, highlighting: (a) technological advances that have lowered costs and improved usability; (b) the growing importance of attention and information processing in today's information-rich organizational environments; (c) the ability of eye-tracking to provide real-time, objective behavioral data that complements traditional self-report methods.

Commonly used eye-tracking indicators include fixation count (FC), which measures the number of times an individual fixates on a location; saccades, which capture changes in fixation position; regression count, which indicates the number of times a participant returns their gaze to a specific target; and fixation duration, which quantifies how long the eyes remain still at a location. Among these indicators, FC is the most commonly used, as it may reflect the intensity of attention allocated to a targeted visual region (Zhan et al., 2022). In survey research, eye-tracking data have been used to examine questionnaire completion processes, for example: to investigate the relationship between eye movement measures and response consistency (Chauliac et al., 2020); to examine participants’ cognitive strategies, including how they interpret items, question stems, and response options (Margot et al., 2023); and to assess cognitive load(Kosch et al., 2018) and monitor attention (Michinov et al., 2015; Skinner et al., 2018).

Visual fixations are also employed to examine the decision-making process (Fisher, 2021; Krajbich et al., 2010; Uggeldahl et al., 2016; Van Loo et al., 2018). According to the attentional drift-diffusion model (aDDM; Krajbich & Rangel, 2011; Krajbich et al., 2012; Liu et al., 2023), when the difference between two options is small, more fixations are typically needed to accumulate sufficient evidence to reach the decision threshold and make a choice. Conversely, when the difference between the options is large, the decision-maker can reach the threshold with fewer fixations and make a decision more quickly. In other words, the number of fixations during item processing increases with rising uncertainty.

In summary, response time (RT) and fixation count (FC), as two types of multimodal process data, serve as key indicators of response uncertainty. When an individual is highly certain about how to respond to an item, the difficulty of selecting among alternatives is low; in this case, RTs are short, and the number of fixations (FC) tends to decrease. Conversely, if an individual perceives two or more alternatives as plausible, both RT and FC increase relative to the previous scenario.

Why Is Multimodal Assessment Needed in Personality Assessment?

In organizational research, noncognitive attributes, such as personality, leadership styles, self-efficacy, and job satisfaction, play a crucial role in guiding high-stakes decisions, including hiring, promotion, and personnel development (Al-Malki & Juan, 2018; Bakker et al., 2012; Bin & Shmailan, 2015; Judge et al., 2007). Measuring these constructs is critical for organizational decision-making. Self-report questionnaires remain the primary tool for their assessment and are widely considered efficient, cost-effective, and easy to administer.

Completing questionnaires is often regarded as a simple task, requiring participants to respond to each item sequentially. For each item, respondents must interpret the question and reflect on it using their own judgment. The Cognitive Aspects of Survey Methodology (CASM) movement emphasized that understanding individual response processes is essential for evaluating questionnaire validity and identifying potential sources of error (Fowler, 2013). The key assumption is that respondents’ cognitive processes underlie their survey responses (Schwarz, 2007; Tourangeau, 2003). The response process is not merely a mechanical selection of responses but a complex sequence of cognitive operations involving several stages, including item comprehension, retrieval of relevant information from memory, judgment formation, and mapping these judgments onto overt responses (Margot et al., 2023). The complexity of the response process cannot be fully captured by a single test score, which reflects only the final outcome of cognitive operations.

In cognitive and achievement assessment, the scope of measurable data has substantially expanded. These data include not only traditional response outcomes but also various process-related indicators. The technology-enhanced assessment (TEA; Sweeney et al., 2017; Webb & Gibson, 2015) system employs built-in sensors to record learners’ real-time psychological and biological responses (e.g., eye-trackers, motion detectors, heart rate monitors), enabling educators to monitor learning and adjust instruction accordingly (Man et al., 2022). With technological advancements, collateral process data collection has become routine in large-scale computer-based assessments, such as the Program for International Student Assessment (PISA). Since 2015, the PISA test has been administered on computers, with all student–computer interactions logged, enabling researchers to examine test-takers’ behaviors in greater detail (Ivanova et al., 2020). Compared with unimodal models that rely solely on response outcomes, multimodal data introduce additional informational constraints in parameter estimation, thereby improving the precision of latent trait estimation (e.g., Liang et al., 2023; Tian et al., 2023; Zhan, 2022; Zhan et al., 2022).

Although multimodal data have shown considerable utility in cognitive and achievement testing, their application in noncognitive assessment remains limited. Since measuring noncognitive traits also involves complex internal psychological processes, extending multimodal research approaches to this domain has become an inevitable trend. This approach shifts the focus from simple response outcomes to a more comprehensive capture of cognitive and behavioral processes, thereby enhancing the precision of noncognitive assessment.

Existing Approaches and Limitations

A review of prior literature on the joint modeling of examinees’ response data and multimodal process data indicates that hierarchical framework modeling, as proposed by Van der Linden (2007), is predominantly used for the combined analysis of responses and response times (RT). However, hierarchical framework modeling also presents several limitations. Ranger (2013) highlighted that the main theoretical limitation of hierarchical framework modeling is that incorporating RT data improves the estimation accuracy of latent ability only when the correlation between latent trait and latent processing speed is nonzero. Moreover, empirical studies have reported a low correlation between latent ability and latent processing speed (Man et al., 2022). According to Bolsinova and Tijmstra (2018), joint hierarchical framework models fall short in fully leveraging RT data because they assume that RTs are influenced solely by latent processing speed and not by latent traits. Crucially, hierarchical framework models do not account for how respondents’ cognitive processes generate the multimodal data observed during task performance.

Cross-loading modeling (Bolsinova & Tijmstra, 2018; Meng et al., 2013; Molenaar et al., 2015) can be viewed as an extension of the joint hierarchical modeling framework. In principle, including cross-loading terms allows auxiliary process indicators to directly inform the estimation of the focal latent trait, rather than being treated solely as correlated secondary outcomes. Regardless of the correlation between the latent trait and latent variables derived from multimodal data, these data can contribute to the estimation of latent trait parameters, thereby enhancing estimation accuracy.

Ferrando and Lorenzo-Seva (2007) initially proposed a response time model with cross-loading terms for dichotomous personality test items. Meng et al. (2013) later extended this model to polytomous IRT and argued that the difficulty of responding to an item reflects uncertainty in the decision-making process, with differences among response probabilities serving as key factors in determining item difficulty. Guo et al. (2024) introduced response uncertainty as a cross-loading term to link the latent trait with response time and found that this approach improved latent trait estimation accuracy in forced-choice assessments.

Therefore, we focus on personality assessment and develop a multimodal polytomous IRT model that integrates response outcomes, RT, and FC, with response uncertainty—measured via information entropy (see the section “Response Uncertainty in Personality Assessment”)—serving as a cross-loading indicator within a joint–cross-loading modeling framework.

Aims and Highlights of the Present Study

In both organizational and broader behavioral research, there have been strong calls for increased use of behavioral data and the integration of multiple methodological sources. With the growing prevalence of computer- and web-based assessments, the collection of process data has become routine. For example, the development of TEAs, together with advances in multimodal data acquisition, has emphasized the need for corresponding improvements in modeling techniques that can leverage these data. Despite the richness and diversity of these data sources, their application in organizational research remains limited. Most traditional IRT-based assessments provide limited information and fail to fully capture individual differences in latent traits, potentially constraining informed organizational decision-making.

To address this gap, this study employs a cross-loading modeling framework to jointly analyze multimodal process data. The model uses cross-loading terms to explain how response uncertainty arising from person–item interactions produces variability in multimodal data, and how these multimodal indicators, in turn, inform the cross-loading terms to enhance the precision of latent trait parameter estimation. As response uncertainty increases, item difficulty also rises, resulting in longer response times and higher visual fixation counts. This approach not only enables a more precise differentiation of individual latent traits but also establishes a methodological foundation for utilizing multimodal behavioral data in organizational decision-making contexts.

The remainder of this paper is organized as follows. First, cross-loading terms within the cross-loading modeling framework are introduced. Next, the three sub-models of the proposed framework—the IRT model, the response time model, and the visual fixation counts (FC) model—are presented, along with the rationale for their integration. Then, a simulation study was conducted to evaluate the feasibility of the proposed model, followed by a real data analysis to demonstrate the utility of multimodal data in practical applications. Finally, the study discusses its implications, limitations, and proposed directions for future research.

Response Uncertainty in Personality Assessment

In the present study, information entropy was introduced to more accurately quantify response uncertainty during the decision-making process. Differences among response probabilities reflect the uncertainty in responding to an item (Meng et al., 2013). In information theory, information entropy measures the uncertainty associated with a random variable (Shannon, 1948): the more uniform the distribution of outcomes, the greater the uncertainty and the higher the entropy; conversely, if an outcome is highly certain, the entropy is low. Therefore, to more effectively capture differences among response probabilities and quantify response uncertainty during the answering process, information entropy was introduced as an index of uncertainty. For K response categories, with integer values ranging from 0, 1, 2, …, K–1, information entropy can be defined as:

H i j = − ∑ k = 0 K − 1 P ( y j = k θ i ) log P ( y j = k θ i ) ,

(1)

Here, H i j represents information entropy, and P ( y j = k θ i ) denotes the probability of selecting a particular response category k, as estimated by the IRT model. This uncertainty can manifest in multimodal data recorded during responding: low response uncertainty is typically associated with shorter RTs and fewer fixations, whereas high uncertainty corresponds to longer RTs and more fixations.

When information entropy approaches its maximum, log(K), meaning that the probabilities of all response categories are equal, i.e., P ( y j = 0 θ i ) = P ( y j = 1 θ i ) = … = P ( y j = K − 1 θ i ) = 1 / K , the examinee is highly indecisive and faces maximal difficulty in selecting an option, as all K alternatives are equally probable. Consequently, both RT and FC are likely to increase due to this heightened uncertainty.

When information entropy approaches its minimum, 0, meaning that the probability of selecting one response category is 100%, the examinee is certain which response to select, and choosing among alternatives involves minimal difficulty. Consequently, a rapid response with fewer fixations is expected.

The Framework of the Proposed Modeling (Multimodal-IRTM)

In this study, we employed a joint cross-loading modeling approach to integrate multiple types of data sources. Specifically, the joint model, referred to as multimodal-IRTM, consists of three components: Sub-Model I describes the response score data using an IRT model (e.g., the Graded Response Model; Samejima, 1969); Sub-Model II describes the response time data using the lognormal RT (LRT) model (Van der Linden, 2007); and Sub-Model III describes visual FC data using the Negative Binomial Fixation (NBF) model (Man & Harring, 2019). We then incorporate the introduced information entropy (see equation (1)) as a cross-loading term to link the three sub-models and to associate individuals’ latent traits with their response times and visual FCs. Further details of the proposed multimodal modeling framework are illustrated in Figure 1.

OPEN IN VIEWER

Figure 1.

A graphical representation of the proposed multimodal-IRTM.

For the Graded Response Model (Samejima, 1969), let there be i = 1, 2…, N individuals, j = 1, 2…, J items, and K response categories (integer values from 0,1, 2…, K − 1 ) . The item parameter a j is the slope parameter, b j ( b j = b j 1 , b j 2 , b j K − 1 ) are a set of strictly ordered threshold parameters. Let the cumulative category response probabilities be:

P ( y i j ≥ 1 ∣ θ i ) = 1 1 + e − 1.7 a j ( θ i − b j 1 ) , ⋮ P ( y i j ≥ K − 1 ∣ θ i ) = 1 1 + e − 1.7 a j ( θ i − b j K − 1 ) .

(2)

Then the probability that examinee i attains exactly score k on item j :

P ( y i j = k ∣ θ i ) = P ( y i j ≥ k ∣ θ i ) − P ( y i j ≥ k + 1 ∣ θ i ) ,

(3)

The variation in respondents’ observed RT when answering an item is primarily influenced by three factors: (a) the information processing speed at which respondents work during the test (i.e., the person main effect), (b) the amount of labor required by the item (i.e., the item main effect), and (c) the person–item interaction effect (Thissen, 1983). Information entropy can be viewed as representing the person–item interaction term. The positive skewness and frequent large outliers inherent in RT data make the lognormal RT model a preferred choice for modeling such data (Van der Linden, 2007). Accordingly, the lognormal RT (LRT) model, incorporating information entropy, can be expressed as follows:

T i j ∼ f ( t i j ; τ i , θ i , ω j , ξ j , β j ) = ω j t i j 2 π exp ( − ω j 2 2 ( log t i j − ( ξ j + β j H i j − τ i ) ) 2 ) ,

(4)

log ( T i j ) ∼ N ( ( ξ j + β j H i j − τ i ) , σ j 2 ) ,

(5)

For visual FC data, we adopt a parallel decomposition framework similar to that used for response times, in which FCs reflect person effects, item effects, and person–item interactions. Information entropy is incorporated to quantify response uncertainty. Following the approach of Man and Harring (2019), the Negative Binomial Fixation (NBF) model for FCs, incorporating information entropy, can be expressed as follows:

V i j ∼ f ( V i j ; ϵ i , θ i , h j , m j ) = Γ ( V i j + h j ) V i j ! Γ ( h j ) ( h j exp ( ϵ i + λ j H i j + m j ) + h j ) h j ( exp ( ϵ i + λ j H i j + m j ) exp ( ϵ i + λ j H i j + m j ) + h j ) V i j ,

(6)

V i j ∼ N B ( exp ( m j + λ j H i j + ϵ i ) , d j − 2 ) ,

(7)

Where V i j be the observed FC of person i to item j, ϵ i is the latent visual engagement of person i that denotes the overall test engagement level for examinee i; m j is the visual-intensity of item j that represents the averaged amount of cognitive engagement for respondent to finish answering item j; h j is the dispersion or shape parameter of the expectation of the random variable V i j ; d j is the visual discrimination parameter, reflecting dispersion of the FCs on item j ; λ j is the regression coefficient, which determines the influence of Information entropy H i j on FCs.

Finally, suppose that a correlation exists between participants’ latent processing speed and latent visual engagement. In this case, the joint distribution of latent processing speed and latent visual engagement is assumed to follow a multivariate normal distribution:

Θ i = ( τ i ϵ i ) ∼ M V N ( ( μ τ μ ϵ ) , Σ person ) , Σ person = ( σ τ 2 σ τ ϵ σ ϵ 2 )

(8)

A multivariate normal distribution is also assumed for the item parameters, such that:

Ξ j = ( b j k ξ j m j ) ∼ M V N ( ( μ b k μ ξ μ m ) , Σ i t e m ) , Σ i t e m = ( σ b k 2 σ b k ξ σ ξ 2 σ b k m σ ξ m σ m 2 )

(9)

The log-likelihood of the proposed model can be expressed as：

ℓ ( θ , Θ , Ξ ∣ y , log ( t ) , V ) = ∑ i = 1 I ∑ j = 1 J log p ( y i j , log ( t i j ) , V i j ∣ θ i , Θ i , Ξ j ) = ∑ i = 1 I ∑ j = 1 J [ l o g p ( y i j ∣ θ i , Ξ j ) + l o g p ( l o g ( t i j ) ∣ θ i , Θ i , Ξ j ) + l o g p ( V i j ∣ θ i , Θ i , Ξ j ) ]

(10)

Bayesian Parameter Estimation

Given the complexity of the proposed model, the parameters of the multimodal-IRTM model were estimated using a Bayesian approach with MCMC sampling, implemented in R via the nimble package. The prior are set following those in Man and Harring (2019), Man et al. (2022), Zhan et al. (2022), Guo et al. (2024), and Zhou and Guo (2026).

First, Y i j , log ( T i j ) and V i j are assumed to be distributed as:

Y i j ∼ Categorical ( P i j ) ,

log ( T i j ) ∼ N ( ( ξ j + β j H i j − τ i ) , σ j 2 ) ,

V i j ∼ N B ( exp ( m j + λ j H i j + ϵ i ) , d j − 2 ) ,

Due to the assumed correlation between latent processing speed and latent visual engagement, the priors for the person parameters of the LRT model and the NBF model are assumed from a multivariate normal distribution, that is:

( τ i ϵ i ) ∼ M V N ( ( 0 0 ) , Σ person ) .

Furthermore, hyper priors for the covariance matrix of the multivariate normal distribution are specified as Σ person ∼ Inverse - Wishart ( R , 2 ) , where R is a two-dimensional identity matrix.

For the GRM model, the person parameter θ is set as standard normal distribution, θ ∼ N ( 0 , 1 ) .

In addition, the priors of item parameters are specified as

a ∼ Lognormal ( 0 , 0.25 ) ,

( b j k ξ j m j ) ∼ M V N ( ( 0 1 2 ) , Σ i t e m ) ,

and b j 1 < b j 2 < b j 3 < b j 4 ,

Σ item ∼ Inverse - Wishart ( R , 6 ) ,

σ j ∼ InvGamma ( 1 , 1 ) ,

h j ∼ InvGamma ( 1 , 1 )

Design and Data Generation

A simulation design was adopted to investigate both parameter recovery and comparative model performance. As demonstrated in prior simulation studies (e.g., Man et al., 2022; Meng et al., 2013; Zhan et al., 2022), the manipulated factors included test length (I = 15, 30), sample size (N = 200, 500, 1000), regression coefficients of information entropy (RC: β j and λ j 0, 0.5, 1, 1.5), and analysis model (GRM vs. multimodal-IRTM). Each condition was replicated 50 times. The number of response categories was set to K = 5, reflecting the widespread use of five-point Likert scales in personality research. When the regression coefficients associated with information entropy were fixed at 0, the multimodal-IRTM model reduced to a combination of the separate GRM, LRT, and NBF models, thereby allowing a direct comparison of whether RT and FC contribute to latent trait estimation.

The latent processing speed τ i and the latent visual engagement ϵ i are generated from a multivariate normal distribution, as follow:

( τ i ϵ i ) ∼ M V N ( ( 0 0 ) ( 0.25 ρ τ ϵ × 0.25 0.25 ) ) ,

The latent personality trait θ was drawn from a standard normal distribution N ( 0 , 1 ) . The correlations between latent trait θ and latent processing speed τ, as well as between and latent visual engagement, were not explicitly specified because the present study adopted a joint cross-loading modeling framework, in which the relationships between θ and τ, and between θ and ε , were implicitly captured by the cross-loading terms (Zhan et al., 2022).

The item ordered threshold b j k , the item time intensity ξ j , and the item visual intensity m j are generated from a multivariate normal distribution, as follow:

( b j k ξ j m j ) ∼ M V N ( ( 0 1 2 ) , ( 1 0.15 0.25 0.15 0.125 0.25 ) ) ,

The remaining parameters of the GRM model, the item slope a j , was drawn from log-normal distribution i.e., l o g ( a j ) ∼ N ( 0 , 0.25 2 ) (Gorney, 2024). The dispersion or shape parameter of the NBF model, h j , was generated from InvGamma (2,6). For simplicity, the time-precision parameter ω j was set to 2, i.e., ω j = 1 / σ j = 2 ，then the standard deviation of the error term σ j is 0.5 (Guo et al., 2024).

Finally, item scores, response times, and visual FCs were simulated using the GRM model (equations (2) and (3)), the LRT model (equation (5)), and the negative binomial model (equation (7)), respectively.

Analysis

Considering the complexity of the proposed model, this study utilizes a cutting-edge R package “nimble” (de Valpine et al., 2017) for Bayesian inference. Nimble is an R package for programming with BUGS models using syntax similar to WinBUGS and JAGS, but with more flexibility in defining the models and algorithms. Users can operate from within R, and nimble will generate the C++ code for faster computation. For each condition, parameters were estimated using the MCMC algorithm with a single chain, a thinning interval of 1, 10,000 iterations, and a burn-in period of 5,000, after which the remaining 5,000 iterations were used for model parameter inference.

To investigate the accuracy of parameter estimation, we computed the BIAS and root mean square error (RMSE) as

BIAS ( x ^ ) = ∑ r = 1 R ∑ n = 1 N ( x − x ^ ) R ⋅ N ,

RMSE ( x ^ ) = ∑ r = 1 R ∑ n = 1 N ( x − x ^ ) 2 R ⋅ N ,

Additionally, correlations (COR) between estimated and true parameter values were evaluated to assess parameter recovery accuracy.

Results

Table 1 summarizes the recovery results for the latent trait ( θ ) under the GRM and the multimodal-IRTM. For the multimodal-IRTM, BIAS values were close to zero, the maximum RMSE was approximately 0.30, and COR values exceeded 0.95 across all conditions. Overall, parameter recovery, as indicated by RMSE and COR, improved under more favorable design conditions. Both test length and sample size enhanced estimation precision; specifically, longer tests and larger samples resulted in lower RMSE and higher COR values. In addition, larger RC were associated with improved parameter recovery.

OPEN IN VIEWER

Table 1.

Latent Traits Parameter Recovery Between GRM and Multimodal-IRTM.

R C	N	I	GRM			Multimodal-IRTM
			BIAS	RMSE	COR	BIAS	RMSE	COR
0	200	15	0.003	0.304	0.955	−0.005	0.313	0.953
		30	−0.001	0.226	0.976	0.000	0.237	0.975
	500	15	−0.003	0.296	0.956	−0.002	0.298	0.956
		30	−0.005	0.217	0.978	−0.009	0.219	0.977
	1000	15	0.004	0.297	0.956	0.005	0.298	0.956
		30	0.007	0.215	0.978	0.003	0.217	0.978
0.5	200	15	0.004	0.305	0.955	−0.005	0.291	0.960
		30	−0.002	0.226	0.976	−0.002	0.217	0.979
	500	15	−0.002	0.296	0.957	−0.001	0.280	0.962
		30	0.003	0.217	0.977	−0.005	0.206	0.981
	1000	15	−0.001	0.298	0.956	0.001	0.280	0.961
		30	0.004	0.213	0.978	0.004	0.199	0.981
1	200	15	−0.006	0.304	0.955	−0.006	0.249	0.972
		30	−0.008	0.227	0.976	−0.003	0.190	0.986
	500	15	−0.002	0.297	0.956	0.003	0.238	0.973
		30	0.004	0.217	0.978	−0.002	0.170	0.987
	1000	15	−0.001	0.297	0.956	0.001	0.234	0.973
		30	0.004	0.213	0.978	0.001	0.166	0.987
1.5	200	15	−0.003	0.305	0.954	−0.021	0.211	0.981
		30	−0.007	0.226	0.976	−0.015	0.162	0.991
	500	15	−0.008	0.297	0.956	−0.018	0.193	0.983
		30	0.002	0.216	0.978	−0.004	0.141	0.992
	1000	15	0.001	0.295	0.957	0.001	0.192	0.983
		30	0.007	0.213	0.978	0.002	0.137	0.992

Note: N = sample size; I = test length; RC = the regression coefficients of information entropy.

In comparing the multimodal-IRTM with the GRM, the multimodal-IRTM demonstrated more accurate latent trait recovery in most conditions. When the RC were equal to zero, the RMSE and COR for θ estimated by the multimodal-IRTM were slightly inferior to those obtained from the GRM. However, when RC exceeded zero, the multimodal-IRTM outperformed the GRM in estimating θ , and estimation accuracy increased as the RC became larger. These findings suggest that incorporating multimodal data can substantially enhance the estimation of latent trait.

Figure 2 and Figure 3 report the recovery performance of item parameters (i.e., a , b 1 , b 2 , b 3 , b 4 ) for the multimodal-IRTM and the GRM under the 15-item condition. For the boxplots, each box summarizes results across 50 independent replications under the corresponding experimental condition. The corresponding results for the 30-item condition, which were similar to those observed in the 15-item condition, are provided in Part 1 of the Supplemental Document. In addition, to further enhance transparency, the in Part 1 of the Supplemental Document also includes the mean Bias, RMSE, and COR for item parameter recovery across all simulation conditions.

OPEN IN VIEWER

Figure 2.

Boxplots of RMSE for item parameter recovery under the 15-item condition.

OPEN IN VIEWER

Figure 3.

Boxplots of COR for item parameter recovery under the 15-item condition.

For the multimodal-IRTM, both item parameters were recovered with good accuracy, with the vast majority of RMSE values below 0.4 and COR above 0.8. Moreover, increases in sample size were associated with improved parameter recovery, reflected in lower RMSE and higher COR values at larger sample sizes. The estimation results for the four threshold parameters indicate that b 2 and b 3 were recovered more accurately than b 1 and b 4 , as evidenced by smaller RMSE values, larger COR coefficients and reduced dispersion in the boxplots. This pattern may be explained by the assumption that θ i ∼ N ( 0 , 1 ) , under which the probabilities of endorsing extreme categories are low, resulting in limited information for estimating the corresponding threshold parameters. In contrast, the intermediate thresholds correspond to categories located in the central region of the ability distribution, where most observations occur, thereby supporting more accurate and stable estimation. This pattern was also observed in the empirical study, in which the posterior standard deviations of b 2 and b 3 were smaller than those of b 1 and b 4 .

Figures 2 and 3 clearly show that the item parameter recovery of the GRM is largely invariant to changes in RC. In contrast, for the multimodal-IRTM, RMSE decreases and COR increases as RC increases, indicating an overall trend of improving estimation accuracy. In other words, as RC increases, the advantage of the multimodal-IRTM over the GRM becomes increasingly pronounced. When RC was equal to zero, similar to the results for θ , the multimodal-IRTM exhibited very slightly higher RMSE and lower COR than the GRM. When RC was 0.5, some item parameter estimates from the multimodal-IRTM demonstrated similar RMSE values but higher COR than those from the GRM, while others yielded lower RMSE and higher COR. When RC was 1 and 1.5, the multimodal-IRTM demonstrated superior item parameter recovery performance relative to the GRM, with uniformly lower RMSE and higher COR values. Overall, when RC was greater than 0, parameters estimated by the multimodal-IRTM generally outperformed those from the GRM.

When RC was equal to zero, and aside from the joint estimation of three sets of item parameters (i.e., the item threshold parameters b 1 , b 2 , b 3 , b 4 , the item time intensity parameters ξ , and the item visual intensity parameters m ) under a multivariate normal distribution, the multimodal-IRTM was effectively equivalent to an extended version of the GRM. As a result, parameter estimates from the multimodal-IRTM were slightly less precise than those from the traditional GRM. When RC exceeded zero, the multimodal-IRTM incorporated a cross-loading term for information entropy, thereby linking the latent trait with the process data and enabling the model to use this additional information to refine θ estimates and improve latent trait recovery. As RC increased, the contribution of information entropy within the LRT and NBF models became stronger, providing additional auxiliary information and further enhancing estimation accuracy.

Tables 2 and 3 present the parameter estimates of the LRT model and the NBF model within the multimodal-IRTM framework, respectively. The multimodal-IRTM demonstrated satisfactory parameter recovery across conditions; larger sample sizes were associated with improved recovery of item parameters, whereas longer tests resulted in more accurate estimation of person parameters. However, as RC increased, the RMSE for the speed parameter increased slightly, a pattern also reported by Guo et al. (2024), which may reflect additional uncertainty introduced into the response model parameters.

OPEN IN VIEWER

Table 2.

Recovery of the LRT Sub-Model Parameters of the Proposed Model.

R C	N	I	τ			ξ			σ		β
			BIAS	RMSE	COR	BIAS	RMSE	COR	BIAS	RMSE	BIAS	RMSE
0	200	15	−0.005	0.130	0.969	−0.018	0.194	0.921	−0.002	0.026	0.010	0.164
		30	−0.005	0.097	0.983	−0.026	0.185	0.933	−0.001	0.026	0.019	0.154
	500	15	0.006	0.126	0.969	0.002	0.121	0.974	−0.001	0.017	0.001	0.101
		30	0.002	0.092	0.984	0.000	0.119	0.974	−0.001	0.017	0.003	0.098
	1000	15	0.000	0.125	0.968	−0.001	0.088	0.986	0.000	0.012	0.001	0.075
		30	−0.001	0.091	0.984	−0.002	0.085	0.985	−0.001	0.012	0.001	0.071
0.5	200	15	−0.007	0.138	0.964	−0.019	0.203	0.914	−0.001	0.027	0.014	0.163
		30	0.004	0.104	0.981	−0.005	0.205	0.918	−0.003	0.027	0.011	0.161
	500	15	0.009	0.135	0.964	−0.002	0.130	0.966	−0.001	0.018	0.010	0.109
		30	0.002	0.099	0.981	0.004	0.127	0.969	−0.001	0.016	0.001	0.103
	1000	15	0.000	0.134	0.964	−0.003	0.097	0.982	0.000	0.012	0.003	0.079
		30	−0.002	0.098	0.981	−0.003	0.093	0.982	−0.001	0.012	0.002	0.077
1	200	15	−0.010	0.156	0.953	−0.088	0.243	0.890	−0.003	0.027	0.065	0.192
		30	0.003	0.115	0.975	−0.070	0.220	0.909	−0.003	0.027	0.061	0.175
	500	15	0.010	0.151	0.954	−0.027	0.159	0.956	−0.002	0.017	0.032	0.123
		30	0.001	0.108	0.977	−0.032	0.152	0.955	−0.001	0.016	0.028	0.120
	1000	15	−0.002	0.149	0.955	−0.012	0.111	0.976	−0.001	0.012	0.010	0.087
		30	−0.002	0.108	0.977	−0.018	0.111	0.975	−0.001	0.012	0.014	0.087
1.5	200	15	−0.012	0.167	0.947	−0.194	0.311	0.869	−0.003	0.028	0.148	0.245
		30	0.001	0.125	0.971	−0.163	0.278	0.891	−0.004	0.028	0.133	0.220
	500	15	0.009	0.161	0.947	−0.086	0.196	0.944	−0.002	0.018	0.076	0.154
		30	0.000	0.117	0.973	−0.076	0.176	0.949	−0.001	0.016	0.062	0.138
	1000	15	−0.001	0.161	0.947	−0.032	0.124	0.970	−0.001	0.012	0.024	0.093
		30	−0.001	0.115	0.974	−0.046	0.137	0.965	−0.001	0.012	0.037	0.105

Note: N  = sample size; J = test length; RC = the regression coefficients of information entropy; τ = latent processing speed; ξ = item time intensity; σ = item standard deviation; β = the regression coefficients of information entropy from the LRT model. COR = correlations between estimated and true item parameters. Since σ and β were fixed as constants during data generation, the COR were not provided.

OPEN IN VIEWER

Table 3.

Recovery of the NBF Sub-Model Parameters of the Proposed Model.

R C	N	I	ε			m			d		λ
			BIAS	RMSE	COR	BIAS	RMSE	COR	BIAS	RMSE	BIAS	RMSE
0	200	15	0.006	0.164	0.949	−0.061	0.247	0.878	−0.009	0.019	0.004	0.189
		30	0.004	0.124	0.972	−0.061	0.235	0.894	−0.008	0.018	0.004	0.177
	500	15	−0.002	0.164	0.948	−0.053	0.169	0.954	−0.009	0.014	0.003	0.123
		30	−0.003	0.122	0.973	−0.052	0.158	0.957	−0.009	0.015	−0.001	0.114
	1000	15	0.001	0.162	0.949	−0.052	0.144	0.963	−0.009	0.013	−0.003	0.099
		30	0.000	0.122	0.973	−0.050	0.127	0.973	−0.009	0.013	−0.001	0.088
0.5	200	15	0.007	0.152	0.954	−0.060	0.245	0.879	−0.003	0.011	0.032	0.192
		30	−0.002	0.115	0.975	−0.050	0.239	0.887	−0.003	0.011	0.031	0.187
	500	15	−0.004	0.153	0.953	−0.039	0.175	0.937	−0.003	0.008	0.021	0.136
		30	−0.002	0.110	0.976	−0.048	0.165	0.949	−0.004	0.008	0.027	0.127
	1000	15	0.001	0.152	0.954	−0.050	0.141	0.963	−0.003	0.007	0.023	0.101
		30	0.002	0.110	0.977	−0.043	0.136	0.964	−0.003	0.006	0.018	0.101
1	200	15	0.008	0.156	0.952	−0.126	0.276	0.874	−0.001	0.006	0.092	0.218
		30	0.000	0.118	0.973	−0.105	0.258	0.876	−0.001	0.007	0.082	0.203
	500	15	−0.005	0.156	0.951	−0.045	0.182	0.941	−0.001	0.004	0.037	0.144
		30	0.001	0.111	0.976	−0.061	0.179	0.942	−0.001	0.005	0.045	0.138
	1000	15	0.003	0.154	0.952	−0.049	0.146	0.962	−0.001	0.003	0.032	0.111
		30	0.002	0.111	0.976	−0.046	0.145	0.961	−0.001	0.003	0.031	0.111
1.5	200	15	0.006	0.165	0.946	−0.202	0.327	0.853	−0.001	0.004	0.158	0.259
		30	−0.001	0.124	0.970	−0.186	0.306	0.875	0.000	0.004	0.150	0.246
	500	15	−0.002	0.161	0.947	−0.103	0.213	0.929	0.000	0.003	0.083	0.162
		30	0.001	0.115	0.974	−0.093	0.200	0.935	0.000	0.002	0.071	0.157
	1000	15	0.001	0.160	0.948	−0.064	0.160	0.954	−0.001	0.002	0.049	0.119
		30	0.000	0.114	0.974	−0.064	0.158	0.954	0.000	0.002	0.050	0.122

Note: ε = latent visual engagement; m = item visual intensity; d = item visual discrimination; λ = the regression coefficients of information entropy from the NBF model. COR = correlations between estimated and true item parameters. For parameters without reported COR, the reasons are the same as described in Table 2.

As shown in Table 2, most BIAS values for the item time-intensity parameter ξ were negative, indicating that the estimated values exceeded the true values and therefore reflected overestimation. For the regression coefficients β of information entropy from the LRT model, most BIAS values were positive, indicating that the estimates were lower than the true values and thus reflected underestimation. The results in Table 3 indicate that the BIAS pattern in the NBF model was similar to that observed in the LRT model. The item visual-intensity parameter m tended to be overestimated, whereas the regression coefficients λ tended to be underestimated. This pattern is consistent with the findings of Guo et al. (2024), who reported opposite signs for the BIAS of item and regression coefficients, with one set overestimated and the other underestimated. The regression coefficients β and λ associated with information entropy were predominantly positive in BIAS, indicating that their estimates generally fell below the true values and suggesting systematic underestimation. This finding is consistent with the well-documented phenomenon that linear relationships tend to be attenuated when predictors contain measurement error.

Data Collection

To demonstrate the practical application of the proposed multimodal-IRTM model in personality assessment, we used the Chinese version of the Big Five Inventory (BFI; Li et al., 2025), whose items were selected from the International Personality Item Pool (IPIP; Goldberg, 1992). Response scores, response times, and eye-tracking data were collected using the Tobii Pro Fusion and Tobii TX300 eye trackers.

At the beginning of the questionnaire, participants were asked to provide demographic information, including age, gender (1 = male; 2 = female), and educational level (1 = Bachelor's degree; 2 = Master's degree; 3 = Doctoral degree or higher). Additionally, all participants provided informed consent prior to completing the formal tasks.

The experiment was conducted in a controlled eye-tracking laboratory setting. The study utilized a web-based program developed in Visual Studio Code, which incorporated the Big Five Personality Inventory. The program was deployed on a computer connected to a Tobii eye tracker, through which participants completed the questionnaire. During the task, the program automatically recorded participants’ response times and eye-movement data from the onset of each item to the corresponding keypress. Participants rated each item on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Prior to the formal experiment, an example item was presented to illustrate the response procedure, and participants were encouraged to seek clarification from the experimenter if needed.

Data Description

Data were collected from undergraduate students enrolled at two universities across different provinces in China. Participants with color blindness, strabismus, severe myopia, or significant interocular differences in myopia were excluded to minimize potential sources of bias. The study initially recruited a total of 262 participants. After applying quality control filters (i.e., eye-tracking data with a sampling rate below 85% (He et al., 2025) and participants who failed either of the two attention-check items), 236 participants were retained for the final analysis. Among the final sample, 53 participants were male and 183 were female. The mean age was 21.58 years (SD = 2.82).

The scale comprised a total of 62 items, including 60 items assessing the Big Five personality traits (i.e., Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness), with each trait measured by 12 items. Additionally, two attention-check items were inserted following the 20th and 40th items. In the current study, Cronbach's α coefficients for the five traits were 0.827, 0.887, 0.914, 0.917, and 0.865, respectively.

We simultaneously collected three types of data: response scores, response times, and visual FCs. Before analysis, observations with response times shorter than 0.3 s were excluded, as such rapid responses suggest that participants may not have adequately engaged with the items (Bunji & Okada, 2020; Guo et al., 2024). Response times were then converted to seconds and log-transformed to reduce skewness and limit the influence of outliers (Van der Linden, 2007). Visual FCs were computed using Tobii's default algorithm, which is based on the I-VT (Velocity-Threshold Identification) method proposed by Salvucci and Goldberg (2000).

Results

As shown in Table 4, across all analyses of the five personality dimensions, convergence diagnostics indicated satisfactory convergence for both the separate-Model and multimodal-IRTM, with at least 95% of parameter estimates in each dimension exhibiting a potential scale reduction factor (PSRF) below 1.2. Table 4 also presents model fit indices for the separate-Model and the proposed multimodal-IRTM applied to the empirical data. Across all five dimensions of the Big Five personality traits, the multimodal-IRTM produced lower LOO and WAIC values than the separate-Model, indicating superior model performance. For multimodal data, jointly modeled approaches achieve better fit than models that analyze each modality separately.

OPEN IN VIEWER

Table 4.

Model Convergence and Model Fit in Real-Data Application.

Personality factor	Separate-model			Multimodal-IRTM
	%PSRF < 1.2	LOO	WAIC	%PSRF < 1.2	LOO	WAIC
Agreeableness (A)	100%	27,151.430	24,376.459	99.9%	26,990.230	23,932.146
Conscientiousness (C)	99.2%	27,436.508	24,792.654	100%	27,264.147	24,380.491
Extraversion (E)	100%	28,178.317	25,421.454	100%	28,079.380	25,135.520
Neuroticism (N)	100%	27,941.122	25,017.553	99.6%	27,829.949	24,769.885
Openness (O)	98.5%	27,584.433	25,029.312	98.8%	27,550.598	24,806.860

Note: Separate-model = fitting three models of data with three measurement models separately; PSRF = the potential scale reduction factor.

Table 5 reports the estimates of item parameters for the three measurement models in the multimodal-IRTM corresponding to the Agreeableness dimension (similar patterns were observed for other dimensions; details are provided in Part 2 of the Supplemental Document), along with the regression coefficients β and λ for information entropy. Most parameters have posterior standard errors below 0.2, indicating stable and reliable estimates. Some item parameters, however, exhibit moderately high posterior standard errors, particularly the intercepts b 1 and b 4 . The parameters β and λ represent the regression coefficients for information entropy in the LRT and NBF models, respectively. All estimates of β and λ are positive, consistent with the hypothesis that information entropy is positively associated with RT and FC. It should be noted that the magnitudes of β and λ depend on the scales of information entropy and the log-transformed RTs and FCs; therefore, these values do not directly reflect the strength of the relationships (Guo et al., 2024).

OPEN IN VIEWER

Table 5.

Estimates of Item Parameters for the Multimodal-IRTM Model in Empirical Data (Dimension A).

Item	Response Score					Response Time		Fixation Count		β		λ
	a	b 1	b 2	b 3	b 4	ξ	σ	m	d
4	0.69 (0.06)	−3.97 (0.09)	−2.82 (0.21)	−2.05 (0.19)	1.08 (0.15)	0.35 (0.20)	0.32 (0.02)	1.38 (0.23)	0.23 (0.01)	0.92 (0.24)	1.06 (0.28)
9	0.68 (0.10)	−3.79 (0.41)	−1.02 (0.18)	0.17 (0.14)	3.21 (0.44)	0.98 (0.21)	0.32 (0.02)	2.19 (0.21)	0.20 (0.01)	0.31 (0.18)	0.31 (0.18)
13	1.46 (0.19)	−3.88 (0.22)	−1.54 (0.15)	−0.86 (0.11)	1.27 (0.12)	0.63 (0.13)	0.37 (0.02)	1.63 (0.15)	0.21 (0.01)	0.80 (0.17)	0.99 (0.20)
22	1.94 (0.21)	−3.42 (0.37)	−1.59 (0.16)	−1.20 (0.11)	0.95 (0.10)	0.46 (0.12)	0.18 (0.02)	1.48 (0.13)	0.30 (0.01)	1.48 (0.17)	1.65 (0.18)
28	0.77 (0.11)	−3.47 (0.46)	−0.58 (0.15)	0.64 (0.15)	3.09 (0.47)	0.94 (0.25)	0.33 (0.02)	2.05 (0.27)	0.20 (0.01)	0.36 (0.21)	0.47 (0.23)
33	0.89 (0.11)	−3.83 (0.27)	−2.33 (0.26)	−1.24 (0.17)	1.48 (0.18)	0.52 (0.15)	0.31 (0.02)	1.57 (0.18)	0.29 (0.01)	0.49 (0.17)	0.53 (0.21)
36	0.72 (0.12)	−2.84 (0.53)	−0.76 (0.20)	0.28 (0.13)	2.29 (0.36)	0.73 (0.23)	0.34 (0.02)	1.79 (0.24)	0.24 (0.01)	0.27 (0.18)	0.33 (0.19)
41	0.70 (0.08)	−3.96 (0.15)	−1.89 (0.21)	−0.65 (0.14)	2.58 (0.35)	0.68 (0.18)	0.38 (0.02)	1.87 (0.22)	0.20 (0.01)	0.46 (0.17)	0.45 (0.21)
50	1.26 (0.19)	−2.96 (0.39)	−1.43 (0.20)	−0.74 (0.13)	0.96 (0.13)	0.86 (0.12)	0.34 (0.02)	1.90 (0.16)	0.21 (0.01)	0.36 (0.12)	0.50 (0.16)
52	0.70 (0.08)	−3.97 (0.12)	−2.29 (0.26)	−0.42 (0.14)	2.54 (0.31)	0.82 (0.18)	0.32 (0.02)	1.92 (0.21)	0.24 (0.01)	0.32 (0.17)	0.37 (0.21)
54	1.10 (0.14)	−2.88 (0.37)	−1.22 (0.15)	−0.48 (0.10)	1.36 (0.17)	0.35 (0.18)	0.32 (0.02)	1.41 (0.22)	0.25 (0.01)	0.69 (0.17)	0.75 (0.21)
61	0.77 (0.12)	−2.50 (0.38)	−0.57 (0.14)	0.47 (0.13)	2.87 (0.43)	0.34 (0.27)	0.35 (0.02)	1.53 (0.31)	0.18 (0.01)	0.79 (0.21)	0.82 (0.25)

Note: ξ = item time intensity; σ = item standard deviation; β = the regression coefficients of the Information entropy from the LRT model; m = item visual intensity; d = item visual discrimination; λ = the regression coefficients of the Information entropy from the NBF model. Values in parentheses indicate standard errors (i.e., posterior standard deviations).

Figure 4 presents the distributions of posterior standard deviations for latent trait estimates across the five personality dimensions. Posterior standard deviations are generally low, indicating that latent traits are estimated with adequate precision for the current sample size (N = 236). Notably, measurement precision varies across dimensions, with Openness showing relatively greater dispersion and Extraversion displaying the most concentrated estimates. The presence of a few right-tailed observations in Agreeableness indicates that some respondents were located in regions of the latent continuum with lower information. Importantly, despite the increased complexity introduced by multimodal process data, posterior standard deviations remain well controlled, supporting robust recovery of individual differences.

OPEN IN VIEWER

Figure 4.

Boxplots of posterior standard deviations of latent traits ( θ ) across five personality dimensions.

To assess the fit of the multimodal model to the multimodal data, we examined scatter plots comparing observed and predicted log-RTs and FCs for each item in the Agreeableness dimension across all respondents (see Figure 5). Results for the other dimensions displayed a similar pattern. For brevity, detailed results for the remaining dimensions are provided in Part 3 of the Supplemental Document. In all cases, the relationships appeared essentially linear. The absence of significant nonlinear trends indicates that the model adequately captured item-level relationships. The scatter plots showed that predicted values closely matched the observed values. For both log-RTs and FCs, the multimodal model accurately captured the underlying data structure.

OPEN IN VIEWER

Figure 5.

Scatter plots of the observed and predicted multimodal data for 12 agreeableness items: (a) log response times (log-RTs) data; (b) fixation counts (FCs) data.

Eye-Tracking Trajectory Analysis

Figure 6 presents the response patterns of participant #228 for two items within a single dimension, where panels (a) and (b) are both positively keyed, indicating that higher scores reflect greater endorsement of the construct, with response options ranging from “1 strongly disagree” to “5 strongly agree.” More examples can be found in Part 4 of the Supplemental Document.

OPEN IN VIEWER

Figure 6.

Response probabilities, information entropy and eye-movement trajectories of participant 228.

As shown in Figure 6(a), the participant's eye-movement pattern was relatively straightforward. More fixations and longer fixation durations were observed on the item stem, indicating that the participant devoted considerable effort to understanding the item. However, fixations on the response options were few, indicating that the participant made the selection with minimal hesitation and confidence. In contrast, panel (b) of Figure 6 shows that the participant exhibited uncertainty regarding the response options. After reading the item stem, the participant initially fixated on option “2 disagree” without making a decision. The participant then returned to the item stem for further cognitive processing, revisited option “2 disagree,” and ultimately changed the selection to “4 agree.” Had the participant been certain, the total number of fixations would have been approximately 12. However, due to hesitation and additional cognitive processing, the participant re-evaluated both the item stem and response options before making a final selection. The additional four fixations likely reflect the participant's response uncertainty.

Both the separate-model (i.e., the GRM) and the multimodal-IRTM were applied to analyze the empirical data. The table below the figure displays the response probabilities for all five categories under both models, alongside the corresponding information entropy. The alignment between predicted response probabilities and eye-movement trajectories shows that the multimodal-IRTM produces category probabilities that more accurately correspond to observed eye-movement patterns during the response process. Response uncertainty for panel (a) was low; however, predicted probabilities from the GRM indicate a comparatively high level of uncertainty. Furthermore, as shown in panel (a), the eye-tracking trajectory shows that the respondent did not fixate on option “4 agree,” whereas the GRM predicted a relatively high probability for this category. Taken together, by incorporating multimodal process data, the multimodal-IRTM better aligns with participants’ observed behavioral patterns than traditional response-only models.

The present study examines the association between participants’ latent traits and eye-FCs by modeling response uncertainty, under the assumption that cognitive uncertainty during the response process is reflected in multimodal measurements. This link is supported by analyses of eye-movement trajectories. Due to uncertainty in decision-making, participants engaged in more extensive cognitive processing, manifested as increased FCs. In other words, these additional fixations, exceeding both participants’ average fixation levels and the item-level average required fixations, are captured in the model by the latent visual engagement and item visual intensity parameters. In summary, by modeling uncertainty through information entropy, participants’ latent traits are linked to response times and visual FCs, enabling the model to represent cognitive processes during responding as reflected in multimodal measurements.

Summary of the Present Study

Personality assessment plays a crucial role in organizational research, facilitating the selection of individuals whose traits align most closely with job requirements. If latent trait estimates can be made more precise, assessments will yield more informative and reliable guidance for personnel selection. Advances in technology have lowered the barrier to collecting rich process data, allowing researchers to capture information (e.g., response time, eye movements, mouse trajectories) that extends well beyond traditional self-reports. Multimodal assessment holds substantial potential. As noted in the introduction, research in cognitive and achievement testing increasingly demonstrates that incorporating multimodal data enhances measurement precision, provides deeper insights into test-taker behavior, and improves assessment quality and informativeness. However, despite their potential, the use of multimodal process data in noncognitive personality assessment remains limited, which is regrettable given its promise to improve measurement quality and support more effective personnel selection, job placement, and broader organizational decisions.

This research, situated in personality assessment using Likert-type items, leverages response uncertainty to connect participants’ latent traits with two types of process data: response times and eye-tracking measures. Using this link as a cross-loading term, we develop a joint model for response scores, response times, and visual FCs, referred to as the multimodal-IRTM. A simulation study evaluated the parameter recovery of the proposed model under multiple conditions. The results indicate that the MCMC algorithm yields reliable parameter estimates and that model parameters exhibit satisfactory recovery.

The simulation study also evaluated the performance of the proposed model relative to the GRM. In the simulation study, when the regression coefficient is positive, the person parameters ( θ ) estimated by the multimodal-IRTM consistently show higher recovery accuracy than those from the GRM. Moreover, as sample size and the magnitude of the regression coefficient increase, the performance gap becomes more pronounced, with a maximum ΔRMSE of 0.104 and a maximum ΔCOR of 0.027. For item parameters, the multimodal-IRTM also shows slight improvements over the GRM, with a maximum ΔRMSE of 0.032 and a maximum ΔCOR of 0.064. The proposed multimodal-IRTM demonstrates improvements over the GRM across both simulation and empirical studies in terms of parameter recovery and model fit. In the empirical study, the multimodal-IRTM achieves better model fit, with improvements reflected in ΔLOO (up to 172.361) and ΔWAIC (up to 444.313).

In addition, the response probabilities estimated by the multimodal-IRTM are more consistent with participants’ eye-movement patterns. The findings in Figure 6 align with our expectations. Compared with the GRM, response probabilities from the multimodal-IRTM more closely matched the category preference patterns observed in eye-movement trajectories. One possible explanation is that this cognitive process arises from the mismatch between the five discrete Likert categories and the continuous latent traits. Confronted with this limitation, respondents engage in a decision-making process in which they select the option that better, rather than perfectly, represents their latent traits. The resulting score therefore reflects a comparative evaluation, and this cognitive decision-making process is evident in the multimodal data, as indicated by longer response times and increased fixations.

What Are the Advantages of the Proposed Multimodal-IRTM in Organizational Research?

As Meißner and Oll (2019) noted, “the time is right to expand the standard methodological toolkit of organizational scholars by bringing eye-tracking to their minds and hands.” In organizational research, incorporating multimodal data into noncognitive assessment represents a promising alternative to traditional self-report–based methods. This study proposes a method capable of simultaneously analyzing participants’ scores alongside multimodal data. By integrating multimodal behavioral data, richer layers of information can be accessed, enabling a more comprehensive and accurate estimation of participants’ latent traits. The key innovations of this approach are outlined as follows:

Enriching Self-Report Analysis with Multimodal Data. The present model provides a framework for integrating multimodal data with self-report scores, enabling self-report measures to be complemented by behavioral information. In future research, additional multimodal sources—such as eye-tracking, physiological signals, facial expressions, and mouse trajectories—could be incorporated to enrich organizational behavior studies. This approach can further enhance the assessment of latent constructs, capture underlying response dynamics, and improve both the precision and interpretability of self-report measures in applied contexts, including talent selection, employee development, and high-potential identification.

What are the Recommendations for Multimodal Assessment

What Are the Limitations of the Current Study and Future Directions?

This study has several limitations that warrant further investigation in future research.

First, the present study employed the graded response model (GRM) as the psychometric foundation for the proposed multimodal framework. The GRM represents one of several viable models within the broader family of polytomous IRT. Future research could broaden this work by incorporating alternative polytomous IRT models, such as the partial credit model (PCM), generalized partial credit model (GPCM), or nominal response model (NRM). Moreover, personality and organizational assessments frequently involve multidimensional latent constructs. For example, the empirical dataset used in this study, the Big Five Personality Inventory, is multidimensional. The GRM was applied separately within each dimension to evaluate the feasibility and interpretability of the proposed multimodal framework. Future research should extend the current GRM-based framework to multidimensional graded response models (MGRM), enabling items to load on multiple latent traits simultaneously.

Second, the present study is conducted within the framework of dominance models, where the probability of endorsing an item increases monotonically as the person's latent trait exceeds the item location parameter. Within this framework, higher latent trait levels correspond to a higher probability of selecting keyed or higher-category responses, reflecting a cumulative response process. However, prior studies have assuming a dominance response process for personality data and have recommended ideal point models instead (e.g., Chernyshenko et al., 2001, 2007; Tay & Ng, 2018). Unlike dominance models, ideal point models assume a non-monotonic response function, where the probability of endorsing an item is highest when the person's latent trait is near the item location parameter. Therefore, incorporating multimodal process data into ideal point frameworks can provide deeper insights into the cognitive mechanisms underlying item endorsement, while improving parameter recovery and overall model fit.

Third, although the present study focused on unidimensional Likert-scale items, a key direction for future research is the adoption of forced-choice response formats. Forced-choice designs present respondents with multiple options simultaneously, requiring comparative judgments that can effectively reduce response biases, including social desirability, acquiescence, and extreme responding. Such designs compel respondents to prioritize among options. Incorporating process data into forced-choice formats has significant potential to deepen understanding of the cognitive mechanisms underlying decision-making. Process data can capture subtle aspects of respondents’ mental states, attentional allocation, and decision dynamics that are not observable from final choice outcomes alone. Extending the current multimodal framework to incorporate forced-choice response data could further improve the precision of latent trait estimation, enhance the ecological and psychological validity of personality and organizational assessments, and offer richer insights into the structure of multidimensional constructs.

Fourth, the empirical study should be interpreted as a preliminary demonstration of feasibility rather than a definitive validation in organizational settings. The sample size (N = 236) provides sufficient information for model estimation in this context; however, it is relatively small from an organizational research perspective and is drawn from a student sample with an age-constrained and female-dominated composition. Such characteristics limit the generalizability of the findings to typical organizational populations, which are generally more heterogeneous in age, gender, and work experience. Therefore, while the empirical results suggest that the multimodal-IRTM can be successfully estimated under moderate sample sizes, they should not be interpreted as evidence of its performance in representative organizational datasets. Future research should evaluate the robustness and generalizability of the proposed model using larger and more diverse samples, particularly in real-world organizational contexts.

Fifth, the current study focused exclusively on two types of multimodal response data: response times and visual FCs. Although these indicators offer valuable insights into the cognitive processes and attentional patterns underlying test-taking behavior, technological advances now allow for the collection of a broader range of process-related data. For example, researchers can record electrodermal responses, electroencephalography (EEG) signals, mouse or touchscreen interactions, and other behavioral or physiological measures. Future research could investigate whether the current multimodal framework can be extended to incorporate these diverse modalities, and whether such extensions might improve measurement precision, enhance model interpretability, and offer a richer understanding of the complex mechanisms underlying responses in personality, cognitive, and organizational assessments.

Sixth, the present study focused exclusively on data collected under normal and effortful response conditions and did not explicitly consider aberrant responding behaviors, such as careless or insufficient effort responding. Multimodal indicators, such as unusually short or long response times, atypical eye-movement patterns, or irregular physiological signals, may capture deviations in engagement, attention, or decision-making strategies that standard response models do not adequately explain. Disengaged examinees may interact with items using shorter response times and fewer fixations than would be expected if they were reading, comprehending, and providing an engaged response (Ulitzsch et al., 2020a). Under normal responding conditions, responses associated with higher uncertainty generally require more time than those associated with lower uncertainty. Responses with low uncertainty generally exhibit longer response times than random or careless responses, which are typically characterized by minimal cognitive engagement and abnormally short response times. Incorporating data from respondents exhibiting careless or insufficient effort responding in future research could allow for a systematic examination of whether multimodal measures improve the detection, modeling, and correction of aberrant responding, thereby enhancing the accuracy and robustness of latent trait estimation.