Organizational Research Methods

Organizational research increasingly uses natural language processing (NLP) to measure textual similarity. Despite common usage, the meaning and consistency of similarity measures (e.g., cosine similarity and Euclidean distance) across common NLP methods (e.g., n-grams and document embeddings) is unclear. This risks misalignment between theoretical constructs and textual measures, undermining the comparability of findings across studies. To address this gap, we review studies using textual similarity in organizational and psychological research, finding a jingle-jangle fallacy: identical labels are used for similarity estimates from different NLP methods, and different labels are used for the same method. Additionally, we examine the consistency of similarity measures across and within NLP methods. Different transformer-based embeddings’ similarity results are interchangeable. However, n-grams yield distinct, inconsistent results and are less appropriate for estimating similarity with distance measures. When applied to multi-word inputs, dictionaries and word embeddings return similar results reflecting linguistic style. We provide best practice recommendations and example code for operationalizing textual similarity, including clarifying which NLP methods correspond to content similarity, linguistic style similarity, and semantic similarity at the word, sentence, and document-levels of analysis.

Alternative approaches to personality measurement, such as open-ended narrative-based assessments, have potential advantages for organizational research and practice. In this research, we investigate factors that affect valid application of natural language processing (NLP) for scoring open-ended personality assessments and when, how, and why such assessments capture personality-related variance. Using a large sample of responses to open-ended assessments, convergence between NLP scores and self-report target scores increased as the degree of customization and the sophistication of the underlying model increased, with the worst psychometric performance occurring for zero-shot large language model (LLM) scores and the best for fine-tuned LLM scores. However, all scoring methods exhibited evidence of validity. Additionally, when trained to predict direct evaluations of the narrative responses, correlations with target scores were large (M = .83). NLP scores also exhibited discriminant and criterion-related validity evidence. However, validity was contingent upon the methodological rigor employed in developing writing prompts. Prompts designed to elicit trait-relevant information outperformed generic prompts, and this occurred because trait-specific prompts increased the amount of trait-relevant information (i.e., narrative units), which was associated with enhanced convergence with target scores.

Discrete choice experiments (DCEs) are a promising yet underutilized research method that can provide rigorous empirical evidence about the microfoundations of choice. This method has potential applications in many areas of management research. While DCE methods are similar to conjoint analysis and policy capturing, they are conceptually and methodologically distinct and deliver unique and valuable results that can—in the right contexts—improve external validity. This paper disambiguates DCE from related methods and provides detailed guidance on best practices for conducting DCE research, with emphasis on the elements of experimental design and analysis that are unique to DCEs. Based on a systematic literature review, we identify several emergent and canonical research domains within management where DCE methods could be used to generate novel theoretical and empirical insights. We supplement this review and best-practice guidance with a demonstration experiment and provide all the code and documentation needed for researchers to conduct DCEs with skill and confidence.

Dominance and unfolding response processes describe two ways in which individuals may respond to rating scale items. The dominance process assumes a monotonic relationship between a latent trait and the probability of endorsement and is typically modeled using a linear factor model within structural equation modeling (SEM). In contrast, the unfolding process assumes single-peaked response functions, with endorsement most likely when item and person locations are close on the latent continuum. Fitting unfolding models usually requires specialized software, which limits their integration with SEM. In this article, we proposed the ordered categorical response unfolding model (OCRUM), which can be estimated in Mplus. We illustrated its use with two empirical datasets and found that item and person locations were comparable to those obtained from the generalized graded unfolding model (GGUM). We also conducted Monte Carlo simulations to examine parameter recovery under varying sample sizes, test lengths, and response formats. Finally, we demonstrated that OCRUM can serve as the measurement component of a general structural equation model, enabling dominance and unfolding response processes to be represented within a single SEM framework.

Argument mining—the automatic identification, classification, and linking of argumentative text—has been studied in natural language processing (NLP) for more than a decade. Despite its claimed potential for applications in legal, political, and social contexts, it remained largely unexplored in organizational research. This article introduces aspect-based argument mining (ABAM) as a methodical innovation for studying how organizations justify decisions, construct legitimacy, and relate to their environments through communicative acts. By scaling up the analysis of argumentative structures beyond the limits of small-scale, qualitative studies, ABAM enables the recognition and systematic analysis of argumentation patterns in large text corpora that were hardly detectable with previous (computational) approaches. The potential is demonstrated by a longitudinal case study of Twitter debates on nuclear energy in Germany, revealing how shifting societal values—particularly the reframing of nuclear energy from a safety to a climate issue—produced growing misalignments between organizational talk of a political party organization and its social media environment.

Qualitative researchers face an enduring question: How many interviews do I need? While a variety of guidelines exist, there is limited consensus over which specific factors should determine the number of interviews required. We examined the determination of interview sample sizes in 562 qualitative studies across six high-impact management and organizational journals over a decade. Our findings reveal considerable variance in interview numbers, yet limited information is often provided on the criteria used to determine them. To promote clearer alignment between sample sizes and methodology, we examined studies with detailed descriptions of their interview sampling. We identified specific “sampling moves” used to determine the number of interviews, categorized into three types—opening, focusing, and closing sampling moves—that researchers use to establish confidence in the sample and support theoretical insights. By implication, our study refutes the notion of a “magic” interview number. Instead, sampling moves are heuristic tools that qualitative researchers can thoughtfully adapt to their analytical aims when determining appropriate sample sizes.

Measurement equivalence/invariance (ME/I) is a prerequisite for cross-group comparisons when using survey data. Although popular structural equation modeling software programs, including Mplus and lavaan, enable tests of ME/I using simple commands, identifying noninvariant items when full ME/I is rejected is more challenging. This paper reviews current procedures for identifying noninvariant items, particularly when there are more than two groups. We recommend systematically rotating the reference items and conducting pairwise comparisons on the factor loadings estimated in the configural invariance model and the intercepts estimated in the metric invariance model. The results are then summarized with the list-and-delete method to identify sets of invariant items and clusters of invariant groups. A custom R package, MEI, is developed to implement our recommended procedures. With simple commands, MEI automatically conducts ME/I tests, identifies noninvariant items, and compares latent means with partial measurement invariance. This allows researchers to interpret cross-group comparison results more precisely. Finally, our procedures for testing ME/I from cross-group comparisons and the MEI package are extended to longitudinal studies with panel data, congruence studies with dyadic data, and multilevel studies with nested data.

Taxonomies provide a systematic way to organize phenomena and have various practical and theoretical benefits for organizational researchers and practitioners. While taxonomy development and maintenance is often a burdensome process (e.g., time-consuming, costly, and prone to judgmental error), advances in natural language processing (NLP) have the potential to streamline this process. In this study, we employed various evaluation metrics (e.g., cosine similarity) to investigate how machine learning (ML) methods and large language models (LLMs) can automate taxonomy development and maintenance. We examined two embedding models, six clustering algorithms, and three generative LLMs (for creating cluster labels) to construct taxonomies and compared their alignment with four established taxonomies (CABIN, IPIP-NEO-120, ATAF, and O*NET). The confirmatory taxonomic method we examined resulted in effective clustering (i.e., similar text statements were consistently grouped), frequently yielded structures similar to the original taxonomies for ATAF, IPIP-NEO-120, and CABIN (with O*NET being more variable), and resulted in extremely efficient taxonomy title generation. These findings can provide researchers with a foundation for how to approach NLP-based taxonomy development and maintenance activities for their own contexts.

We argue that reporting control variable results strengthens the transparency and credibility required for programmatic knowledge building in regression-based empirical research. Control variables are not just technical adjustments; their results provide diagnostic information that can reveal otherwise hidden biases and model misspecification. We present a practical multistep approach that employs control variable coefficients to uncover harmful combinations of multicollinearity, omitted variable biases and correlated measurement error. These harmful combinations, in turn, may generate type 1 errors (false positives) among variables of theoretical interest. We also show how to distinguish true suppressor effects from artifacts of poor model specification. Full reporting supports programmatic research by enabling scholars to compare results across studies, build on prior findings, and refine theory over time. Based on these benefits, we challenge a recent call to omit control variable results from manuscripts. Instead, we recommend that journals and reviewers require their inclusion in all published results tables.

Rhythms form an essential part of organizational life, involving embodied patterns of repetition and difference that structure work processes, against the ongoing background of wider organizational and environmental rhythms. Organizational literature increasingly recognizes the importance of rhythms; yet little methodological work exists, either at the level of theorization or practical guidance. The current study draws on Lefebvre's foundational work on rhythmanalysis to elaborate an organizational methodology for studying rhythms. We argue that rhythmanalysis provides a critically oriented approach to understanding social dynamics and advances theorizing about organizational environments by overcoming the dichotomy between entities and processes, stability and change. In this article, we propose methodological guidelines for developing the field of rhythmanalysis in organizational settings by illustrating how it can be conducted through the reanalysis of ethnographic material. We discuss the methodological contributions of rhythmanalysis for a critical exploration of organizational dynamics.

Recent advances in process data collection have made it possible to efficiently collect multimodal behavioral indicators, such as response times and eye-tracking measures. These multimodal data have been widely applied in cognitive and achievement assessments, where they have improved the accuracy of latent construct estimation. However, the use of informative multimodal process data in noncognitive assessments, such as personality measures widely used in organizational research, has received considerably less attention. To address this gap, we integrate response time and eye-tracking data into a conventional item response model to capture respondents’ response processes, thereby improving differentiation across trait levels and enhancing noncognitive assessment. Simulation studies were conducted to evaluate the performance of the proposed model and compare it with a conventional IRT model. Results indicate that model parameters can be accurately recovered and that incorporating multimodal data significantly improves the accuracy of person latent trait estimates. Finally, an empirical analysis was conducted to demonstrate the applicability and advantages of the proposed model in personality assessment.

Sentiment analysis (SA) has grown considerably in organizational science research over the past two decades, particularly in the last few years. While enthusiasm for integrating advanced natural language processing algorithms is encouraging, authors are not reaping the benefits of such tools fully. Our systematic review of SA application in the organizational sciences suggests that authors struggle to appreciate all of the decisions that are inherent to SA, the choices that are available at each decision point, and the consequences of each choice. To address this gap, we use a working example to illustrate four critical decision points authors confront when conducting SA, and the subsequent impact different choices can have on one's conclusion. Decision points include selecting the SA method, computing a sentiment score, preprocessing the data, and using an appropriate level of analysis. We conclude with a framework outlining five dimensions (e.g., accuracy, interpretability, computational cost) to guide the selection of an SA approach based on study goals and needs, along with seven recommendations to authors wishing to apply SA.

Content analysis has enabled organizational scholars to study constructs and relationships that were previously unattainable at scale. One particular area of focus has been on sentiment analysis, which scholars have implemented to examine myriad relationships pertinent to organizational research. This article addresses certain limitations in sentiment analysis. More specifically, we bring attention to the challenge of accurately attributing sentiment in text that mentions multiple firms. Whereas traditional methods often result in measurement error due to misattributing text to firms, we offer coreference resolution—a natural language processing technique that identifies and links expressions referring to the same entity—as a solution to this problem. Across two studies, we demonstrate the potential of this approach to reduce measurement error and enhance the veracity of text analyses. We conclude by offering avenues for theoretical and empirical advances in organizational research.

For various reasons, respondents to forced-choice assessments (typically used for noncognitive psychological constructs) may respond randomly to individual items due to indecision or globally due to disengagement. Thus, random responding is a complex source of measurement bias and threatens the reliability of forced-choice assessments, which are essential in high-stakes organizational testing scenarios, such as hiring decisions. The traditional measurement models rely heavily on nonrandom, construct-relevant responses to yield accurate parameter estimates. When survey data contain many random responses, fitting traditional models may deliver biased results, which could attenuate measurement reliability. This study presents a new forced-choice measure-based mixture item response theory model (called M-TCIR) for simultaneously modeling normal and random responses (distinguishing completely and incompletely random). The feasibility of the M-TCIR was investigated via two Monte Carlo simulation studies. In addition, one empirical dataset was analyzed to illustrate the applicability of the M-TCIR in practice. The results revealed that most model parameters were adequately recovered, and the M-TCIR was a viable alternative to model both aberrant and normal responses with high efficiency.

Researchers often combine longitudinal panel data analysis with tests of interactions (i.e., moderation). A popular example is the cross-lagged panel model (CLPM). However, interaction tests in CLPMs and related models require caution because stable (i.e., between-level, B) and dynamic (i.e., within-level, W) sources of variation are present in longitudinal data, which can conflate estimates of interaction effects. We address this by integrating literature on CLPMs, multilevel moderation, and latent interactions. Distinguishing stable B and dynamic W parts, we describe three types of interactions that are of interest to researchers: 1) purely dynamic or WxW; 2) cross-level or BxW; and 3) purely stable or BxB. We demonstrate estimating latent interaction effects in a CLPM using a Bayesian SEM in Mplus to apply relationships among work-family conflict and job satisfaction, using gender as a stable B variable. We support our approach via simulations, demonstrating that our proposed CLPM approach is superior to a traditional CLPMs that conflate B and W sources of variation. We describe higher-order nonlinearities as a possible extension, and we discuss limitations and future research directions.