Abstract
In contemporary organizational research, when dealing with large heterogeneous datasets and complex relationships, statistical modeling focused on developing substantive explanations typically results in low predictive accuracy. In contrast, machine learning (ML) exhibits remarkable strength for prediction, but suffers from an unexplainable analytical process and output—thus ML is often known as a “black box” approach. The recent development of explainable machine learning (XML) integrates high predictive accuracy with explainability, which combines the advantages inherent in both statistical modeling and ML paradigms. This paper compares XML with statistical modeling and the traditional ML approaches, focusing on an advanced application of XML known as evolving fuzzy system (EFS), which enhances model transparency by clarifying the unique contribution of each modeled predictor. In an illustrative study, we demonstrate two EFS-based XML models and conduct comparative analyses among XML, ML, and statistical models with a commonly-used database in organizational research. Our study offers a thorough description of analysis procedures for implementing XML in organizational research, along with best-practice recommendations for each step as well as Python code to aid future research using XML. Finally, we discuss the benefits of XML for organizational research and its potential development.
Keywords
With the continuous advancement of research methods and big data technologies in social science, the two often-separate research paradigms of explanation and prediction are undergoing significant developments (Hofman et al., 2021; Zhang et al., 2022). Internet-based big data technologies, as catalysts of change, generate massive amounts of digital data (such as social media data) that are often noisier, less structured, and less “designed” compared to traditional organizational research data (e.g., survey data). This has prompted many researchers to move from statistical models for explanation to more complex machine learning (ML) approaches for prediction (Putka et al., 2018; Zhang et al., 2021). The shift is not merely a technical adaptation; it is also an opportunity to explore new methods and paradigms for examining complex organizational phenomena in the era of big data.
Specifically, statistical modeling focused on explanation struggles with processing large-scale heterogeneous data and extracting relationships from complex data patterns, leading to the problem of low prediction accuracy (Chen et al., 2012; George et al., 2014). Statistical modeling, which mainly relies on linear models (e.g., regression, structural equation models, or multilevel models), aligns with theory testing and empirical examination of hypotheses using explanatory models, which limit accurate predictions (Shrestha et al., 2021). Accordingly, ML has emerged as a superior and flexible tool in organizational research, providing substantially expanded opportunities for enhancing prediction accuracy (Varian, 2014).
ML is increasingly applied in organizational research, such as neural networks and support vector machines (SVMs; Somers et al., 2021; Spisak et al., 2019). Recent applications of ML have demonstrated their exceptional predictive accuracy for effectively identifying a variety of data patterns (Loh et al., 2023; Putka et al., 2018). Although intriguing, these ML models are often considered “black boxes” due to their opaque working mechanisms and difficulties in achieving good “explainability” in terms of the relationships among variables and the theoretical implications of these patterns of covariance (Hagras, 2018; Yarkoni & Westfall, 2017). With low levels of explainability, it is difficult to understand how prediction results are achieved, which in turn complicates the explanation of relationships between predictors and outcomes (Ding et al., 2020). The lack of clarity in understanding the predictive processes of ML limits understanding of the modeling process and hinders the practical application of ML by researchers.
To address this limitation, the emergence of explainable machine learning (XML) offers new opportunities to integrate explainability in statistical modeling with the predictability of ML (Hofman et al., 2017). XML is characterized by high prediction accuracy and intuitive explanations, focusing on a detailed analysis of the contribution that the predictor makes in predicting the outcome. It provides an understanding of, and thus trust in, how ML makes decisions or predictions. The more ML can be expressed in simple and understandable language, the easier it is for researchers to comprehend the rationale behind its decisions or predictions. To date, XML has been successfully applied in research such as investigating factors that predict organizational performance and employee turnover (Das et al., 2023; Ylinen & Ranta, 2024), providing notable insights for understanding various organizational and management issues (Li et al., 2019). As the need for transparency in the decision-making process in organizations grows, XML is likely to expand its scope in both management research and practical applications.
The focus of our paper is to (1) explore XML and demonstrate its importance and practical utility in the field of organizational research and (2) systematically compare XML, traditional ML, and statistical modeling, detailing their predictive performance and explainability both methodologically and empirically with illustrative data. This research provides three significant contributions to the methodological development of organizational research. First, we introduce how XML integrates predictive and explanatory models, providing a powerful tool for organization scholars. XML maintains the degree of explainability found in statistical modeling—which is highly valued in empirical research for theory testing—while optimizing the predictability that is crucial for various applications in management practices. Second, our comparative analysis of XML with traditional ML and statistical modeling expands the understanding of these methods and enables a critical evaluation of them. Third, through an empirical demonstration, we offer a set of step-by-step guidelines with best-practice recommendations and analysis codes for applying XML to future organizational research. Our paper not only aids the academic community in gaining a deeper understanding of the operational intricacies of XML, but it also provides guidance to researchers in order to facilitate the dissemination and application of XML.
Explanation Models and Prediction Models in Organizational Research
Statistical Modeling
Statistical modeling is a set of analytical techniques based on mathematics and probability theory that makes inferences about the characteristics of populations, such as regression analysis, structural equation modeling, and multilevel modeling (McCullagh, 2002). These methods are typically based on a set of strict modeling assumptions (e.g., linear relationships and homoscedasticity), explicit parameter estimates (e.g., slopes and intercepts), and relevant statistical tests (e.g., significance tests). Classic statistical models, such as linear regression, have been fundamental analytical tools in organizational research (James et al., 2013).
The excellent explainability of statistical models arises from their well-defined model structure (i.e., the equations that define the model). Analyses based on these methods are often guided by established theoretical frameworks that are mapped directly onto the statistical models (i.e., the model's specification). This approach to scientific research is largely deductive and assumes specific data generation processes to clearly explain the relationships between variables. For example, when employing ordinary least squares (OLS) regression to test the effect of independent variable X on dependent variable Y, the observed unstandardized coefficient (e.g., b = .5, p < .001) reflects a result that is straightforward to interpret. Specifically, the unstandardized coefficient of .5 suggests that, on average, a 1-unit increase in X is associated with a 0.5-unit increase in Y. Accordingly, parameter estimates in statistical models are often quantitative measures of the impact of predictor variables such as X, which enhances the explainability of the model (Highland et al., 2021).
Hence, the widespread use of statistical modeling in organizational research is partly due to the explainability of the results, which facilitates theory testing. The model structure and parameter estimate together create a high level of explainability, which justifies statistical modeling's prominence in many organizational research and decision-support scenarios. However, despite their excellent explainability, statistical models have considerable limitations in other areas, especially their predictive utility for multiple reasons (Mannering et al., 2020).
First, statistical modeling encounters challenges when confronted with large-scale, high-dimensional, or unstructured data due to a reliance on strong assumptions. For example, while a generalized linear model (GLM) underpins a large number of statistical models, more predictively accurate methods allow for a variety of nonlinear relationships. In situations where a linearity assumption is contravened, the predictive performance of statistical modeling can be substantially undermined. Second, statistical modeling prioritizes in-sample model fitting rather than out-of-sample prediction, thus maximizing model-based explanations of observed data rather than predictive accuracy for new observations. Third, within the landscape of contemporary “big data,” the proliferation of predictors can become exceedingly pronounced. Statistical models and their estimators, often limited by practical and computational constraints that allow only a small number of variables, tend to operate under the assumption of homogeneous linear relationships between predictors and outcomes across the entirety of a sample—something that is unlikely with “big data.”
Machine Learning
ML attracted attention in the 1950s due to its accurate prediction of complex phenomena (Badillo et al., 2020). It is a technology that enables computers to learn patterns from even huge datasets to make optimal predictions. The core of ML lies in discovering hidden data patterns to optimize prediction accuracy. Specifically, it involves minimizing out-of-sample prediction errors in the estimation process, even with a large number of substantive variables in the analysis, while not assuming linear relationships among variables (Oswald et al., 2020; Putka et al., 2018). For instance, artificial neural network algorithms autonomously learn and identify complex patterns in data, processing nonlinear and high-dimensional data without explicit model-based functional assumptions (Arrieta et al., 2020). Consequently, this data-driven approach does not suffer from prediction inaccuracy in statistical modeling. In recent years, an increasing number of studies have begun applying ML in fields ranging from natural sciences such as health (Esteva et al., 2017; Klute et al., 2019) to social sciences such as economics (Akyildirim et al., 2021) and organizational research (Najafi-Zangeneh et al., 2021; Sajjadiani et al., 2019), fulfilling critical prediction tasks such as classification.
A significant challenge with the explainability of ML, however, lies in the increasing complexity of researchers’ datasets (Sullivan, 2022). ML can very easily lose transparency and explainability when applied to large-scale, nonlinear problems. This can necessitate ML with multilayer complex network structures, many parameters, or combinations of multiple simple models that are optimized for prediction by capturing intricate patterns and nonlinear relationships in data. These models include artificial neural networks and ensemble learning methods, optimizing prediction rather than explainability. The algorithms that drive these models, and the complex functions related to modeling variables they produce, are rarely directly linked to substantive phenomena of interest to researchers, such as explicitly estimating the relationships between modeling variables (Mühlbacher et al., 2014; Shrestha et al., 2021). Therefore, complex ML struggles to provide sufficient information to explain the internal, substantive mechanisms that underlie their specific predictive outcomes.
Consequently, models with such complex input–output relationships that are difficult to intuitively explain using the model itself are often referred to as “black box” models in ML-related literature (Henninger et al., 2023; Rai, 2020; Rudin, 2019). However, explainability is crucial in applications of ML because it provides transparency in terms of why a model is accurate in prediction, which is important for decision-makers in both industry and academia whose work requires an understanding of how predictions are made. In practical applications for management decision-making, such as strategic management, corporate governance, and human resource management, the ability to explain complex decision processes is vital for maintaining accountability, as these decisions can have profound effects on organizational performance and employee well-being.
Explainable Machine Learning
With the widespread application of ML in research and applied contexts, there is a growing need for algorithmic fairness, transparency, and explainability. Recent advances in ML have been aimed at integrating high predictive performance with the explainability that is typically associated with statistical modeling. The resulting area of methodological development is referred to as XML (see Miller, 2019), which has shown its importance for enhancing trust and promoting the responsible use of artificial intelligence (AI; Gunning et al., 2019). According to Murdoch et al. (2019, p. 22071), XML aims to extract relevant knowledge about the relationships reflected in datasets, while explicitly demonstrating how ML builds local models to fit data (Arrieta et al., 2020). The resulting explainable information can be presented to researchers in mathematical equations, visualizations, and natural language outputs. Utilizing these advantages, XML is being applied in critical areas with stringent reliability and fairness requirements, such as medical imaging (ElShawi et al., 2021; Litjens et al., 2017), energy performance (Fan et al., 2019), and supply chain risk (Baryannis et al., 2019).
Specifically, the “explainability” aspect of XML is the capacity to provide interpretations in terms that are understandable to a human (Arrieta et al., 2020; Kosasih et al., 2024). It is designed to help people understand how ML learns, what they learn from data, and why they reach a given conclusion. For example, Yong et al. (2018) used XML to predict currency exchange rates. Following the steps of data preparation, splitting, and training in ML, they utilized the XML algorithm to identify multiple predictive “local models.” This provided interpretable estimates for the relationships between variables, making XML a hybrid approach with both ML methods and interpretable characteristics based on the third step.
The key to achieving explainability lies in adopting appropriate techniques or methods to reveal how the model learns from data and makes predictions. Based on the timing of the explanation, XML is typically divided into two approaches as shown in Table 1, including pre-hoc explainability and post-hoc explainability (Murdoch et al., 2019). Pre-hoc explainability primarily occurs before or during the model training process. It is achieved by selecting model structures and features that offer high explainability to construct “transparent box” models. Rule-based models are similar to pre-specified statistical models and are therefore preferred for constructing XML with pre-hoc explainability. Such approaches are transparent and interpretable by design, because their decisions can be traced back to understandable components, such as rules or prototypes. Because the parameters can be directly linked back to practical problems or theory, pre-hoc XML is helpful by allowing explicit theory testing, verification, and error analysis (Jiménez-Luna et al., 2020).
Explanation Approaches of XML Models.
Note. SHAP stands for Shapley additive explanations; LIME stands for local interpretable model-agnostic explanations; ICE stands for individual conditional expectation; Grad-CAM stands for gradient class activation mapping; XML stands for explainable machine learning.
Post-hoc XML is the second way to integrate ML with explainability. Post-hoc explanations are conducted after ML-based model training is complete, primarily analyzing the model's output, weights, or utilizing other explanatory tools to elucidate the decision-making process of complex models. This can be categorized into two types of post-hoc XML based on interpretative approaches. One type is model-agnostic and does not depend on a model's specific internal structure, such as the Shapley additive explanations (SHAP; Lundberg & Lee, 2017), local interpretable model-agnostic explanations (LIME; Ribeiro et al., 2016), and the individual conditional expectation plots (Goldstein et al., 2015) methods. The other type includes model-specific methods, such as gradient class activation mapping (Grad-CAM) using neural network models (Chattopadhay et al., 2018; Selvaraju et al., 2020). A description of explanatory approaches in XML is presented in Table 1.
It is worth noting that compared with pre-hoc approaches, these post-hoc approaches can be problematic because they are more aligned with the black box models’ behaviors rather than human understandings. In turn, their explanations can be unreliable and misleading due to difficulties in understanding what the black box is doing from a methodological and substantive perspective. Post-hoc methods can also lead to complicated decision pathways that are ripe for human error (Rudin, 2019). Accordingly, subsequent elaboration of XML in this study will focus on the pre-hoc approach.
Comparisons of Three Analytical Paradigms
Statistical modeling, ML, and XML, as three main analytical paradigms, possess unique characteristics and have changed our understanding of data analysis. Table 2 summarizes these distinctions across six attributes in three analytical stages, including modeling foundations, modeling processes, and modeling outcomes. These distinctions reflect their unique roles in organizational research and highlight the respective efficacy of each approach.
Differences Between Statistical Modeling, Traditional ML, and XML.
Note. ML = machine learning; XML = explainable machine learning.
Regarding foundational modeling, the three methods exhibit significant contrasts in their analytical goals and data requirements. Statistical modeling typically uses small sample sizes and highly structured data (Zhang et al., 2022), aiming to guide management practice through rigorous theoretical validation and testing (McCullagh, 2002). In contrast, ML and XML are more suited for handling mixed and complex data types, including numerical, textual, audio, and video data (George et al., 2014). ML primarily focuses on data mining to optimize predictions, whereas XML strives to enhance the transparency and comprehensibility of decision-making processes while maintaining the predictive strength of ML (Arrieta et al., 2020). These foundational differences highlight the distinct applications of each method in theory and practice and emphasize their specific roles and analytical capabilities in organization research.
Regarding the modeling process, statistical modeling relies on clear mathematical formulas and strict assumptions (James et al., 2013; Razavi, 2021), making its results easy to understand and explain (such as linear relationships), thereby offering high transparency. In contrast, both traditional ML and XML adopt data-based predictive modeling approaches, relying on algorithms to autonomously learn and adjust model parameters (Arrieta et al., 2020; Putka et al., 2018). However, these two methods significantly differ in transparency. The internal workings and algorithm adjustment processes of traditional ML are often opaque, whereas XML employs techniques like LIME and SHAP to explain and visualize the decision logic of models (Lundberg & Lee, 2017; Ribeiro et al., 2016). However, the transparency of the XML analysis paradigm can be influenced by factors such as data diversity, model complexity, and the researcher's expertise, leading to XML generally being considered to have medium to high transparency. This differentiation in modeling approaches and transparency highlights the unique contributions of each paradigm in terms of scientific rigor and decision-making effectiveness.
Regarding modeling outcomes, the method results and application scopes of these three analytical paradigms differ markedly. Statistical modeling primarily focuses on testing pre-determined hypotheses through in-sample fitting rather than out-of-sample prediction, leading to low predictability but high explainability. Owing to its solid theoretical guidance and strict methodological assumptions, it has been widely used for hypothesis testing and validating relationships among variables (Bennett et al., 2022; McCullagh, 2002), becoming a mainstream research method in social sciences. In contrast, traditional ML focuses on pattern recognition and data mining from large datasets to achieve optimal predictive performance, making revolutionary advancements in fields like image and voice recognition (Haeb-Umbach et al., 2019; Portugal et al., 2018). However, its use of complex, nonlinear, multiparameter algorithmic models makes understanding the specific predictive mechanisms of these models a challenge (Henninger et al., 2023; Rudin, 2019). While striving to achieve the same high predictive accuracy as ML, XML incorporates visualization tools to illustrate the relationship between predictors and outcomes (Mi et al., 2020). This intuitive interpretation of data patterns enhances individuals’ understanding of, and trust in, the model's output. Consequently, XML achieves both high predictability and moderate to high explainability, which is particularly important in decision-making environments that require transparency and a high degree of accountability, such as in risk management and medical diagnostics.
Evolving Fuzzy Systems-Based Explainable Model
This section elaborates on evolving fuzzy systems (EFSs) as the methodological foundation of XML. EFS-based XML was selected primarily because of its unique ability to represent knowledge in a form that is intuitive, and its capability to mimic human reasoning processes by using linguistic IF–THEN fuzzy rules. This aligns with the objectives of XML, which aims to make the decisions of ML transparent and explainable. Furthermore, the inferencing processes in EFS are specifically designed to emulate human thought patterns, enabling each decision made by the system to be explicitly explained and verified. This feature not only enhances the transparency of the AI decision-making process but also facilitates collaboration between human experts and the AI system, merging human expertise with the computational power of fuzzy systems. This integration of human-understandable knowledge representation and human-like reasoning makes EFS a uniquely suitable choice for XML, standing out from other approaches in the field. We will elaborate on both the methodological and mathematical foundations of EFS. Given the technical complexities of this section, we have included a glossary in Appendix A (Supplemental materials) to explain important terminologies to facilitate the understanding of readers.
Methodological Foundation of EFS
Fuzzy systems are one of the three pillars of computational intelligence and are widely used as a powerful tool for modeling nonlinear problems with uncertainties and imprecision (Gu et al., 2023). Unlike traditional ML including neural networks, fuzzy systems are based on linguistic IF–THEN fuzzy rules, which provide a natural and intuitive representation of knowledge that is understandable by humans (see Fiss, 2007, 2011; Fiss et al., 2013; Greckhamer et al., 2018). The fuzzy inferencing processes are designed to mimic human reasoning, enabling an explicit explanation for each decision made by fuzzy systems that can be further examined and verified by domain experts. Human experts can also work in conjunction with fuzzy systems to create collaborative AI systems, using the expertise of humans alongside the computational capabilities of fuzzy systems. Because of the (1) transparent system structure, (2) explainable reasoning mechanism, (3) ability to handle uncertainties, and (4) use of human-understandable IF–THEN rules to present abstract knowledge, fuzzy systems gained significant attention across various domains in the current move toward XML-based AI (Fernandez et al., 2019; Hagras, 2018).
EFS can be used as the main tool to learn a transparent IF–THEN fuzzy rule-based predictive model from data (Gu et al., 2023). As a prominent approach to XML, EFS is a special class of fuzzy systems that are capable of self-organizing the IF–THEN rules from data streams in a computationally efficient, single-pass, data-driven manner—without making assumptions about a data generation model or the need for prior knowledge on data distribution. 1 A typical EFS can autonomously identify a number of clusters through online clustering to disclose the underlying patterns of data. This is typically achieved through the utilization of statistical operators that are calculated based on the mutual distances of data points. The centers of these clusters would be the points with the highest likelihood of belonging to a given cluster in a probabilistic framework. The extracted clusters are then converted into a set of human-understandable IF–THEN fuzzy rules to preserve the knowledge mined from data and will be used for fuzzy inference. In organizational contexts, the IF–THEN fuzzy rule could be “if data x from employee i are closest to cluster A, then employee i is labeled ‘A.’” Note that, compared to traditional cluster analyses (Lawrence & Zyphur, 2011; Nylund et al., 2007), EFS avoids exhaustive searching across models with different numbers of clusters, and instead uses a single-pass data-driven method for cluster identification (i.e., class enumeration; for an example see Gu et al., 2020). This helps to circumvent the limitations faced by techniques such as GLM trees, which are restricted to precise interpretations of cluster characteristics under limited tree depth.
Furthermore, EFS employs a multimodel structure to precisely approximate complex relationships, where each individual IF–THEN rule presents a local model representing a different hypothesis or approximation of the underlying system dynamics (Angelov et al., 2017). EFS is a powerful tool for handling streaming data with dynamically changing patterns due to the capability of self-evolving multimodel structure and self-updating parameters from new data (Angelov et al., 2017), as in organizational contexts with employees’ performance and other data which may be gathered continuously throughout a working day.
With its efficiency, explainability, transparency, and accuracy, the EFS-based XML applied in our study can achieve both high predictive accuracy as well as explainability. As we will show, this includes predictor–criterion relationships similar to effect sizes, and this can all be done with typical computational resources available to organizational researchers on their personal computers (Fernandez et al., 2019).
Mathematical Foundations of EFS
A typical EFS takes the form of linguistic IF–THEN fuzzy rules, which offer intuitive representations of empirical relationships for easy interpretation (Angelov & Gu, 2018). Each linguistic IF–THEN rule represents a particular local model of data, namely, a cluster where the premise (IF) part is defined by the arithmetic mean of the cluster (represented by premise parameters, reflecting the importance of predictors within the specific cluster characteristics), and the consequent (THEN) part is defined by coefficients reflecting the linear relationship between the predictors and the outcome within the local model (represented by consequent parameters).
To elaborate, the main aim of the XML training is to identify a number of local models (clusters) that preserve the underlying structure and patterns from observed data. Each local model is formulated in the form of an IF–THEN fuzzy rule
To elaborate further, the overall output of the XML y based on the input variables
In this study, we use two recently introduced EFS: (1) an autonomous learning multimodel (ALMMo) system (Angelov et al., 2017); and (2) a self-adaptive fuzzy learning (SAFL) system (Gu & Shen, 2021) to construct predictive EFS-based XML. Both the ALMMo and SAFL represent the state-of-the-art in EFS literature, with the source codes being available on public websites (e.g., MathWorks and Github) for interested readers and researchers to use. In comparison with ALMMo, SAFL has superior structure-evolving and parameter-updating mechanisms. As a result, SAFL typically offers greater prediction accuracy than ALMMo and has a lower risk of overfitting due to the utilization of a more advanced fuzzy reasoning mechanism. For readers’ convenience, we also present Python code for ALMMo and SAFL models in Appendices B and C (Supplemental materials), which are used in illustrative study.
Empirical Illustration and Comparison
In this section, we use National Basketball Association (NBA) data to demonstrate the application of EFS-based XML and the comparative analyses with traditional ML and statistical modeling. Specifically, we developed a predictive model for job performance of player, designed to include a broad spectrum of predictors, ranging from personal behaviors to team dynamics. This fits with the long-standing focus in organizational research on predicting job performance, aiming not just to enhance the accuracy and transparency of predictions but also to highlight the advantages of XML in comparison with traditional ML and statistical modeling. We now introduce the research background and the logic of model construction.
Research Context of Illustrative Data
The NBA is a professional basketball league in the United States, which was established in 1946 and comprises 30 teams with top basketball players from around the world (Ertug & Castellucci, 2013). The NBA represents the pinnacle of basketball competition and serves as an important platform for showing individual player skills and performance. In the NBA, the performance of players, team records, and various statistical data, such as scoring, rebounds, and assists, are accurately recorded. This data is public and has been applied in various empirical studies in organizational research (e.g., Campbell et al., 2013; Christie & Barling, 2010; Grijalva et al., 2020). The public and official nature of NBA data lends it a degree of reliability and authority (Christie & Barling, 2010; Zhang, 2017). As a highly competitive and professionalized organization, the NBA data provides a rich empirical resource for organizational research (Chen & Garg, 2018; Grijalva et al., 2020).
NBA data is appropriate for empirical illustration and comparison of XML for two reasons. First, the NBA is a useful context for investigating research questions in organization studies because it allows examining various predictors of performance (Swider et al., 2017), such as individual attributes (Grijalva et al., 2020), and scholars widely advocate using sports data to study organizational phenomena (Wolfe et al., 2005). Second, the richness and accessibility of NBA data make it ideal for illustrative purposes. As a highly competitive and dynamic environment, the NBA provides extensive and detailed records (Christie & Barling, 2010), covering a broad range of individual behavioral indicators and also reflecting complexities in leadership, collaboration, and organizational development.
This offers a unique opportunity to study the job performance of professional basketball players, testing the effectiveness of our XML in handling this complexity. It also enables in-depth exploration of the intricate relationships between external environments, individual characteristics, and performance, which has been extensively applied in management research (Giambatista, 2004; Obenauer & Langer, 2019). Moreover, the official website of the NBA contains various public data records of past and present players, addressing issues of data availability and replicability.
Construction of Explainable Predictive Model of Job Performance
Over the past few decades, job performance has been a core outcome variable in organizational research that serves as a crucial indicator of employee value and potential (Campbell & Wiernik, 2015). Because employee job performance is directly linked to the overall success and effectiveness of an organization, investigating the predictive factors conducive to enhancing job performance is of paramount significance (Campbell et al., 1993; Carpini et al., 2017). For instance, previous research showed collaboration among employees is an important predictor in enhancing job performance (Farh et al., 2012; Mathieu et al., 2008).
However, existing empirical research on job performance is largely limited by using statistical models. Specifically, previous research often relies on subjective survey data from employees (Campbell & Wiernik, 2015; DeNisi & Murphy, 2017), and fails to establish strong predictions due to the assumptions of statistical modeling (Landis, 2001; Zyphur et al., 2008). In light of this, we attempted to develop a prediction model of employee performance using XML.
Using the NBA data from 2015 and 2016, which are training data and testing data respectively, this predictive model incorporates a diverse array of predictive factors, including (1) self-serving behavior, coworker-serving behavior, and team-serving behavior; (2) the physical attributes of height and weight; (3) work experience, including working time, working amount, uncooperative behavior, age of joining the team, and age; and (4) team characteristics, including team performance, average age, average tenure, inequality within teams, and team cooperation. Through this multidimensional analysis, we aim to provide a holistic, accurate, and transparent method for predicting job performance, thereby supporting organizations in making more informed management decisions. In Appendix D (Supplemental materials), we provided a detailed description of this dataset.
Results of Comparison for Predictability Between ML and XML
Here we offer an empirical demonstration based on the illustrative data to compare the performance of the XML (i.e., ALMMo and SAFL) with traditional ML (e.g., Least Absolute Shrinkage and Selection Operator [LASSO] and the neural networks). In this illustration, as clarified in Appendix D (Supplemental materials), we use the training data to construct the predictive models and the testing data as the out-of-sample test to examine the prediction performance of the trained models. All of the ML and XML involved in this illustration were implemented in Python. Specifically, we employed the well-known scikit-learn library (https://scikit-learn.org/stable/) to run analyses based on LASSO (Tibshirani, 1996), Bayesian ridge regression (BRR) (Hoerl & Kennard, 1970), random forest (RF) regressor (Breiman, 2001), and SVM regressor (Collobert & Bengio, 2001). In addition, we applied the glmtree library (https://pypi.org/project/glmtree/) to run analysis based on generalized linear model-based tree regressor (GLMTree) and the tensorflow library (https://www.tensorflow.org/) to implement analyses based on multilayer perceptron (MLP) (Murtagh, 1991; Tang et al., 2015), recurrent neural networks (RNNs) (Woźniak et al., 2020), and long short-term memory (LSTM) networks (Fischer & Krauss, 2018). 2 MLP uses the standard sigmoid activation function, RNNs and LSTMs use the tanh activation function, which are relatively standard.
The hyperparameter settings of the two XML and eight comparative models for empirical illustration are listed in Table 3 (the default settings are used for the hyperparameters and therefore not listed in the table). However, due to the lack of prior knowledge, it is not practically feasible to obtain the best hyperparameter setting for each individual model a priori. Therefore, we performed a 10-fold cross-validation based on the training data to identify the optimal set of hyperparameters for each individual predictive model with the root mean square error (RMSE) as a performance metric, thereby maximizing the prediction performance of the 10 predictive models to enable a fairer comparison.
Hyperparameter Settings of the XML Models and Comparative Models.
Note. This table summarizes the range of hyperparameter settings considered for various predictive models in the study. For the two evolving fuzzy systems models, ALMMo and SAFL, hyperparameters such as threshold D_0, learning rate eta_0, and regularization term omega_0 are set within specific ranges to optimize the clustering process. The ranges for the hyperparameters alpha and lambda in the LASSO and Bayesian ridge regression (BRR) models are designed to adjust the model complexity and penalty strength. For the random forest (RF) model, n_estimators represents the number of decision trees, while max_depth controls the depth of the trees. The support vector machine (SVM) model adjusts through the kernel function type (kernel), penalty parameter C, and slack variable epsilon. The hyperparameter settings for the GLMTree are aimed at regulating tree complexity, including maximum depth (max_depth), minimum number of samples per leaf node (min_samples_leaf), and regularization parameter (reg_lambda). The choices of units, learning rate (learning_rate), and batch size (batch_size) for the multilayer perceptron (MLP), recurrent neural network (RNN), and long short-term memory (LSTM) models reflect the depth of the network structure and details of the training process. The selection of hyperparameters is based on cross-validation and predefined ranges to ensure the optimization of model predictive performance.
Once the optimal hyperparameters for the predictive models were identified, we trained the 10 predictive models based on the training data and evaluated their performances on the test data. The prediction results by the 10 predictive models are reported in Table 4 using RMSE and R2 as the performance metrics. 3 To allow a certain degree of randomness, we carried out 10 repeating experiments for each model by randomly shuffling (i.e., permuting) the order of the training data and the results are reported in the form of mean ± standard deviation. For visual clarity, the best result per metric is highlighted. As shown in Table 4, the XML SAFL outperforms all the comparative models in both performance criteria. The ALMMo model achieves a predictive performance that aligns with the average prediction levels of the majority of ML models. These results indicate that the XML methods are either similar or superior to typical ML for prediction.
Performance Comparison Between two XML Models and Eight Comparative Models.
Note. Predictive performance assessment includes two metrics: root mean square error (RMSE) and coefficient of determination (R2), both presented as mean ± standard deviation. RMSE measures the magnitude of the difference between the model's predicted values and the actual values; a smaller RMSE indicates higher prediction accuracy, suggesting smaller errors in data fitting by the model. R2 reflects the extent to which the model explains the variability in the data; a higher R2 indicates stronger explanatory power of the model regarding the inherent changes in the data, i.e., higher consistency between the model's predictions and the actual data. The results in the table reveal differences in data fitting among the models, with the SAFL showing the highest predictive accuracy.
ALMMo = autonomous learning multimodel; SAFL = self-adaptive fuzzy learning; BRR = Bayesian ridge regression; RF = random forest; SVM = support vector machine; GLMTree = generalized linear model-based tree; MLP = multilayer perceptron; RNN = recurrent neural network; LSTM = long short-term memory.
Results of Comparison for Explainability among XML, ML with Post-hoc Explanatory Features, and Statistical Modeling
Results of Explainability Utilizing EFS-Based XML
Means, standard deviations, and correlations among the study variables are given in Appendices E and F (Supplemental materials). In the main analyses, we examined the relationship between each independent variable and the outcomes of all local models (clusters) identified by the EFS-based method (ALMMo and SAFL) based on the training data. We followed the approach of prior studies in XML literature (e.g., Yong et al., 2018), first identifying the characteristics of each cluster based on the premise parameters. Table 5 summarizes the characteristic explanations and difference types for each predictor to facilitate the interpretation of the identified clusters. Subsequently, each consequent parameter represents a predictor–outcome relationship in a cluster. Furthermore, conclusions are drawn based on the average of multiple consequent parameters derived from all clusters in the broader XML. Note that each average consequent parameter is similar to a regression coefficient in statistical modeling, indicating the overall predictor–outcome relationship learned by EFS. This is one of the key features of the EFS-based XML, which allows researchers to interpret estimated relationship weights and draw conclusions directly.
Characteristic Explanation and Difference Types of Predictors.
In our research, the hyperparameters of ALMMo are set as:
Premise Parameters of ALMMo Learned from Training Data.
Note. Each row in the table represents the center of an independent local model, namely, the arithmetic mean of data samples associated with the local model.
ALMMo = autonomous learning multimodel.
Consequent Parameters of ALMMo Learned from Training Data.
Note. Each row in the table represents an independent local model, with the parameters included indicating the relationships between predictors and outcomes within a local model. The “Average” row provides the mean values of the parameters across all local models, thereby assessing the average influence of each factor within the model. We regard the absolute value of the average parameter as indicating an irrelevant predictor when it is less than .010.
ALMMo = autonomous learning multimodel.
Premise Parameters of SAFL Learned from Training Data.
Note. Each row in the table represents the center of an independent local model, namely, the arithmetic mean of data samples associated with the local model.
SAFL = self-adaptive fuzzy learning.
Consequent Parameters of SAFL Learned from Training Data.
Note. Each row in the table represents an independent local model, with the parameters included indicating the relationships between predictors and outcomes within a local model. The “Average” row provides the mean values of the parameters across all local models, thereby assessing the average influence of each factor within the model. We regard the absolute value of the average parameter as indicating an irrelevant predictor when it is less than .010.
SAFL = self-adaptive fuzzy learning.
The premise parameter results based on the ALMMo method are shown in Table 6. Cluster 1 shows that working time (0.765) is the most significant feature, indicating that players with longer playing time are likely the main or core players on the team. Height (0.540), inequality of the number of starts (0.533), working amount (0.530), and weight (0.450) are the next important features, which reflects that cluster 1 may represent “inside main player,” like power forwards. Similarly, cluster 2 represents “outside main veterans,” like shooting guards or small forwards. Cluster 3 represents “experienced inside rotation players,” like centers. Cluster 4 represents “inside bench players,” like power forwards. Cluster 5 represents “inside main veteran,” like power forwards or centers. Cluster 6 represents “outside bench veterans,” like swingmen. Cluster 7 represents “key outside rotation veterans,” like shooting guards. Cluster 8 represents “outside main scoring players,” like shooting guards or offensive small forwards.
The consequent parameter results for clusters 1–8 in Table 7 collectively indicate that self-serving behavior and team-serving behavior positively influence job performance, while working amount shows a strong negative influence. Simultaneously, predictors corresponding to some relatively important features have a minimal impact on job performance, nearing zero. For example, in cluster 1, working time (0.02), height (0.004), and inequality of the number of starts (0.004) have little effect on job performance. Most importantly, considering the average consequent parameters of each predictor–outcomes relationship in the eight clusters (as shown in Table 7), self-serving behavior (average = .406, range from .378 to .472), coworker-serving behavior (average = .062, range from .004 to .081), team-serving behavior (average = .160, range from .124 to .179), average age of all players in a team (average = .033, range from .024 to .050), age of joining the team (average = .027, range from −.039 to .048), uncooperative behavior (average = .012, range from −.046 to .040), working time (average = .018, range from .011 to .023), and player weight (average = .028, range from .003 to .061) positively predict job performance. Whereas team cooperation (average = −.017, range from −.030 to −.008), working amount (average = −.303, range from −.356 to −.278), and player height (average = −.014, range from −.128 to .028) negatively predict job performance. Moreover, team performance (average = .001, range from −.054 to .028), average tenure of team member (average = .005, range from −.013 to −.073), inequality of team members’ playing time (average = −.007, range from −.045 to .011), inequality of the number of starts (average = .009, range from .002 to .018), and player age (average = −.007, range from −.020 to −.001) were poor predictors of job performance, all having very small averages.
The premise parameter results based on the SAFL method are shown in Table 8. Cluster 1 represents “inside main players,” like power forwards or centers. Cluster 2 represents “outside rotation players,” like shooting guards or offensive small forwards. Cluster 3 represents “outside main veteran,” like swingmen. Cluster 4 represents “outside bench players,” such as shooting guards or offensive small forwards. Cluster 5 represents “experienced inside rotation players,” like centers. Cluster 6 represents “key outside rotation players,” like offensive small forwards. Cluster 7 represents “inside bench veterans,” like power forwards or centers. Cluster 8 represents “outside rotation veterans,” like swingmen.
Additionally, the consequent parameter results for clusters 1–8 in Table 9 similarly show that self-serving behavior and team-serving behavior have a strong positive impact on job performance, while working amount exhibits a strong negative impact. Furthermore, it was found that the predictors with less impact on job performance were fewer in number. According to the average consequent parameters of each predictor–outcomes relationship in the eight clusters (as shown in Table 9), self-serving behavior (average = .583, range from .292 to .948), coworker-serving behavior (average = .049, range from −.108 to .157), team-serving behavior (average = .254, range from .104 to .509), average age of all players in a team (average = .011, range from −.081 to .085), average tenure of team member (average = .024, range from −.122 to .285), inequality of team members’ playing time (average = .051, range from .030 to .326), team cooperation (average = .040, range from −.034 to .255), age of joining the team (average = .049, range from −.005 to .159), working time (average = .022, range from −.008 to .043), and player height (average = .022, range from −.089 to .150) positively predict job performance. However, team performance (average = −.021, range from −.163 to .032), inequality of the number of starts (average = −.015, range from −.144 to .045), uncooperative behavior (average = −.023, range from −.160 to .083), working amount (average = −.408, range from −.698 to −.215) negatively predict job performance. It is worth noting that only player weight (average = −.005, range from −.202 to .073) and player age (average = .004, range from −.015 to .061) were poor predictors of job performance, with very small averages.
Tables 7 and 9 show that SAFL has superior learning capabilities compared to ALMMo, enabling it to more accurately identify a greater number of predictive factors that affect job performance. For instance, “team performance” and “inequality of the number of starts” can positively predict job performance, whereas “average tenure of team members” and “inequality of team members’ playing time” predict negatively.
Results of Explainability Utilizing ML with Post-hoc Explanatory Features
As we previously clarified, the explainability of traditional ML can be improved by using the post-hoc explanatory features. Despite acknowledging limitations of post-hoc models (e.g., they are not aligned to human understanding), it is essential to include them in our analysis to allow a comprehensive comparison. To compare and validate the explainability and results of the EFS-based XML, we have chosen the widely applicable SHAP as a post-hoc model to explain the results of BRR and RF (Lundberg & Lee, 2017), which exhibit good predictive performance and are amenable to interpretation. The importance of the features in the two models is depicted in Figure 1 in terms of SHAP values. The average consequent parameter values of the IF–THEN rules learned by ALMMo and SAFL from the training data with the optimal hyperparameters are depicted in Figure 2, which shows that the first (self-serving behavior), second (coworker-serving behavior), third (team-serving behavior), and thirteenth (working amount) features (i.e., predictors) play the most critical roles in the decisions of both ALMMo and SAFL.
Shapley additive explanations (SHAP) values of different models: (a) Bayesian ridge regression (BRR) and (b) Random forest (RF).
Average consequent parameter values of autonomous learning multimodel (ALMMo) and self-adaptive fuzzy learning (SAFL).
By comparing Figures 1 and 2, it is evident that the explanations provided by SHAP are almost identical to the explanations provided by ALMMo and SAFL (in the form of feature importance). All four models involved in this visual example mostly agree on the importance of specific features and the relative importance of different features vis-à-vis their weights. However, one may notice that SHAP can only evaluate the overall importance of different features on the models’ decisions. In contrast, ALMMo and SAFL not only provide the overall importance of the features on the predictive model, but also give the importance of each feature on every local model as shown in Tables 5–7. Importantly, explanations provided by ALMMo and SAFL are derived naturally from data due to their inherent explainability without involving an additional model. This example demonstrates the advantages of the proposed pre-hoc XML over post-hoc XML.
Results of Explainability Utilizing OLS in Statistical Modeling
We applied OLS regression, widely regarded as a foundational method of statistical modeling, to examine all predictors of job performance that are included in ML. As shown in Table 10, while the model's overall fit does not surpass that of SAFL, ALMMo, and several ML models (e.g., BRR and RF), but exceeds SVM. With an R2 value of .332, which indicates the model based on OLS regression explains less than half the variability in job performance. This result underscores that XML outperforms OLS regression in the performance of explainability due to better model fit. The enhanced data fit provided by XML can be ascribed to its methodology of establishing multiple local models, thereby capturing the nuances within the dataset more effectively. Especially, as previously elaborated, statistical modeling encompasses numerous model assumptions. During our modeling process, we identified two variables (i.e., self-serving behavior and working amount) with exceptionally high variance inflation factors (>5), indicating potential multicollinearity issues. Consequently, these two predictor variables were excluded from the final model. In contrast, ML is less constrained by such assumptions.
Results of OLS Regression for Job Performance.
Note. The estimates refer to the regression coefficients, reflecting the magnitude of impact each predictor has on job performance. The standard error (SE) provides precision of the estimates, confidence intervals (CI) show the reliable range of the estimates, and the p-values test the statistical significance of each predictor, *p < .05, **p < .01, ***p < .001. The R2 value reflects the overall model's ability to explain the variability in the data.
OLS = ordinary least squares.
Regarding independent variables, both coworker-serving and team-serving behaviors were strongly linked to job performance, evidenced by estimates of .281 (p < .001) and .495 (p < .001), respectively. Moreover, the average age of all team players was positively correlated with job performance, as demonstrated by estimates of .139 (p = .034). Uncooperative behavior and player age were inversely related to job performance, with an estimate of −.194 (p = .001) and −.086 (p = .037), suggesting a detrimental effect of increased player uncooperative behavior and age on performance. Other variables did not exhibit statistical significance in this context. These results are largely consistent with results using EFS-based XML.
Discussion
From statistical modeling to XML, methods of scientific research have undergone a transformation from simple to complex, ultimately striving to seek transparency in complex models. This methodological evolution presents unique challenges and opportunities to organizational research. The rise of XML overcomes the limitations of traditional methods (such as statistical modeling and traditional ML), promoting the development of new methods and paradigms for organizational research. In light of this, we have focused on a cutting-edge EFS-based XML (including SAFL and ALMMo), which enhances model explainability by clearly demonstrating the contribution of each predictor. Through analytically and empirically comparing XML with statistical modeling and traditional ML, we highlight the advantages of EFS-based XML. In what follows, we discuss the methodological and practical implications of XML and provide best-practice recommendations for using XML in future organizational research.
Methodological and Practical Implications
The application of XML has the possibility of making a significant contribution to organizational research methods. First, XML bridges the explanatory power of statistical modeling with the predictive accuracy of ML. It effectively handles large-scale, heterogeneous, and complex data, fostering a balance between predictability and explainability while making few assumptions. The introduction of EFS-based XML as a pre-hoc explanatory approach, particularly its advanced forms such as SAFL, can significantly enhance the transparency and credibility of models. By being mapped onto a natural human decision-making process, EFS-based XML can make accurate predictions using complex data in organizational research, and present the basis for decision-making with it by clearly displaying the contribution of each predictor. This provides organizational researchers with a powerful tool to better understand and predict dynamics within and outside of organizations in explainable ways. Thus, XML offers a new direction for the method development of organizational research, providing new tools for addressing complex organizational problems. This also responds to the call of Miller (2019) for exploring and enhancing the explainability of ML in the social sciences generally.
In today's data-driven research environment, choosing between explanation model and prediction model is crucial for generating deep insights alongside reliable predictions. Through a systematic comparison of three analytical paradigms, including statistical modeling, traditional ML, and XML, in terms of methodological principles and empirical analyses, our study aims to provide organizational researchers with guidance for method selection. Our comparative results reveal that traditional ML primarily focuses on maximizing predictive performance, yet at the cost of explainability. This finding aligns with Doornenbal et al. (2022), which indicates that traditional ML surpasses statistical modeling in predictive performance but loses explainability. In organizational research, a clear understanding and explanation for processes and outputs of modeling is essential for theory development; the lack of explainability in traditional ML could lead to critical decision-making bias by not knowing which variables are most important or how they relate to important outcomes. In contrast, XML not only demonstrates similar or even superior performance in terms of prediction accuracy, they also address the issue of explainability. Our comparative analyses provide organizational researchers with guidance in method selection, enabling them to make informed choices based on their research objectives and needs.
From the perspective of integrating theory and practice, our work provides a new opportunity to bridge the gap between management theory and practice (Cronin et al., 2021; Ledford et al., 2016). Specifically, in management research, the emphasis on theory testing and explanation often leads researchers to prioritize theoretical contributions at the expense of the practical implications of their studies, creating a divide between theory and practice (Torres, 2024). In terms of data modeling, organizational studies tend to focus on using statistical models to test the explanatory mechanism based on theories, but overlook the prediction accuracy for future management practice (Cronin et al., 2021). Alternatively, in practical management contexts, practitioners are more concerned with how models can effectively predict future changes to assist in managerial decision-making. This disconnection between theory and practice is a bottleneck in making organizational research relevant. The application of XML, conveniently, provides organizational researchers with additional tools that can help bridge this gap. It enables scholars to better understand data and simultaneously test the predictive performance of models, thereby more effectively translating theoretical knowledge into practical actions and strategies. In other words, XML represents a potential approach to integrating theory and practice.
Best-Practice Recommendations for Future Research
With the insights from our research, we now outline a set of best-practice recommendations for using XML in organizational research. Table 11 outlines six steps, essential action details, recommended considerations, and illustrative implementation of XML. This guide aims to deepen the understanding of XML and to promote its application in future research.
Best-Practice Recommendations for Future Research.
Note. MSE = mean squared error; RMSE = root mean squared error; MAE = mean absolute error; NBA = National Basketball Association; XML = explainable machine learning; EFS = evolving fuzzy system; SAFL = self-adaptive fuzzy learning; ALMMo = autonomous learning multimodel; ML = machine learning; BRR = Bayesian ridge regression; RF = random forest; SHAP = Shapley additive explanations; OLS = ordinary least squares; AUC = area under the curve; ROC = receiver operating characteristic curve.
Step 1: Goal Setting
Setting clear goals is a crucial initial step in any research practice. Researchers must accurately define their predictive goals, which will guide their selection of a variable set related to these goals to construct an overall research framework. The starting point of this process involves a detailed analysis of the research background and data availability to ensure sufficient relevant data and variables are selected. By integrating existing literature and theories, researchers need to determine whether they are dealing with classification, regression, or clustering problems in their research. This directly influences the data analysis methods used and the final model selection. In our study, we focus on constructing an XML to predict employee work performance. After synthesizing results from existing research, we proposed a set of predictors impacting employee performance and used XML to provide in-depth insights but also high levels of predictability.
Step 2: Data Preparation
Data preparation is fundamental in establishing the model's accuracy and credibility. This step primarily involves data preprocessing, which emphasizes comprehensive and accurate data collection to ensure the dataset covers all relevant variables without omitting critical information. For this, researchers need to identify and appropriately handle missing or anomalous values, which may involve using statistical methods for estimation or exclusion, and normalizing the data as necessary to ensure the effectiveness of subsequent analyses (Little & Rubin, 2019; Schafer, 1997). For instance, in our example, we collected a dataset of NBA players’ performance and its predictors and excluded all records with missing data.
Data splitting, another crucial aspect of data preparation, aims to rationally divide the dataset into training and testing data. This process requires considering the volume and complexity of the data to determine an appropriate split ratio. Importantly, researchers must avoid data leakage by ensuring that information from the testing data is not inadvertently used during the model training process. In our case, one year data was used as the training data, while data in another year served as the testing data to evaluate the model's predictive performance after training.
Step 3: Choosing XML Model
Selecting the appropriate model is crucial for conducting successful XML research. Researchers require to choose a suitable XML based on the specific requirements of the predictive task and data characteristics. For model selection, considering the compatibility of the model complexity with the data type and structure is the primary task. For instance, when prediction models are not particularly complex, traditional prediction models like GLMTree, where each cluster (or leaf node) is distinctly described by a unique combination of splitting variables, can be quite effective, particularly when the tree depth is limited, allowing for clear interpretation of the interactions among these variables (Zeileis et al., 2008). Employing these traditional models alone is feasible under such conditions. However, as the depth of the tree increases, the task of tracking specific variable combinations that define clusters becomes increasingly challenging. The complexity of interactions between splitting variables makes it difficult to comprehend the precise characteristics of an individual cluster without considering its relationship with other clusters within the tree structure. Consequently, the interpretability of a GLMTree with a deep structure is significantly difficult. In the context of dealing with complex data and model specification, researchers may need to turn to more sophisticate XML to capture multilayered data features and provide deeper insights. In XML (e.g., SAFL), each cluster captures a distinct local data pattern and is represented by a center that probabilistically has the highest likelihood of belonging to that cluster. These cluster centers are concise summaries of the key statistical features of the clusters they represent. By examining these cluster centers, it is feasible to interpret the local patterns exhibited by these clusters without the need to explicitly consider their interrelationships. In our illustrative study, we employed two EFS-based XML, including the SAFL and ALMMo, which are capable of handling complex data relationships and providing clear and intuitive explanations about the factors affecting employee performance, offering valuable insights for managerial decision-making.
Step 4: Model Training
In the core process of predictive model construction, model training is used to ensure that the predictive model accurately captures data patterns. This step involves applying the selected XML and other comparative ML to the training data and then comparing their performance. To enhance the generalizability of model and ensure accurate predictions for unknown data, fine-tuning the hyperparameters of model using K-fold cross-validation has become standard practice. Ten-fold cross-validation, as a widely adopted approach, reduces the likelihood of model overfitting by dividing the data into 10 subsets, using nine for model training and the remaining subset for performance evaluation. Although increasing the number of folds in cross-validation can further stabilize the model, it inevitably leads to a higher computational burden. In this study, we trained the SAFL and ALMMo, along with eight other ML, using the training data. We fine-tuned their parameters through 10-fold cross-validation to achieve the best performance.
Step 5: Model Evaluation
After the completion of model training, the model evaluation phase is crucial for verifying the effectiveness of the model. The essence of this step lies in assessing the performance of the selected XML and other ML to reveal their generalizability when dealing with unknown data. Currently, a range of appropriate evaluation metrics can provide quantitative measures of a model's predictive power. In classification tasks, this includes metrics such as accuracy, precision, recall, F1 score, and area under the curve (AUC)–receiver operating characteristic curve (ROC) curve; for regression tasks, it encompasses metrics like R2, mean squared error, RMSE, and mean absolute error (Chai & Draxler, 2014). These metrics allow us to compare the predictive performance of different models in a fair and transparent manner. In our study, RMSE and R2 were used to evaluate predictive performance across 10 models, including both traditional ML and XML. The combination of these two metrics showed that XML had a better prediction performance than ML.
Step 6: Explanatory Analysis
Explanatory analysis plays a pivotal role in enhancing model transparency and facilitating the understanding of researchers or practitioners. The core of this step lies in showing and interpreting the relationships between predictors and outcomes. To effectively demonstrate this process, the use of charts and visualization tools is important. These tools translate complex data structures and model computations into intuitive and easy-to-understand visual formats, thereby aiding users in quickly grasping the most important features of a model.
In fields requiring precise decision support, such as organizational research, this in-depth analysis is particularly vital. It not only deepens an understanding of the model's decision-making process but also helps to establish trust in the prediction performance of a model (Martens et al., 2016). Furthermore, comparing the explainability of an XML with other models highlights its relative advantages in different application scenarios, further enhancing the model's practical value.
In our research, we provided detailed reports of the interpretation results of SAFL and ALMMo models, displaying the contribution of each predictor to employee performance using tables and bar charts. By comparing these results with the interpretations of the BRR and RF models based on SHAP values and the OLS regression model of statistical modeling, the robust explanatory framework of SAFL and ALMMo was demonstrated. This approach precisely quantifies the marginal contributions of individual features to model predictions, illustrating their superiority for constructing interpretable models.
Limitations
Our study is subject to several limitations. In terms of method, we used XML to construct a prediction model of employee performance, but we did not do this in a way that should necessarily be interpreted causally—along the way we avoided causal language for this reason. Consistent with typical ML and exercises in statistical modeling, XML should only be used to make causal inferences when characters of data allow this. Our prediction model demonstrates a good fit to the data, including for out-of-sample predictions in the testing data. That is, the data from professional basketball players provide valuable insights into performance, but these results may not be applicable to non-sport professional environments. Therefore, there are limitations when generalizing our findings to different types of organizations or industries. By conducting empirical studies in other industries and professional environments using XML, researchers can gain a more comprehensive understanding of the applicability and effectiveness of XML for predicting job performance.
Conclusions
Recent XML research has demonstrated exceptional performance in both natural and social sciences (Baryannis et al., 2019; Senoner et al., 2022). XML enables researchers to obtain not only accurate predictive results but also to understand the underlying mechanisms driving these outcomes. In our study, we introduce EFS-based XML as a useful explainable method. The purpose of EFS-based XML is to provide in-depth explanations of model feature importance while maintaining predictive performance, which is crucial for understanding the modeling process. We have shown the methodological utility and potential applications of EFS-based XML in organizational research by exploring its foundations and providing a comparative empirical illustration of its applications. We hope our work stimulates more organizational research using XML to make the field more predictively useful while optimizing the explainability of its results.
Supplemental Material
sj-docx-1-orm-10.1177_10944281251323248 - Supplemental material for Shedding Light on the Black Box: Integrating Prediction Models and Explainability Using Explainable Machine Learning
Supplemental material, sj-docx-1-orm-10.1177_10944281251323248 for Shedding Light on the Black Box: Integrating Prediction Models and Explainability Using Explainable Machine Learning by Yucheng Zhang, Yuyan Zheng, Dan Wang, Xiaowei Gu, Michael J. Zyphur, Lin Xiao, Shudi Liao and Yangyang Deng in Organizational Research Methods
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the National Natural Science Foundation of China (Grant Nos. 72272048, 72432003, 71902164, and 71972065).
Supplemental Material
Supplemental material for this article is available online.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
