Abstract
Background
Complex machine learning (ML) models have revolutionized predictions in clinical care. However, for laparoscopic colectomy (LC), prediction of morbidity by ML has not been adequately analyzed nor compared against traditional logistic regression (LR) models.
Methods
All LC patients, between 2017 and 2019, in the National Surgical Quality Improvement Program (NSQIP) were identified. A composite outcome of 17 variables defined any post-operative morbidity. Seven of the most common complications were additionally analyzed. Three ML models (Random Forests, XGBoost, and L1-L2-RFE) were compared with LR.
Results
Random Forests, XGBoost, and L1-L2-RFE predicted 30-day post-operative morbidity with average area under the curve (AUC): .709, .712, and .712, respectively. LR predicted morbidity with AUC = .712. Septic shock was predicted with AUC ≤ .9, by ML and LR.
Conclusion
There was negligible difference in the predictive ability of ML and LR in post-LC morbidity prediction. Possibly, the computational power of ML cannot be realized in limited datasets.
Keywords
Key Takeaways
• Machine learning is an average predictor of morbidity after laparoscopic colectomy. • Machine learning (ML) does not predict morbidity better than logistic regression. • Simple, static surgical datasets may limit ML predictive capability.
Introduction
As colectomies represent a significant portion of general surgery procedures, colectomies are responsible for a great percentage of complications observed within the operative space. 1 Over the past 20 years, surgeons have shifted to laparoscopic colectomy (LC), which has been associated with decreased complication rates, shorter hospital stays, and lesser overall costs, when compared to open colectomy.2-5 Despite these benefits, LC post-operative morbidity remains high, affecting up to one-fifth of patients. 4 In the ongoing pursuit of predicting (and possibly preventing) adverse outcomes, classical statistical models, such as linear and logistic regression (LR) functions, have been extensively explored for colorectal procedures with variable success.6-8
Recently, computational power and complex analytical expertise has allowed artificial intelligence (AI) to enter into the medical field, claiming improved predictability of outcomes. Although simple, cost-effective, and computationally non-intensive, classical statistical models lack the sophistication that fuel machine learning (ML) algorithms, which are a subset of AI. ML models can constantly reiterate and revise their underlying equations as they gather more data, allowing better capture of intricate, non-linear relationships between multiple variables, which may closer reflect the true complexity of human disease. In contrast, classical statistical models do not dynamically change with data, as they instead base all predictions on immutable equations and assume that all variables interact in a linear, additive manner. However, these theoretical benefits of ML are far from proven for prediction of post-operative surgical morbidity.
As ML methods are published with increased frequency on nearly every type of patient and procedure, we sought to understand if ML models predict post-operative surgical morbidity with higher accuracy than traditional statistical techniques. To examine this question, we chose to retrieve data on LC patients from a national surgical database: the American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP). The ACS-NSQIP is a large sample set representing multiple institutions around the country; it is nationally validated and has produced results that have bettered the quality of surgical care, suggesting generalizable and trustworthy findings. Using this database we developed classical statistical and ML models to predict post-operative outcomes in LC patients.
Our goal was to compare the older (classical statistical) and newer (ML) predictive models. We hypothesized that ML will show superior predictive ability compared to classic statistical models.
Methods
As this study solely analyzed cases from a de-identified dataset (the ACS-NSQIP), Institutional Review Board approval was not required.
Patient Population
All ACS-NSQIP patients who underwent an LC procedure between January 1, 2017 and December 31, 2019 were included in this study. Patients with LC procedures were determined by the following CPT codes: 44204 (laparoscopy, surgical; colectomy, partial, with anastomosis), 44205 (laparoscopy, surgical; colectomy, partial, with removal of terminal ileum with ileocolostomy), and 44207 (laparoscopy, surgical; colectomy, partial, with anastomosis, with coloproctostomy).
Data Pre-Processing and Selection
Several pre-processing steps were utilized to optimally prepare the dataset for input into the prediction models, as shown in Figure 1. We leveraged Hao et al’s pipeline, which implements advanced ML-based classification methods and extracts the data’s most predictive variables without compromising model performance.
9
Below, we have outlined the modifications that we made to the pipeline, to accommodate the ACS-NSQIP dataset. Feature selection for input into ML models.
Based on the data distribution, we identified the median missing data percentage and removed variables with 90% or more missing data from the analysis. For all remaining continuous variables, the missing data was imputed by utilizing the median and thereafter were normalized. For all remaining categorical variables, missing data was handled as an additional value, supplementary to the ACS-NSQIP values. Variables which were neither categorical/continuous (eg, operative notes) nor informative for prediction of morbidity (eg, work relative value units), were removed by a neutral, senior clinical researcher who was not part of the study. Furthermore, variables whose entire entries consisted of a single value were removed.
Categorical variables were transformed to binary ones via “one-hot encoding.” This method consists of changing a categorical variable’s value into multiple binary ones, with values of 1 or 0. For example, a single variable such as “Race,” with values “White,” “African-American,” or “Asian,” is reformatted into 3 different ones: is this patient white? Is this patient African-American? Is this patient Asian? A value of 1 is assigned to the variable that correctly represents the race of the patient and a value of 0 is assigned to the remaining race-related variables. This method is responsible for the increase in variables observed in the 6th step of Figure 1.
Finally, two statistical analyses were performed to remove biased and redundant predictors. First, a variance analysis amongst the populations with and without complication was conducted. We utilized a chi-squared test for binary one-hot encoded variables and a Kolmogorov-Smirnov test for continuous ones, which both return P-values. Predictors that were statistically significant were retained. Second, a Pearson’s correlation analysis for all remaining predictors yielded those with high correlation amongst each other. For pairs of variables with correlation coefficients of .8 or higher, one variable was kept in the analysis, while the other was removed. After all pre-processing steps were completed, 88 variables remained.
Outcome Measures
The following outcomes were captured in ACS-NSQIP: deep wound infection, surgical site infection (SSI), organ/space SSI, wound dehiscence, bleeding, pneumonia, deep venous thrombosis, pulmonary embolism, myocardial infarction, cerebrovascular incident, acute renal failure, sepsis, urinary tract infection, reintubation, unplanned return to the operating room, and unplanned hospital readmission. “Any complication” was defined as the presence of one or more of these complications in a given patient. Complications were further grouped into “severe complication” (any presence of organ/space SSI, pneumonia, reintubation, pulmonary embolism, acute renal failure, stroke, cardiac arrest requiring CPR, myocardial infarction, bleeding, sepsis, unplanned reoperation, or unplanned readmission) and “non-severe complication” (any presence of superficial SSI, deep SSI, wound disruption, renal insufficiency, urinary tract infection, or deep venous thrombosis). Finally, seven of the most common complications in our target patient population were chosen to be individually analyzed by the predictive models in related sub-analyses.
Machine Learning Models
Three complex models, Random Forests (Breiman, 2001), XGBoost (Chen and Guestrin, 2016), and L1-L2 with recursive feature elimination (L1-L2-RFE), were utilized.
Random Forests (RF) is an algorithm based off a collection, or “ensemble,” of decision trees. Each tree individually classifies the outcome, after being trained on a random subset of the overall training dataset, utilizing a random subset of predictive variables. Then, the RF model combines all trees using majority voting, to attain high accuracy and generalization for the end predictive result.
XGBoost is a model that also utilizes decision trees. Decision trees are built sequentially, and high-performing ones are optimized via a gradient descent algorithm that minimizes error. The model prevents overfitting of the data through various other ML techniques, such as cross-validation and regularization.
For both decision-tree-based models, their architecture (number of trees, branches at each node, tree depth, etc.) was tuned to optimize AUC, with the use of 5-fold cross validation.
L1-L2-RFE is an augmented logistic regression model, which leverages ML-based functions, such as LASSO, Ridge regression, and recursive feature elimination, which minimize error, enhance accuracy, and avoid overfitting/underfitting.
Multivariate Logistic Regression Model
Although no definite consensus exists on which statistical model to use for prediction, regression analysis is typically one of the top choices. Given that outcomes were binary, logistic regression was chosen as a comparison prediction model. The same highly informative predictors utilized in the ML models were implemented in a multivariate LR (LR) model.
Training, Testing, Evaluation, and Optimization of the Models
Training of the models was done on 75% of the cases, while testing was performed on the remaining 25%. Performance was analyzed via area under the ROC curves (AUC).
Statistical Testing
Selected Patient Population Characteristics.
Results
Breakdown of Post-Operative Complications Observed in Patients with Complication.
Performance of Various ML and LR Models in Predicting Any, Severe, and Non-Severe 30-Day Post-Operative Complication.
Performance of Various ML and LR Models in Predicting Individual 30-Day Post-Operative Complication.
Discussion
Our study found that ML did not outperform LR in prediction of 30-day post-operative morbidity after LC. Furthermore, ML and LR, across all groupings of complications and individual outcomes, largely performed similarly and sub-optimally. Supporters of ML will argue that ML does show promise in predicting morbidity, as septic shock and bleeding were predicted with high AUC (Table 4). Although partially correct, this argument fails to account for the fact that LR also predicted these outcomes with similar AUC. Therefore, no significant advantage of ML over LR was attained. This phenomenon could be due to the pre- and intra-operative related variables found within the NSQIP dataset. The predictive ability of the selected variables from the pipeline, was perhaps very strong for septic shock and bleeding, but not for all other outcomes analyzed. Thus, the method by which they were analyzed (ML or LR) was likely insignificant, as they were innately strong predictors for only certain complications.
Most ML-based predictive models using NSQIP data have similarly reported average predictive results, yet a beckoning call for ML integration into clinical practice can be continuously noted across the literature. Not only do these manuscripts report AUC’s of .6-.8 for ML/LR models, but also find ML and LR models to perform very similarly.10,11 Even when supposedly advanced proprietary ML models are used for prediction, the reported AUC’s rarely exceed the .85 to .87 range.12,13 As a result, caution against pleas for ML integration into surgical practice should be considered, as life-saving decisions are difficult to make with confidence based on algorithms that correctly predict the desired outcome with 60% to 80% accuracy.
However, optimistic dreams for AI, functioning by the surgeon’s side and augmenting decision-making, should not vanish. In contrast to the surgical world, AI has performed exceptionally well in other branches of medicine. In pathology, convolutional neural networks discern between different grades of diabetic retinopathy with high accuracy. 14 In genetics, deep learning models can use histopathology slides to screen patients quickly and efficiently for specific genetic mutations. 15 In cardiology, IBM’s Watson, a tool powered by ML, natural language processing, and big data analytics, can detect heart failure one to two years earlier than typical diagnosis, utilizing comprehensive electronic health record (EHR) data. 16 A core theme amongst all successful AI endeavors in medicine is granular, informative, big datasets that allow AI algorithms to learn dynamically and make intelligent connections utilizing the data.
Our study found that the NSQIP, like many other surgical datasets, has a substantial amount of missing data for LC patients; 53.6% of the ACS-NSQIP variables had over 90% of their entries empty and only 88 of 108 one-hot encoded predictor variables were found to be high-quality, informative predictors after adjusting for collinearity and variance. Clinically relevant variables regarding alcohol use, prior recent operation, and chemotherapy were unfortunately excluded in the analysis, as the relevant fields remained vacant in the database. This lack of data maintenance is consistent with the surgical literature, where one study showed that their sub-population of NSQIP had missing data in 93% of patient records, and that different methods of handling missingness consistently produced different odds ratios for the variables selected as significant predictors of morbidity. 17
Along similar lines, inadequate selection of variables, when designing a dataset, may not provide an accurate description of the patient. What is really the predictive power of a single blood pressure or heart rate value—typically the first one—when it may change the next minute? These databases typically include a single—and often irrelevant—value of a variable that supposedly represents dynamic human physiology.
On the other hand, ML-based, electronic health record (EHR) studies have reported better predictive results than their ML-based NSQIP counterparts.18-20 As EHR’s depict patient hospital stay with higher granularity and accuracy than NSQIP, ML models thrive. A push for creating bigger datasets, filled with EHR structured data for each patient, that ML could then extract multiple inferences from, should be pursued. The most powerful of tools in artificial intelligence are only as good as the data they are supplied. It is irrational to expect accurate and precise prediction of adverse events, when the data provided to these algorithms is limited and incomplete. Surgeons may realize better predictive results if they utilize well-structured, informative EHR data over old-fashioned, traditional databases.
Beyond the quality of the data inputted into the ML models, our study was limited by certain factors, such as the complexity of model tuning performed. Model tuning consists of testing various parameters of the model architecture to best optimize ML performance for a given dataset. This process has varying degrees of complexity, where the trade-off for better model performance is time-intensive computational power. We plan to acquire devices with increased computing power to re-train our models and fine-tune their architectures for future studies. However, our research environment reflects daily reality for most academic teams; even teams with a professional focus on predictive models do not always have state-of-the-art computers available.
In conclusion, ML models seemed to perform similarly to LR in prediction of 30-day post-operative morbidity after LC. This study does not intend to invalidate the power of modern ML tools. However, our results suggest that surgeons should be shrewd when calling for ML and AI integration in the surgical world, when new studies that examine ML predictive models have not really achieved improved results over their older classical statistical counterparts. To advance ML-based predictive models, concurrent entry of accurate and complete data needs to be attained from EHR. Until then, ML algorithms—beyond some notable exceptions—will offer mediocre results, which will reflect the weakness of the data rather than the weakness of the model.
Footnotes
Acknowledgments
We would like to acknowledge all the hardworking members who serve to upkeep the ACS-NSQIP database.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
