Abstract
Background
Cardiac surgery-associated acute kidney injury (CSA-AKI) is related to increased morbidity and mortality. However, limited studies have explored the influence of different feature selection (FS) methods on the predictive performance of CSA-AKI. Therefore, we aimed to compare the impact of different FS methods for CSA-AKI.
Methods
CSA-AKI is defined according to the kidney disease: Improving Global Outcomes (KDIGO) criteria. Both traditional logistic regression and machine learning methods were used to select the potential risk factors for CSA-AKI. The area under the receiver operating characteristic curve (AUC) was used to evaluate the performance of the models. In addition, the importance matrix plot by random forest was used to rank the features' importance.
Results
A total of 1977 patients undergoing cardiac surgery at Fuwai hospital from December 2018 to April 2021 were enrolled. The incidence of CSA-AKI during the first postoperative week was 27.8%. We concluded that different enrolled numbers of features impact the final selected feature number. The more you input, the more likely its output with all FS methods. In terms of performance, all selected features by various FS methods demonstrated excellent AUCs. Meanwhile, the embedded method demonstrated the highest accuracy compared with the LR method, while the filter method showed the lowest accuracy. Furthermore, NT-proBNP was found to be strongly associated with AKI. Our results confirmed some features that previous studies have reported and found some novel clinical parameters.
Conclusions
In our study, FS was as suitable as LR for predicting CSA-AKI. For FS, the embedded method demonstrated better efficacy than the other methods. Furthermore, NT-proBNP was confirmed to be strongly associated with AKI.
Keywords
Introduction
The incidence of cardiac surgery-associated acute kidney injury (CSA-AKI) varies between 7% and 40%, depending on the definition, patient characteristics, and the type of cardiac surgery. 1 This common and serious complication poses a great threat to patient outcomes 1–7 as well as the health care system, 7 and early prediction and detection of AKI can bring major benefits. 8 Currently, the Kidney Disease Improving Global Outcome (KDIGO) criteria are recommended for the diagnosis of AKI. However, these criteria are not applicable for timely prediction.5,9 Several risk scores have been tested for helpfulness in risk stratification.3,10–13 However, most of these potential risk factors in these models were analyzed with the traditional logistic regression method, which could only handle the linear relationship between variable and outcome2,4
With the booming data in electronic medical records (EMR), it is difficult to build precise models with high-dimensional data. The irrelevant and redundant features negatively affect the accuracy. 14 Machine learning, equipped with the capacity to remove redundant attributes, can improve task learning efficiency and optimize the performance of predictive models. 15 Although, an increasing number of studies have explored the predictive model of CSA-AKI by machine learning, 16 there is no consensus on the appropriate feature selection (FS) method for CSA-AKI. 5
This study aimed to compare the different FS methods based on machine learning with logistic regression (LR) from three aspects: predictive performance, outcome accuracy, and feature importance.
Materials & methods
Study population
This retrospective observational study was approved by the Ethics Committee of Fuwai Hospital in Beijing, China [2022-1086]. Patients who underwent cardiac surgery with cardiopulmonary bypass at our institution from December 2018 to April 2021 were enrolled. Informed consent was exempted.
Data collection
Preoperative variables, including demographic characteristics, laboratory values, and medical and medication histories, were extracted. In addition, surgery time, CPB time, aortic clamp time, and surgery type were also extracted.
Study design
Two groups enrolled in different features were used to explore the impact of various FS methods on CSA-AKI. For group I, we included all features extracted, a more inclusive group. As for group II, we retained the reported features related to CSA-AKI.3,6,17–20 LR and ML engineering were performed simultaneously in the two groups to determine suitable variables. The whole process showed in Figure 1. The flow chart shows the whole process of our study.
AKI definition
The primary outcome was postoperative AKI, defined according to Kidney Disease Improving Global Outcomes (KDIGO) criteria, which was determined as the maximal change in serum creatinine level during the first seven postoperative days. 21 The baseline serum creatinine (SCr) level was defined as the most recent measurement before surgery, and an increase in serum creatinine by 1.5–1.9 times baseline or by ≥ 26.5 μmol/l (0.3 mg/dl) from baseline was identified as KDIGO-AKI. We did not choose decreasing urine output as our objective indicator because it is not specific and can be affected by various clinical variables; meanwhile, the definition of oliguria is not constant with changeable cutoff values for different situations. 22
Statistical analysis
Continuous variables of patient characteristics were expressed as the mean ± standard deviation or median with an interquartile range. The categorical variables of patient characteristics were reported as the frequency number and percentage. Student’s t-test or the Mann–Whitney U test was used for continuous variables, and the χ2 test or Fischer’s exact test was applied for categorical variables (p < .05 indicates statistical significance).
Feature selection and model evaluation
Logistic regression was the traditional method for detecting the association between features and AKI (p < .1 indicates statistical significance). In contrast, ML was used as a novel method to select relevant features. Generally, the feature selection based on ML can be divided into three main categories: filter, wrapper, and embedded methods
14
(Figure 2).In this study, we aimed to compare maximal information coefficient (MIC), recursive feature elimination (RFE), and random forest (RF) with LR. (1) Filter (2) Wrappe (3) Embedded Basic process of different feature selection. (a) the process of Filter, (b) the process of Wrapper, and (c) the process of Embedded.
The importance matrix plot for the RF method was used to rank the features selected. Furthermore, we used area under receiver operating characteristic curve(AUC) to evaluate the model’s performance, which was compared with the LR by the Delong test. All analyses were developed in open-source software libraries (Python version 3.9).
Results
Characteristics of the analyzed cohort
Patient characteristics and postoperative outcomes.
Outcomes of different selection methods
Features selected by ML and LR methods.
Abbreviations: BMI = body mass index; NYHA = Classification of NYHA heart function; smoke1m = smoke within a latest month; group = high or low dosage tranexamic acid; ACEI = angiotensin converting enzyme inhibitors; Temp = body temperature before surgery; HR = heart rate; SBP = systolic blood pressure; DBP = diastolic blood pressure; sd = SBP-DBP; LVEF = left ventricular ejection fraction; LVEDD = Left ventricular end diastolic dimension; MED10 = fully recovered ischemic central nervous system injury; MED11 = lacunar cerebral infarction; MED12 = non-invasive tests suggesting carotid artery stenosis >79%; MED13 = previous cardiac valvular disease; MED14 = previous coronary heart disease; MED15 = previous congenital heart disease; MED16 = previous aortic disease; MED17 = previous carotid surgery; MED18 = previous peripheral vascular disease; Intra1 = volume of intraoperative blood salvage; Intra2 = lowest temperature during CPB; Intra3 = hemoglobin at end of CPB; Intra4 = CPB again; Intra5 = surgery duration; Intra6 = CPB duration; Intra7 = aortic clamping duration; CPB = cardiopulmonary bypass; IABP = Intra-aortic balloon pump; ECMO = Extracorporeal Membrane Oxygenation.
In Group II, 41 features reported to be related to CSA-AKI were enrolled. Eventually, there were 35 features selected by the LR method,30 features selected by MIC, 31 by the wrapper (RFE), and nine features selected by the embedded (RF) method. (Table 2)
Comparison of different FS methods
In terms of AUCs, ML techniques and traditional statistical approaches demonstrated excellent predictive performance in both groups. In Group I, the features selected by LR, and MIC methods all achieved an AUC of 0.97. RFE and RF performed the AUC with 0.98 and exhibited a significant difference compared with LR (p < .01). In Group II, the LR method performed an AUC of 0.98. The MIC and RFE method achieved an AUC of 0.97. Furthermore, the RF method performed an AUC of 0.99 and showed a significant difference from the LR method (p < .01). (Figure 3) The AUC of different models in two groups. (a) group I, (b) group II.
Regarding the selection accuracy of ML methods compared with the traditional LR method, we concluded that for Group I, the accuracy of the MIC method was 72.34% compared with LR. Meanwhile, the accuracy was 91.66% for the RFE and 100% for the RF method. Furthermore, in Group II, the accuracy of the MIC method was 96.67%, 95.54% for the RFE, and 100% for the embedded method.
Identification of significant features
Top 10 features of different FS methods.
The importance matrix plot of all FS in group I. (a) Filter, (d) Wrapper, and (e) Embedded. The importance matrix plot of all FS in group II. (d) Filter, (e) Wrapper, and (f) Embedded.
Discussion
In this retrospective cohort study, we compared LR and ML techniques to select potential risk factors for AKI. Firstly, we concluded that different enrolled numbers of features impact the final selected feature number. The more inputs, the more outputs with all FS methods. Secondly, as for the performance, all selected features by various FS methods demonstrated excellent AUCs. Meanwhile, the embedded method demonstrated the highest accuracy compared with the LR method, while the filter method showed the lowest accuracy. Thirdly, regarding the importance ranking of features, our results confirmed some features that previous studies have reported and found some novel clinical parameters.
Tremendous research has shown that ML outperformed LR for the prediction of AKI.2,4,24,25 However, a recent meta-analysis reported that ML algorithms were comparable to regression models in developed models. 26 Researchers used traditional logistic regression or Cox regression to explore the potential risk factors for AKI.1,6,17,27–29 Recently, some studies found other features might influence the progression of AKI [17-19, 28, 30, 31], yet there is a lack of sufficient multicenter evidence to support their findings. The performance of different FS methods in predicting CSA-AKI remains unknown. Hence, we included as many features as possible in Group I, and 25 more were not commonly recognized before compared with Group II. We used LR and ML approaches to select the relevant features in the two groups. We identified that the more you input, the more likely its output with all FS methods. In addition, the number of features selected by ML methods was less than LR methods, which might promote establishing an easier and more effective predictive model. Meanwhile, the number of selected features by the embedded method is smaller than the filter and wrapper method, which may be attributed to the wrapper and the embedded method are based on the filter method and are better at handling and processing data. 30
Our study showed the AUCs between FS and LR methods were significant statistically. Still, all performances demonstrated excellent, which implies that FS and LR methods can be performed comparably well in selecting predictors of CSA-AKI. Furthermore, no matter dealing with a lower or higher dimensional dataset, the embedded method performed better than other methods, which might be attributed to the feature subset search process of the embedded method incorporated into the classifier training process. Notably, the AUCs of all FS methods in our study were beyond 0.9. Perhaps this is because we included serum creatinine (SCr) in our analysis, which is an important indicator of the definition of KDIGO-AKI. However, Koyner Carey et al. found that the algorithm for predicting severe AKI did not change significantly after excluding the SCr variable. 24 Future studies should investigate the complex connection between baseline SCr and postoperative AKI. Additionally, we included many intraoperative variables that might improve the performance of our models.
In terms of the selection accuracy of the ML methods compared with the LR method, our analysis indicated that the embedded method achieved the highest accuracy despite the number of the features inputted. In contrast, the filter method achieved less accuracy. It may be due to the wrapper, and embedded methods have a built-in algorithm, in which the feature selection process and algorithm training are performed simultaneously. Therefore, their results are more accurate and reliable. 30
According to the importance matrix plot, the most influential factor was NT-proBNP in all FS methods. The relation between NT-pro BNP and CSA-AKI may be due to the “cardio-renal” syndrome, which is out of the analysis of our study. 31 In addition, the laboratory values such as PT, BUN, total and direct bilirubin, and features to evaluate left ventricular function were found to have a higher potential impact on the progression of CSA-AKI. In addition, preoperative hemodynamic variables such as heart rate and systolic blood pressure were ranked in the top 10 in our cohort, suggesting better managing these features before surgery might benefit the patient. Abundant evidence denotes that intraoperative variable, including surgery and the CPB procedure, are closely associated with postoperative AKI.3,4 In one single-center cohort, patients at low risk of AKI were reclassified as high risk after including intraoperative variables. 32 We also confirmed that CSA-AKI was associated with the aortic clamp time, CPB, and surgery time. In addition, we found that the preoperative use of β-blockers might help mitigate AKI. However, it is still controversial whether pharmacological interventions are beneficial for high-risk patients.33,34 Additionally, the top 10 features included were almost the same between two groups in different methods. However, the importance ranking of these features were different which may attribute to the interaction between features. when more features are enrolled, their importance percentages will change. Furthermore, some features such as ALT, and the volume of intraoperative blood salvage, which were statistically important, were only detected by ML methods. Additionally, Lee, Hofer et al. found that a hybrid of FS and LR could perform comparably with deep neural network. This implies that FS and LR could be combined in future research, as FS could reduce the number of parameters, decrease the learning time, and avoid the problems of dimensionality; meanwhile LR could output explainable variables with low computational cost, 35 which need more investigation when faced with booming data and high dimensional statistics.
Our study also has several limitations. First, this is a retrospective analysis with single-center data and a relatively small number of cases. The performance of machine learning algorithms might be different for a larger dataset with a different distribution of patient characteristics in different institutions. Second, the most important variables are not clinically modifiable, and whether our results could benefit high-risk patients is unknown. Nevertheless, further prospective trials are imperative to evaluate whether the adjustment of modifiable predictors could yield beneficial results. But we have confirmed the importance of the intraoperative variables. Third, we did not include some biomarkers, such as cystatin C, tissue inhibitor of metalloproteinases 2 (TIMP-2), and insulin-like growth factor-binding protein 7 (IGFBP7), 36 which have been reported to have high specificity and sensitivity, irrespective of potentially interfering conditions in our analysis. Further investigation could explore whether these biomarkers could help better detect postoperative AKI.
Conclusions
In conclusion, ML was as suitable as LR for selecting potential risk factors for CSA-AKI. For ML, the embedded method demonstrated better efficacy than the other methods. Furthermore, NT-proBNP was confirmed to be strongly associated with AKI.
Footnotes
Author contributions
QL and JJ S contributed to manuscript preparation. HL, and YY C participated in the analysis and interpretation of the data, CH Z and JS contributed to reviewing and editing, CH Z contributed to visualization, and JS contributed to supervision and administration of the project. All authors read and approved the final manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (no. 81970290).
