Predicting Psychiatric Readmission of Children and Adolescents Using Machine Learning

Abstract

Introduction:

Recurrent psychiatric hospitalizations in youth are costly and burdensome to patients and families. While postdischarge planning rarely stratifies patients based on their risk of readmission, predictive modeling could identify at-risk youth to facilitate targeted intervention strategies. This study developed and validated machine learning models to predict psychiatric readmission in children and adolescents.

Methods:

This retrospective cohort study included 11,225 patients (20,339 psychiatric admissions) ≤ 18 years old with at least one psychiatric admission and an anxiety or depressive disorder diagnosis at any time during the study period from two tertiary-care academic medical centers. Machine learning models were developed to predict psychiatric readmission versus no readmission within 30, 90, and 180 days of discharge using electronic health record data from one institution. These models were internally validated at the development site and externally validated at the second institution.

Results:

Model development and internal validation using random forest models predicted 30-, 90-, and 180-day readmission with area under the receiver operating characteristic curves (AUROC) equal to 0.739, 0.742, and 0.746, respectively. The most important features identified across follow-up periods included previous psychiatric admissions, length of stay, age, and antipsychotic prescriptions. Models trained using a reduced set of 18 features obtained similar performance, with AUROCs ranging from 0.741 to 0.745. External validation of the models retrained using 12 features available at both institutions yielded AUROCs of 0.598, 0.634, and 0.657 for 30-, 90-, and 180-day readmission, respectively.

Conclusion:

The findings of this study suggest that machine learning models can identify children and adolescents at risk of psychiatric readmission.

Keywords

machine learning readmission hospitalization anxiety depression pediatrics

Introduction

Psychiatric admissions in children and adolescents increased by 26% from 2009 to 2019 and have occurred in parallel with escalating health care utilization (e.g., inpatient length of stay, interfacility transfers) (Arakelyan et al., 2023; Lebrun-Harris et al., 2022; Tkacz and Brady, 2021). Psychiatric disorders are expensive to treat, and hospitalizations are costly, with pediatric psychiatric hospitalizations estimated to cost $1.37 billion in 2019 and depressive disorders being the most costly to treat (Arakelyan et al., 2023; Bardach et al., 2014; Soni, 2001). Furthermore, 12%–28% of youth requiring psychiatric hospitalization are readmitted within 6 months of discharge (Arnold et al., 2003; Joyce et al., 2019; Phillips et al., 2020; van Alphen et al., 2017), and most readmissions occur within 90 days (Blader, 2004; Fontanella, 2008; James et al., 2010; Miller et al., 2020). Unplanned, recurrent admissions can be disruptive and deleterious because they impair emotional and social functioning, hinder academic success, adversely affect quality of life into adulthood, and impose substantial burdens on families (Best et al., 2004; Weller et al., 2015). Identifying children and adolescents at risk of readmission could improve the allocation of institutional resources and ensure that high-risk patients receive adequate treatment and interventions (McBain et al., 2022; Trask et al., 2016; Zima et al., 2016).

Previous all-cause pediatric readmission models have commonly excluded psychiatric admissions (Davis et al., 2022; Goodman et al., 2022). Furthermore, only one rule-based (i.e., risk factors determined by literature review) psychiatric readmission tool has been developed in a cohort of patients aged 2–12 years old (Ramsbottom and Farmer, 2018). Therefore, the objective of this study was to develop and validate machine learning models to predict psychiatric readmission within 30, 90, and 180 days of discharge in children and adolescents diagnosed with anxiety or depression using real-world data from the electronic health record (EHR).

Methods

Patient population

This is a retrospective cohort study using EHR data from two tertiary-care academic medical centers. Patients ≤ 18 years of age who had at least one psychiatric admission at Cincinnati Children’s Hospital Medical Center (CCHMC) between September 1, 2013, and December 31, 2022, or Vanderbilt University Medical Center (VUMC) between January 1, 2017, and December 31, 2019, were eligible. Inclusion criteria included having a diagnosed anxiety or depressive disorder during the study period as identified using the International Classification of Diseases (ICD)–9-CM (309.21, 313.0) and ICD-10-CM codes (F32, F33, F41, F43.1, F43.21, F43.22, F43.23, F40). No other exclusionary criteria based on disease or diagnosis history were made. This study was approved by the local institutional review boards at CCHMC and VUMC. The reporting of these data follows the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) Reporting Guidelines (Collins et al., 2015).

Definition of outcomes

The three outcomes evaluated in this study were any psychiatric readmission to the same hospital within 30, 90, and 180 days of discharge from an index psychiatric hospitalization. A specific primary diagnosis or reason for psychiatric hospitalization (e.g., anxiety or depression) was not required to be considered as a readmission. Furthermore, admissions and readmissions to other regional psychiatric hospitals were not considered. Each admission was classified as a case if there was a subsequent psychiatric hospitalization within the respective follow-up period or a control if there was no psychiatric hospitalization. To mitigate the risk of redundant admissions due to interdepartmental transfers (e.g., patient initially boarded in hospital medicine and transferred to inpatient psychiatry), admissions were merged if a discharge and subsequent admission for a patient occurred within three hours of each other.

Data collection

Patient data were obtained from the EHR at both institutions and used for model development. These included demographics, diagnoses, prescription orders, laboratory results, pharmacogenetic testing results (e.g., cytochrome P450 2C19 and CYP2D6 phenotypes from routine clinical testing) (Ramsey et al., 2019), hospital encounters, health insurance, and data captured in flowsheets, which are tabular documentation forms (e.g., Columbia-Suicide Severity Rating Scale, Brief Rating of Aggression by Children and Adolescents) (Barzman et al., 2011). When multiple measurements were obtained during an admission, the average was calculated.

Missing data and imputation

Features that had >25% missingness were removed (see the Feature Selection section). For features related to substance abuse panel results (e.g., amphetamine, marijuana, cotinine), a new value corresponding to a “missing” result was imputed. Patients with missing CYP2C19 or CYP2D6 phenotypes were imputed to be normal metabolizers because clinicians treat patients with unknown pharmacogenetic testing in accordance with the standard of care. For each observation, one feature was created to flag observations where ≥50% of that admission’s laboratory values (e.g., complete blood count and comprehensive metabolic panel results) were missing. This was implemented to capture potential differences in care between observations that could influence readmission. Individual laboratory variables that were missing >25% were still excluded prior to the creation of this feature during the feature selection process. The remaining features were imputed using a bootstrap aggregation, or bagged trees, approach based on all available features. Heights and weights were imputed separately using the same approach but used only sex, age, and either height or weight when available, as these are known biological determinants for child and adolescent anthropometrics. Imputation via bagged trees was implemented using the recipes package in R. Nominal variables were one-hot encoded, ordered variables were ordinal encoded, and continuous variables were centered at a mean of zero and scaled to a standard deviation of one.

Feature selection

Prior to preprocessing, the raw feature set included over 200 variables. To improve the generalizability and interpretability of the final feature set and reduce model complexity, feature selection steps were performed using only the training set. First, two board-certified child and adolescent psychiatrists (S.E.V. and J.R.S.) reviewed the raw feature set to identify any irrelevant or missing covariates that could be removed or added, respectively. Second, features that had >25% missingness were removed from the dataset (N = 50). This step ensured features that would require a large proportion of imputation did not enter the model development phase and risk biasing predictions. Third, variance thresholding removed features that were sparse or exhibited low variance. Finally, recursive feature elimination (RFE) was implemented to identify a subset of features that can predict readmission. Three-fold cross-validation and a random forest algorithm with 1000 trees were used to optimize the area under the receiver operating characteristic curve (AUROC). Through this approach, models were sequentially trained using progressively smaller subsets of features. The final feature set was selected by identifying the smallest set of features where the mean AUROC was within a 1% difference of a model including all features. The 1% difference threshold was chosen a priori to balance predictive performance with model parsimony. Visual inspection of plots comparing the performance of the top n selected features provided supplementary guidance in the selection of the final feature set. RFE was implemented separately for each outcome (i.e., n = 3 total for predicting 30-, 90-, and 180-day readmission) to ensure that the final features used for training each model were representative of that follow-up period. Because the top features were similar for each outcome, additional models were developed using the features that completely overlapped between the top 20, 40, and 60 features for each endpoint. The top 60 features identified by the RFE procedure for each outcome are available in Supplementary Table S1. Feature reduction occurred only in the training set and did not incorporate data from the testing set.

Model development

A time-based cross-validation procedure of data from CCHMC was implemented to create independent training and testing sets. Data from 2013 to 2019 were used for training, and data from 2022 were used for testing the models. The readmission status of patients in the testing set whose follow-up period extended into 2023 was included to avoid right-censoring due to insufficient follow-up time. Data from 2020 and 2021 were not utilized for model training or testing because of changes in the frequency of admissions and the potential for variation in clinical care during or due to the COVID-19 pandemic (Amill-Rosario et al., 2022; Gutiérrez-Sacristán et al., 2022; Leeb et al., 2020; Sorter et al., 2024). Admissions <1 day or >21 days in length were excluded because they potentially correspond to observations that do not represent regular care (e.g., discharge against medical advice, prolonged stay due to issues related to residential placement or state custody). The following algorithms were compared for each outcome: logistic regression, L1-regularized (lasso) regression, L2-regularized (ridge) regression, elastic net regression, random forest, naïve bayes, extreme gradient boosting (XGBoost), support vector machines, and multilayer perceptron. Hyperparameter tuning was performed using 10-fold cross-validation via grid search during model training, with AUROC as the optimization metric. A description of the hyperparameter tuning procedures is provided in Supplementary Table S2. Variable importance scores were calculated for the random forest models to aid in the interpretability of the model predictions and correspond to the scaled Gini impurity index. All models were developed using the recipes and caret packages in R.

Sensitivity analysis

Three sensitivity analyses were conducted to assess the stability of the model predictions. First, because a temporal split was used for creating training and testing sets, there were patients who were included in both sets due to when they were admitted, which could cause data leakage. Therefore, in the final model, patients who appeared in duplicate were removed during model training, and performance was reevaluated on the testing set. Second, to ensure that the imputation of missing data did not influence predictions, the performance of the models was evaluated using only observations with complete data in both the training and testing sets. Finally, while admissions during the COVID-19 pandemic (2020–2021) were not initially included during model development, the testing set was expanded to include psychiatric admissions from 2020 to 2022 to assess how robust the models were to temporal variations in admissions during this period.

External model validation

The models developed at CCHMC were externally validated using data from VUMC to assess the generalizability of the predictions. However, certain data were not collected outside of CCHMC (e.g., Brief Rating of Aggression by Children and Adolescents, pharmacogenetic testing). During internal model development, we identified 18 features that ranked in the top 20 across all follow-up periods. Of these, 12 were available at both sites; therefore, the models at CCHMC were retrained using these 12 features, and performance was evaluated at VUMC. Preprocessing steps applied to the data from CCHMC were similarly applied to the data from VUMC. The models trained and tested at CCHMC were developed independently from the validation data at VUMC.

Statistical analysis

The AUROC was used to evaluate model performance on the test set. The optimal probability threshold for classification was determined using the Youden index, which maximizes the sum of sensitivity and specificity. The AUROCs of models were compared using DeLong’s test. For the final model, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated at probability thresholds ranging from 0.05 to 0.5 on the testing set. Additionally, the number needed to detect (NND) one case correctly and the number needed to screen (NNS) to prevent one misclassification were calculated to provide additional measures of model efficacy (Linn and Grunau, 2006; Rembold, 1998). The NND was determined by taking the inverse of the sensitivity + specificity – 1, or the inverse of the Youden index, and the NNS was determined by taking the inverse of the PPV + NPV – 1. The expected calibration error, slope, and intercept were used to assess model calibration (Schwarz and Heider, 2019). Univariate logistic regression was implemented to evaluate associations between model features and readmission outcome and calculate odds ratios. p-values were considered statistically significant at α = 0.05. All analyses were performed using R (version 4.4.0).

Results

Patients

From CCHMC, there were 18,493 admissions (mean [standard deviation] age, 14.7 [2.3] years; 12,697 [69%] female) from 9,705 patients included in the study, with 2,707 (15%), 4,799 (26%), and 6,379 (34%) admissions with a readmission within 30, 90, and 180 days, respectively. At the patient level, 2,743 of 9,705 (28%) patients had at least one readmission within 180 days of discharge. A total of 16,195 (87%) admissions were used for training, and 2,298 (13%) admissions were used for testing. Anxiety and depression were the most common diagnoses and were frequently comorbid at each admission (Supplementary Table S3). From VUMC, there were 1,846 admissions (mean [SD] age, 14.1 [2.8] years; 1,139 [62%] female) from 1,520 patients, with 134 (7%), 211 (11%), and 283 (15%) admissions with a readmission within 30, 90, and 180 days, respectively. A summary of admission characteristics at both sites is provided in Table 1.

Table 1.

Admission Characteristics

Characteristic	CCHMC		VUMC
Characteristic	Training (n = 16,195)	Testing (n = 2,298)	External validation (n = 1,846)
Age (years), median (IQR)	15.0 (2.9)	15.0 (2.8)	14.6 (3.9)
Sex, n (%)
Female	10,995 (68%)	1,702 (74%)	1,139 (62%)
Race, n (%)
Asian	121 (1%)	27 (1%)	33 (2%)
Black	3,405 (21%)	423 (18%)	258 (14%)
Other	914 (6%)	126 (5%)	70 (4%)
White	11,490 (71%)	1,640 (71%)	1,406 (76%)
Unknown	265 (2%)	82 (4%)	79 (4%)
Ethnicity, n (%)
Hispanic	531 (3%)	95 (4%)	131 (7%)
Non-Hispanic	15,574 (96%)	2,177 (95%)	1,324 (72%)
Unknown	90 (1%)	26 (1%)	391 (21%)
Readmission, n (%)
30 days	2,384 (15%)	323 (14%)	134 (7%)
90 days	4,223 (26%)	576 (25%)	211 (11%)
180 days	5,610 (35%)	769 (33%)	283 (15%)
Alpha-2-agonist count before admission, mean ± sd	0.3 ± 0.6	0.4 ± 0.7	0.4 ± 0.6
Antipsychotic count before admission, mean ± sd	1.2 ± 1.6	1.3 ± 1.7	0.8 ± 1.3
Antipsychotic count during admission, mean ± sd	0.8 ± 0.8	0.9 ± 0.9	0.7 ± 0.8
Depressive disorder, n (%)	12,589 (78%)	2,009 (87%)	1,644 (89%)
Length of stay, mean ± sd	6.4 ± 3.1	7.5 ± 3.3	8.7 ± 3.8
Previous inpatient psychiatry admissions (last year), mean ± sd	1.0 ± 1.8	1.0 ± 1.6	0.2 ± 0.6
Previous inpatient psychiatry admissions (total), mean ± sd	2.0 ± 3.5	2.1 ± 4.3	0.3 ± 0.8
Previous outpatient psychiatry count (last year), mean ± sd	2.1 ± 5.7	2.6 ± 6.7	0.3 ± 2.1
Psychotropic medication count during admission, mean ± sd	2.3 ± 1.6	2.6 ± 1.7	2.5 ± 1.6
SSRI count before admission, mean ± sd	0.6 ± 0.8	0.8 ± 0.8	0.6 ± 0.8
Total mental health diagnoses, mean ± sd	2.7 ± 1.6	2.9 ± 1.6	3.5 ± 1.8

CCHMC, Cincinnati Children’s Hospital Medical Center; sd, standard deviation; VUMC, Vanderbilt University Medical Center.

Model performance

A total of 22, 31, and 33 features were selected for the 30-, 90-, and 180-day readmission models, respectively. The AUROCs of the models in predicting 30-, 90-, and 180-day readmission ranged from 0.694 to 0.743, 0.709 to 0.742, and 0.704 to 0.750, respectively (Table 2). Predicted probabilities were highly correlated across individual models and model types for each follow-up period (Supplementary Fig. S1). The random forest and XGBoost models yielded the highest AUROCs and had similar performance across follow-up periods. Because these were all not statistically different (all p > 0.05), we focused on random forest due to its balance between performance, interpretability, and generalizability. The AUROC of the random forest models on the testing set for predicting 30-day readmission was 0.739 (95% CI: 0.710–0.768), 90-day readmission was 0.742 (95% CI: 0.719–0.765), and 180-day readmission was 0.746 (95% CI: 0.724–0.767) (Fig. 1A–C). The predicted probabilities of readmission by the observed class are shown in Figure 1D–F. The expected calibration errors for the 30-day (0.02, slope = 1.3, intercept = 0.3), 90-day (0.05, slope = 1.3, intercept = 0.05), and 180-day (0.04, slope = 1.2, intercept = −0.1) were low, indicating good calibration (Fig. 1G–I). Sensitivity analyses demonstrated that model performance was unaffected by duplicate patients in training and testing sets, data imputation, and the COVID-19 pandemic (Supplementary Table S4).

FIG. 1.

Random forest model performance for predicting 30-, 90-, and 180-day readmission. Area under the receiver operating characteristic curve (AUROC) for predicting (A) 30-day readmission, (B) 90-day readmission, and (C) 180-day readmission. The gray dotted line reflects a model that does no better than chance. Predicted probabilities by class for the (D) 30-day readmission, (E) 90-day readmission, and (F) 180-day readmission models. The central line reflects the median, and the top and bottom lines reflect the interquartile range. Calibration plots comparing predicted probabilities to the average true probabilities in each bin for the (G) 30-day readmission, (H) 90-day readmission, and (I) 180-day readmission models. The dots reflect the mean predicted probability, and the lines represent the confidence intervals for that bin. The gray dotted line reflects perfect calibration.

Table 2.

Machine Learning Model Performances for Predicting 30-, 90-, and 180-Day Readmission

Model	30-Day		90-Day		180-Day
Model	Training	Testing	Training	Testing	Training	Testing
Logistic regression	0.712 (0.701, 0.724)	0.724 (0.694, 0.754)	0.725 (0.716, 0.734)	0.732 (0.708, 0.755)	0.733 (0.725, 0.741)	0.738 (0.716, 0.759)
Lasso regression	0.714 (0.703, 0.726)	0.729 (0.700, 0.759)	0.726 (0.717, 0.735)	0.733 (0.710, 0.757)	0.734 (0.726, 0.742)	0.738 (0.717, 0.760)
Ridge regression	0.714 (0.703, 0.725)	0.727 (0.697, 0.757)	0.726 (0.717, 0.734)	0.733 (0.710, 0.756)	0.733 (0.725, 0.741)	0.737 (0.716, 0.759)
Elastic net regression	0.714 (0.703, 0.725)	0.728 (0.699, 0.758)	0.726 (0.717, 0.735)	0.733 (0.710, 0.757)	0.734 (0.726, 0.742)	0.738 (0.717, 0.760)
Random forest	0.729 (0.718, 0.740)	0.739 (0.710, 0.768)	0.736 (0.727, 0.745)	0.742 (0.719, 0.765)	0.746 (0.738, 0.754)	0.746 (0.724, 0.767)
Naïve Bayes	0.698 (0.687, 0.710)	0.713 (0.683, 0.743)	0.708 (0.699, 0.717)	0.709 (0.685, 0.733)	0.715 (0.706, 0.723)	0.704 (0.681, 0.726)
XGBoost	0.731 (0.720, 0.742)	0.743 (0.715, 0.772)	0.743 (0.734, 0.751)	0.742 (0.719, 0.765)	0.750 (0.742, 0.758)	0.750 (0.729, 0.771)
Support vector machine	0.692 (0.680, 0.703)	0.694 (0.663, 0.725)	0.714 (0.705, 0.723)	0.720 (0.696, 0.744)	0.721 (0.712, 0.729)	0.734 (0.713, 0.756)
MLP	0.707 (0.696, 0.719)	0.718 (0.689, 0.747)	0.718 (0.709, 0.727)	0.724 (0.701, 0.747)	0.726 (0.718, 0.734)	0.730 (0.708, 0.752)

Values reflect the AUROC (95%) confidence interval.

MLP, multilayer perceptron; XGBoost, extreme gradient boosting.

Figure 2 shows the sensitivity, specificity, NND, and NNS across probability thresholds for each model. For predicting 30-day readmission, setting the threshold at a value that minimized the NND (or maximized the Youden index) achieved a sensitivity of 0.718 (95% CI: 0.667–0.765), specificity of 0.646 (95% CI: 0.624–0.666), PPV of 0.249 (95% CI: 0.222–0.278), NPV of 0.933 (95% CI: 0.919–0.945), NND of 2.75, and NNS of 5.49. For predicting 90-day readmission, setting the threshold at a value that minimized the NND achieved a sensitivity of 0.703 (95% CI: 0.665–0.739), specificity of 0.661 (95% CI: 0.639–0.683), PPV of 0.410 (95% CI: 0.380–0.441), NPV of 0.869 (95% CI: 0.850–0.887), NND of 2.74, and NNS of 3.58. For predicting 180-day readmission, setting the threshold at a value that minimized the NND achieved a sensitivity of 0.717 (95% CI: 0.684–0.747), specificity of 0.668 (95% CI: 0.644–0.692), PPV of 0.521 (95% CI: 0.491–0.551), NPV of 0.824 (95% CI: 0.802–0.844), NND of 2.60, and NNS of 2.90.

FIG. 2.

Random forest probability thresholds. Sensitivity and specificity (left y-axis) and the number needed to detect (NND) and the number needed to screen (NNS) (right y-axis) across probability thresholds for the random forest (A) 30-day readmission, (B) 90-day readmission, and (C) 180-day readmission models. The vertical, black dotted line on each plot represents the probability threshold that minimizes the NND, or maximizes the Youden index, for that model.

The selected features and their coefficients for each model, along with results from univariate logistic regression analyses, are reported in Supplementary Tables S5, S6, and S7. Across all three follow-up periods, many features demonstrated increased likelihood of readmission, including having previous psychiatric admissions, longer length of stay, and being prescribed antipsychotics before or during admission. For example, each day of hospital stay increased the odds of being readmitted within 30 days following discharge by 11%. Comparatively, features like age and having a depressive disorder diagnosis were protective. While these findings help to elucidate how these features contribute to model predictions, they do not support causal relationships, especially as the random forest models can capture interactive and nonlinear associations, which are likely contributing to the model’s superior performance over logistic regression (Table 2).

In comparing the top 20, 40, and 60 features identified during feature selection for each outcome, 18, 31, and 45 features overlapped between all three periods, respectively (Supplementary Table S1). For predicting 30-, 90-, and 180-day readmission using the 18 features that overlapped in the top 20, the AUROCs were 0.744 (95% CI: 0.716–0.773), 0.741 (95% CI: 0.717–0.764), and 0.745 (95% CI: 0.723–0.766), respectively. All three were not statistically different (all p > 0.05) or superior to their respective models developed using the top 22, 31, and 33 features. Using the 31 features that overlapped in the top 40 yielded AUROCs ranging from 0.738 to 0.744 (95% CI: 0.709–0.755), while using the 45 features that overlapped in the top 60 yielded AUROCs ranging from 0.734 to 0.742 (95% CI: 0.706–0.763). The coefficients of the top 18 features for each model developed during internal validation are shown in Figure 3A, with previous inpatient psychiatry admissions within the last year, total previous inpatient psychiatry admissions, and length of stay as top features across models.

FIG. 3.

Scaled feature coefficients of the random forest models for (A) internal validation and (B) external validation. BRACHA, Brief Rating of Aggression by Children and Adolescents; OAS, Overt Aggression Scale; SSRI, selective serotonin reuptake inhibitor.

External model validation

Considering the 18 features overlapping across follow-up periods during internal validation, 12 were available at both sites. The models at CCHMC were retrained using these 12 features and then externally validated using data from VUMC (Fig. 3B), which predicted readmission within 30, 90, and 180 days with AUROCs of 0.598 (95% CI: 0.551–0.645), 0.634 (95% CI: 0.596–0.671), and 0.657 (95% CI: 0.624–0.690), respectively. Comparatively, the performance of these models on the CCHMC testing set predicted 30-, 90-, and 180-day readmission with AUROCs of 0.735 (95% CI: 0.706–0.765), 0.737 (95% CI: 0.714–0.761), and 0.743 (95% CI: 0.722–0.765), respectively.

Discussion

In this prognostic study, we developed and validated machine learning models to predict psychiatric readmission within 30, 90, and 180 days from discharge using EHR data from a large pediatric population. Internal model validation yielded good performance for this application across models (AUROCs: 0.739–0.746). The NND also ranged from 2.60 to 2.75, meaning that under three admissions need to be assessed by these models to correctly identify one readmission case. However, external validation performances were modest, with AUROCs equal to 0.598–0.657. The top predictive features were also similar across follow-up periods and aligned with previously identified risk factors of psychiatric readmission in youth (Edgcomb et al., 2020; Madden et al., 2020). These models have the potential to provide individualized readmission risk to identify those in need of tailored intervention strategies at discharge.

This is the first study to predict pediatric psychiatric readmission using machine learning, and our models provided comparable performance to previous adult and all-age psychiatric readmission models (AUROC: 0.63–0.81) (Boag et al., 2021; Cearns et al., 2019; Morel et al., 2020; Palacios-Ariza et al., 2023; Rumshisky et al., 2016; Vigod et al., 2015; Zhu et al., 2022). However, two of these previous models demonstrated superior performance, which could be because they incorporated features derived from clinical notes using natural language processing techniques. For example, Rumshisky et al. (2016) used a latent Dirichlet allocation model to extract topics from inpatient discharge summaries and observed better discrimination of psychiatric readmission risk when training models with topics versus baseline clinical features alone (AUROC: 0.784 vs. 0.618). Similarly, Zhu et al. (2022) included features corresponding to the most informative words from chief complaint narrative notes and predicted 30-, 90-, and 180-day psychiatric readmission with AUROCs of 0.814, 0.798, and 0.740, respectively. This decreasing performance over longer follow-up periods also contrasts with our study, as we observed similar performance across outcomes. It would be prudent for future work to investigate whether free-text narratives (e.g., discharge summaries, progress notes) improve readmission predictions in youth. Nevertheless, one strength of our study is using a time-based cross-validation procedure to evaluate our models, which better reflect real-world predictions as compared to holdout cross-validation utilized by previous studies. Furthermore, as evidenced by the sensitivity analyses, our models were robust to potential biases related to data leakage, data imputation, and temporal variations impacting the incidence of admissions during the COVID-19 pandemic. Finally, although we externally validated our models using features available at both sites, this yielded modest performance across follow-up periods, which may be associated with population heterogeneity or differences in care at each site (Fontanella, 2008; Phillips et al., 2020). This conclusion is supported by similar performance between the 12- and 18-feature models on the CCHMC testing set, suggesting these features are not influencing the decrease in performance in the external validation.

We identified a set of features that were consistent predictors of 30-, 90-, and 180-day readmission. Prior psychiatric hospitalizations, longer inpatient length of stay, younger age, and more psychiatric diagnoses and comorbidities were associated with greater readmission risk, which align with previous psychiatric readmission models (Boag et al., 2021; Cearns et al., 2019; Morel et al., 2020; Palacios-Ariza et al., 2023; Vigod et al., 2015; Zhu et al., 2022). Several variables related to increased aggression were associated with greater readmission risk and may capture differences related to behavioral function and symptom severity (Fontanella et al., 2009; Vigod et al., 2015). Given associations between medication regimen and readmission, additional attention to treatment effects, medication adherence, and polypharmacy may be warranted, especially in youth prescribed antipsychotics (Cearns et al., 2019; Fontanella et al., 2009; Grudnikoff et al., 2019; Saldaña et al., 2014; Sanchez et al., 2021; Strawn et al., 2025). It is important to emphasize, however, that the association of antipsychotic medication prescriptions with readmission is likely a proxy for illness severity and complexity and is not a direct cause of readmission. Furthermore, having a depressive disorder diagnosis was associated with reduced odds of readmission across follow-up periods, consistent with previous adult studies (Morel et al., 2020; Vigod et al., 2015), although this was not significantly associated in a meta-analysis for psychiatric readmission in children and adolescents (Edgcomb et al., 2020). Other studies also identified that shorter length of stay is associated with greater readmission risk (Morel et al., 2020; Vigod et al., 2015; Zhu et al., 2022), which contrasts with our study and Edgcomb et al. (2020), who observed longer length of stay is associated with readmission. This could reflect differences in care between adults and children and adolescents, as well as the possibility that longer stays in youths corresponds to increased resource utilization and intervention for severely ill patients.

These findings have important clinical implications and may identify youth who could benefit from targeted postdischarge interventions. Those who are more ill (e.g., have had multiple prior hospitalizations, longer hospitalization, and taking antipsychotic medication) are more likely to be readmitted. Yet, postdischarge planning rarely stratifies patients based on the risk of readmission. This suggests a need to move beyond one-size-fits-all approaches to discharge planning and raises the possibility that tailored strategies could be developed. Operationally, a predictive model could be integrated into EHR systems to calculate risk scores at or near discharge. This would enable identification of higher-risk patients who may require more resources or services versus lower-risk patients who could follow standard discharge protocols. For example, patients at an elevated risk may benefit from enhanced case management, postdischarge care conferences, or step-down interventions like intensive outpatient programs (Rubenson et al., 2024). Additionally, school-based support, such as individualized education programs or 504 plans, may help to mitigate the impact of academic stressors on psychiatric symptom worsening and subsequent hospitalizations in youth at high risk for readmission (Weiss et al., 2015). Family-based interventions, particularly for patients with high familial conflict and dysregulation, could be used for secondary prevention (Walkup et al., 2021). These results also raise the possibility that improved capture of diagnostic features (e.g., heterotypic comorbidity and presence of mixed features), which predict longitudinal course, may enhance risk stratification and inform interventions that reduce rehospitalization (Leppanen et al., 2025; Lohr et al., 2023). By identifying which youth are most vulnerable to readmission, these models may ultimately enable clinicians to personalize follow-up care, improve outcomes, and allocate limited outpatient resources more effectively.

The probability threshold for our models was selected by minimizing the NND, or maximizing the Youden index, a common approach to optimizing the sensitivity and specificity of predictive models. While further work is needed to identify an appropriate threshold that considers the cost-effectiveness for this application, it may be beneficial to prioritize higher sensitivities to correctly identify positive cases in the future. Although this approach would reduce specificity and increase false positives, it may be acceptable for prognostic interventions in this context. Importantly, our models also have high NPVs (0.827–0.933), demonstrating their ability to predict non-readmission cases. This could be useful for improving resource allocation, including identifying patients who are at low risk of readmission to ensure resources are directed toward patients who are at high risk. Additionally, with longer follow-up periods, we observed a wider, stabler range of NND and NNS values across thresholds. This flexibility would allow for selecting a threshold that optimizes clinical benefit (i.e., reducing the threshold to improve sensitivity with minimal impact on NND or NNS).

Our study focused on patients with anxiety or depressive disorders—the most common primary psychiatric diagnoses associated with psychiatric hospitalization in youth (Arakelyan et al., 2023; Bardach et al., 2014; Bitsko et al., 2022). However, these diagnoses were not required to be the primary reason for psychiatric hospitalization or to be documented at the time of an index admission. This decision was made because psychiatric diagnoses and diagnostic coding vary and evolve across encounters and providers, and restricting to admissions with only these diagnoses may exclude clinically relevant patients. In fact, our cohort included individuals with other psychiatric comorbidities that were assessed as prospective features in our models (e.g., trauma/stress disorder, eating disorder). Therefore, the model may be broadly applicable to predicting psychiatric readmission rather than specific to admissions for anxiety or depression, although the cohort it was developed in was enriched for patients with anxiety and depression.

There are several limitations that require discussion. First, we were unable to account for patient data or changes in care, including psychiatric treatment, that occurred outside of the academic medical centers (e.g., community-based psychotherapy). Additionally, selection bias may exist if certain patients are more likely to be lost to follow-up (Gianfrancesco et al., 2018). Second, missing CYP phenotypes were imputed as normal metabolizers and may not reflect their natural distribution, yet these features were not included in the final model. Using random forest as the RFE algorithm may also lead to favoring features that are informative to tree-based algorithms (e.g., random forest and XGBoost) over other algorithms during model development. Furthermore, an ensemble model was not explored as individual models provided similar performance (AUROC: 0.69–0.75), and predictions were highly correlated, suggesting the available feature space may be limiting improvements in performance more than the type of algorithm. Additionally, this study solely utilized structured EHR data and did not incorporate features derived from clinical narratives, which may have limited model discrimination compared to previous studies and models leveraging these data and natural language processing techniques (Rumshisky et al., 2016; Zhu et al., 2022). Future research may consider these other features and ensemble approaches to determine how they affect predictions. Third, machine learning models can learn biases from the data on which they are trained and could perpetuate inequalities in healthcare (Gianfrancesco et al., 2018). Our patient population was predominantly white and non-Hispanic, which may limit the generalizability of our findings to more diverse populations. We also did not evaluate whether there are differences in predictions across sociodemographic subgroups. Future analyses should assess for bias in these predictions while recognizing that sociodemographic variables have been associated with differences in readmission risk in youth, although these findings are inconsistent (Carroll et al., 2022; Edgcomb et al., 2020; Madden et al., 2020). Finally, the prevalence of readmission at CCHMC was higher than previously reported rates in youth and across all follow-up periods compared to VUMC, although this varies among hospitals (Feng et al., 2017; Fontanella, 2008; Leon et al., 2006). One reason for this could be that CCHMC is among the largest child and adolescent inpatient psychiatric hospitals nationally and has greater bed capacity compared to other psychiatric hospitals for children and adolescents. Further validation across multiple institutions will be needed to ensure the generalizability of these findings.

This study is the first to develop and externally validate machine learning models to predict psychiatric readmission within 30, 90, and 180 days following discharge in children and adolescents. A subset of features provided comparable performance across follow-up periods. These models will be foundational in identifying youth at risk of psychiatric rehospitalization, which can improve institutional resource allocation and mental health treatment outcomes for children and adolescents.

Supplemental Material

sj-docx-1-cha-10.1177_10445463261466534 — Supplemental material for Predicting Psychiatric Readmission of Children and Adolescents Using Machine Learning

Supplemental material, sj-docx-1-cha-10.1177_10445463261466534 for Predicting Psychiatric Readmission of Children and Adolescents Using Machine Learning by Ethan A. Poweleit, Jeffrey R. Strawn, Rhonda D. Szczesniak, Wei-Qi Wei, Tracy A. Glauser, Laura B. Ramsey, and Judith W. Dexheimer

Footnotes

Acknowledgments

The authors thank Dr. Samuel Vaughn and Dr. Cheryl Cobb for their expertise regarding inpatient psychiatric care at CCHMC and VUMC, respectively. They also thank Mr. Parth Divekar, Mr. Christopher Guardo, and Dr. Henry Ong for assisting with EHR data extraction. Portions of this article were previously published as part of a PhD dissertation and can be found at: .

Author Disclosure Statement

L.B.R. has received research support from BTG International. J.R.S. has received research support from AbbVie, the National Institutes of Health, PCORI, the Yung Family Foundation, and receives material support from Myriad. He receives royalties from UpToDate and Cambridge and has consulted with Cerevel, Otsuka, and Intracellular Therapeutics. All other authors declared no competing interests for this work.

References

Amill-Rosario

, Lee

, Zhang

, et al. Psychotropic prescriptions during the COVID-19 pandemic among U.S. Children and Adolescents Receiving Mental Health Services. J Child Adolesc Psychopharmacol 2022;32(7):408–414.

Arakelyan

, Freyleue

, Avula

, et al. Pediatric mental health hospitalizations at Acute Care Hospitals in the US, 2009-2019. JAMA 2023;329(12):1000–1011.

Arnold

, Goldston

, Ruggiero

, et al. Rates and predictors of rehospitalization among formerly hospitalized adolescents. Psychiatr Serv 2003;54(7):994–998.

Bardach

, Coker

, Zima

, et al. Common and costly hospitalizations for pediatric mental health disorders. Pediatrics 2014;133(4):602–609.

Barzman

, Brackenbury

, Sonnier

, et al. Brief Rating of Aggression by Children and Adolescents (BRACHA): Development of a tool for assessing risk of inpatients’ aggressive behavior. J Am Acad Psychiatry Law 2011;39(2):170–179.

Best

, Hauser

, Gralinski-Bakker

, et al. Adolescent psychiatric hospitalization and mortality, distress levels, and educational attainment: Follow-up after 11 and 20 years. Arch Pediatr Adolesc Med 2004;158(8):749–752.

Bitsko

, Claussen

, Lichstein J,

et al. Mental Health Surveillance Among Children—United States, 2013-2019. MMWR Suppl 2022;71(2):1–42; doi: 10.15585/mmwr.su7102a1

Blader

. Symptom, family, and service predictors of children’s psychiatric rehospitalization within one year of discharge. J Am Acad Child Adolesc Psychiatry 2004;43(4):440–451.

Boag

, Kovaleva

, McCoy

, et al. Hard for humans, hard for machines: Predicting readmission after psychiatric hospitalization using narrative notes. Transl Psychiatry 2021;11(1):32.

10.

Carroll

, Hall

, Brown

, et al. Association of race/ethnicity and social determinants with rehospitalization for mental health conditions at Acute Care Children’s Hospitals. J Pediatr 2022;240:228–234.e1.

11.

Cearns

, Opel

, Clark

, et al. Predicting rehospitalization within 2 years of initial patient admission for a major depressive episode: A multimodal machine learning approach. Transl Psychiatry 2019;9(1):285.

12.

Collins

, Reitsma

, Altman

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594.

13.

Davis

, Zhang

, Lee

, et al. Effective hospital readmission prediction models using machine-learned features. BMC Health Serv Res 2022;22(1):1415.

14.

Edgcomb

, Sorter

, Lorberg

, et al. Psychiatric readmission of children and adolescents: A systematic review and meta-analysis. Psychiatr Serv 2020;71(3):269–279.

15.

Feng

, Toomey

, Zaslavsky

, et al. Readmission after pediatric mental health admissions. Pediatrics 2017;140(6):e20171571.

16.

Fontanella

. The influence of clinical, treatment, and healthcare system characteristics on psychiatric readmission of adolescents. Am J Orthopsychiatry 2008;78(2):187–198.

17.

Fontanella

, Bridge

, Campo

. Psychotropic medication changes, polypharmacy, and the risk of early readmission in suicidal adolescent inpatients. Ann Pharmacother 2009;43(12):1939–1947.

18.

Gianfrancesco

, Tamang

, Yazdany

, et al. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018;178(11):1544–1547.

19.

Goodman

, Casale

, Rychlik

, et al. Development and validation of an integrated suite of prediction models for all-cause 30-day readmissions of children and adolescents aged 0 to 18 years. JAMA Netw Open 2022;5(11):e2241513.

20.

Grudnikoff

, McNeilly

, Babiss

. Correlates of psychiatric inpatient readmissions of children and adolescents with mental disorders. Psychiatry Res 2019;282:112596.

21.

Gutiérrez-Sacristán

, Serret-Larmande

, Hutch

, Consortium for Clinical Characterization of COVID-19 by EHR (4CE). et al.; Hospitalizations associated with mental health conditions among adolescents in the us and France during the COVID-19 pandemic. JAMA Netw Open 2022;5(12):e2246548.

22.

James

, Charlemagne

, Gilman

, et al. Post-discharge services and psychiatric rehospitalization among children and youth. Adm Policy Ment Health 2010;37(5):433–445.

23.

Joyce

, King

, Nash

, et al. Predicting psychiatric rehospitalization in adolescents. Adm Policy Ment Health 2019;46(6):807–820.

24.

Lebrun-Harris

, Ghandour

, Kogan

, et al. Five-year trends in US Children’s Health and Well-being, 2016-2020. JAMA Pediatr 2022;176(7):e220056.

25.

Leeb

, Bitsko

, Radhakrishnan

, et al. Mental health–related emergency department visits among children aged <18 years during the COVID-19 pandemic—United States, January 1–October 17, 2020. MMWR Morb Mortal Wkly Rep 2020;69(45):1675–1680.

26.

Leon

, Snowden

, Bryant

, et al. The hospital as predictor of children’s and adolescents’ length of stay. J Am Acad Child Adolesc Psychiatry 2006;45(3):322–328.

27.

Leppanen

, Kieseppä

, Eskelinen

, et al. Clinical predictors of readmission to psychiatric inpatient care: A 20-year follow up study of former adolescent inpatients. Psychiatry Res 2025;351:116606.

28.

Linn

, Grunau

. New patient-oriented summary measure of net total gain in certainty for dichotomous diagnostic tests. Epidemiol Perspect Innov 2006;3(1):11.

29.

Lohr

, Jawad

, Wood

, et al. Factors associated with child and youth mental health readmissions from a US National Database. JAACAP 2023; doi: 10.1016/j.jaacop.2023.12.001

30.

Madden

, Vajda

, Llamocca

, et al. Factors associated with psychiatric readmission of children and adolescents in the U.S.: A systematic review of the literature. Gen Hosp Psychiatry 2020;65:33–42.

31.

McBain

, Cantor

, Eberhart

. Estimating psychiatric bed shortages in the US. JAMA Psychiatry 2022;79(4):279–280.

32.

Miller

DAA

, Ronis

, Slaunwhite

, et al. Longitudinal examination of youth readmission to mental health inpatient units. Child Adolesc Ment Health 2020;25(4):238–248.

33.

Morel

, Yu

, Liu-Ferrara

, et al. Predicting hospital readmission in patients with mental or substance use disorders: A machine learning approach. Int J Med Inform 2020;139:104136.

34.

Palacios-Ariza

, Morales-Mendoza

, Murcia

, et al. Prediction of patient admission and readmission in adults from a Colombian cohort with bipolar disorder using artificial intelligence. Front Psychiatry 2023;14:1266548.

35.

Phillips

, Steelesmith

, Campo

, et al. Factors associated with multiple psychiatric readmissions for youth with mood disorders. J Am Acad Child Adolesc Psychiatry 2020;59(5):619–631.

36.

Ramsbottom

, Farmer

. Reducing pediatric psychiatric hospital readmissions and improving quality care through an innovative Readmission Risk Predictor Tool. J Child Adolesc Psychiatr Nurs 2018;31(1):14–22.

37.

Ramsey

, Prows

, Zhang

, et al. Implementation of pharmacogenetics at Cincinnati Children’s Hospital Medical Center: Lessons learned over 14 years of personalizing medicine. Clin Pharmacol Ther 2019;105(1):49–52.

38.

Rembold

. Number needed to screen: development of a statistic for disease screening. BMJ 1998;317(7154):307–312.

39.

Rubenson

, Gurtovenko

, Simmons

, et al. Systematic review: Patient outcomes in transdiagnostic adolescent partial hospitalization programs. J Am Acad Child Adolesc Psychiatry 2024;63(2):136–153.

40.

Rumshisky

, Ghassemi

, Naumann

, et al. Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Transl Psychiatry 2016;6(10):e921.

41.

Saldaña

, Keeshin

, Wehry

, et al. Antipsychotic polypharmacy in children and adolescents at discharge from psychiatric hospitalization. Pharmacotherapy 2014;34(8):836–844.

42.

Sanchez

, Lytle

, Neudecker

, et al. Medication adherence in pediatric patients with bipolar disorder: A systematic review. J Child Adolesc Psychopharmacol 2021;31(2):86–94.

43.

Schwarz

, Heider

. GUESS: Projecting machine learning scores to well-calibrated probability estimates for clinical decision-making. Bioinformatics 2019;35(14):2458–2465.

44.

Soni

. Top Five Most Costly Conditions among Children, Ages 0-17, 2012: Estimates for the U.S. Civilian Noninstitutionalized Population. In: Statistical Brief (Medical Expenditure Panel Survey (US)). Agency for Healthcare Research and Quality (US): Rockville, MD; 2001. Available at: http://www.ncbi.nlm.nih.gov/books/NBK447180/ [November 5, 2021].

45.

Sorter

, Stark

, Glauser

, et al. Addressing the pediatric mental health crisis: Moving from a reactive to a proactive system of care. J Pediatr 2024;265:113479.

46.

Strawn

, Mills

, Neptune

, et al. Electronically monitored antidepressant adherence in adolescents with anxiety disorders: A pilot study. J Child Adolesc Psychopharmacol 2025;35(3):145–154; doi: 10.1089/cap.2024.0102

47.

Tkacz

, Brady

. Increasing rate of diagnosed childhood mental illness in the United States: Incidence, prevalence and costs. Public Health Practice 2021;2:100204.

48.

Trask

, Fawley-King

, Garland

, et al. Do aftercare mental health services reduce risk of psychiatric rehospitalization for children? Psychol Serv 2016;13(2):127–132.

49.

van Alphen

, Stewart

, Esposito

, et al. Predictors of rehospitalization for depressed adolescents admitted to acute psychiatric treatment. J Clin Psychiatry 2017;78(5):592–598.

50.

Vigod

, Kurdyak

, Seitz

, et al. READMIT: A clinical risk index to predict 30-day readmission after discharge from acute psychiatric units. J Psychiatr Res 2015;61:205–213.

51.

Walkup

, Friedland

, Peris

, et al. Dysregulation, catastrophic reactions, and the anxiety disorders. Child Adolesc Psychiatr Clin N Am 2021;30(2):431–444.

52.

Weiss

, Blizzard

, Vaughan

, et al. Supporting the transition from inpatient hospitalization to school. Child Adolesc Psychiatr Clin N Am 2015;24(2):371–383.

53.

Weller

, Faulkner

, Doyle

, et al. Impact of patients’ psychiatric hospitalization on caregivers: A systematic review. Psychiatric Services 2015;66(5):527–535.

54.

Zhu

, Jiang

, Hu

, et al. Individualized prediction of psychiatric readmissions for patients with major depressive disorder: A 10-year retrospective cohort study. Transl Psychiatry 2022;12(1):170.

55.

Zima

, Rodean

, Hall

, et al. Psychiatric disorders and trends in resource use in pediatric hospitals. Pediatrics 2016;138(5):e20160909.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.13 MB