Substance Use Treatment Completion Among Unhoused Young Adults: A Predictive Modeling Approach

Introduction

Young adults aged 18–24 comprised approximately 7% (n = 47,436) of the unhoused population identified in the United States of America’s 2023 annual point-in-time count (US Department of Housing and Urban Development, 2023). Homelessness arises from the complex interplay of individual, relational, socioeconomic, and environmental factors. Homelessness in the United States is defined as lacking access to a fixed, regular, and adequate nighttime residence (McKinney-Vento Homeless Assistance Act, 1987). Unhoused people may be living in places not meant for human habitation, in shelters or other temporary living arrangement such as hotels subsidized by governmental programs or transitional housing, or are at immanent risk of losing their housing. Unhoused young adults can experience overlapping forms of marginalization and structural disadvantage that increase vulnerabilities to substance misuse. Compared to adolescents and older adults experiencing homelessness, younger adults report shorter duration of homelessness but have reported higher rates of recent life stressors related to social relationships, mental and physical health, education/job training, and employment (Sample & Ferguson, 2020; Tompsett et al., 2009). These stressors are often coupled with the challenges of age-appropriate milestones, such as identity development, navigating friendships and relationships, and development of autonomy (Munson et al., 2017). Initiation of substance use and risk of substance use disorder (SUD) increases in early adulthood, although young adults display heterogeneity in their substance use trajectories (Chen & Jacobson, 2012; Maggs et al., 2023).

Substance misuse, which has biological, personal, relational, and environmental antecedents, has been cited by some unhoused people as a contributing factor for either entering into or finding it difficult to exit homelessness (Barile et al., 2020; O’Toole et al., 2004; Sample & Ferguson, 2020). Substance misuse can contribute to homelessness by disrupting engagement in occupational tasks such as work or school/training programs that can impact persons’ financial resources and create risks for housing instability (Stablein et al., 2021). In addition, substance misuse can disrupt relationships with peers, family, or community members and decrease social, emotional, and instrumental supports that might protect against housing instability (Stablein et al., 2021).

Young adults in addiction treatment have been found to have more severe risk profiles than older adults that could contribute to poorer addiction treatment outcomes in the absence of adequate psychosocial and clinical supports (Andersson et al., 2021). Although unhoused youth aged 13–26 who used injected drugs were found to have higher odds of utilizing mental health services, prior research with unhoused youth found an inverse association between substance use and help-seeking behaviors (Crosby et al., 2018; Solorio et al., 2006). Addressing substance misuse among unhoused young adults might facilitate their engagement in supportive services. In addition, supporting substance use recovery may promote retention in housing services, as both substance use and drug-related activity can contribute to eviction from housing services (Cole, 2024). Early adulthood is a crucial time to change substance use patterns to promote more positive trajectories and mitigate risks of violence, accidental injury, criminal justice system involvement, poor physical and mental health outcomes, and premature mortality. Understanding the predictors of treatment completion among unhoused young adults may aid in the development of strategies to facilitate successful engagement into substance use treatment services.

Insight into behavioral health, service-related, and demographic features predictive of substance use treatment completion in unhoused young adults may inform strategies to promote positive health and social outcomes for this vulnerable group. Improvement in substance use behaviors and treatment completion have been shown to be positively associated, although clinical improvement and treatment completion are theorized to be two distinct constructs (Sahker et al., 2022). Prior research indicates that treatment participation and completion is associated with more positive quality of life outcomes (Gonzales et al., 2009). Outcomes for those who do not complete treatment are heterogeneous and treatment dropout is not always associated with poor social or clinical outcomes (Szafranski et al., 2019). Nevertheless, treatment services, particularly those delivered by caring, recovery-oriented service providers and peers, have been cited by people who use drugs as playing an important role in their own recoveries (Dell et al., 2022a; Schoenberger et al., 2022).

Publicly available administrative data in the United States, such as the Treatment Episode Data Set-Discharges (TEDS-D), has often been utilized to understand factors associated with treatment completion or discontinuation for diverse subpopulations receiving addiction treatment. Such studies have identified disparities in treatment completion by people of minoritized racial/ethnic backgrounds, psychiatric comorbidities, insurance status and other socioeconomic factors (Friesen & Kurdyak, 2020; Goldman et al., 2020; Krawczyk et al., 2017; Suntai, 2021). Increasingly, researchers have sought to develop and evaluate predictive models using publicly available, routinely conducted administrative data to predict treatment completion or non-completion among different subpopulations of people receiving substance use treatment services (Acion et al., 2017; Baird et al., 2022; Kong et al., 2022; Nasir et al., 2021; Shikalgar et al., 2024; Stafford et al., 2022). Less research has focused on predictive models and their relevance for people experiencing homelessness. Using machine learning (ML) approaches to predict treatment outcomes from routinely collected baseline data may provide clinicians and program managers with insights into both population-level and individual-level needs to improve service outcomes.

The use of ML tools in psychiatry and other behavioral health disciplines has been characterized as a “paradigm shift” from classical hypothesis testing to evaluating the predictive performance of models among unseen or untested data to inform clinical treatment (Chekroud et al., 2021). An advantage of ML approaches is their ability to handle high dimensional data and non-linear relationships. Although ML approaches are becoming more popular in addiction medicine, additional research is needed to explore the generalizability of ML models to diverse populations and treatment settings (Mak et al., 2019). The development and evaluation of ML models to understand patterns in treatment completion among large real-world datasets may inform theory-driven research and specific clinical practice to improve treatment completion for persons at risk of non-completion.

Materials and Methods

Results

Aim 1: Model Performance

Overall, each model adequately predicted treatment completion on the test data, although differences in performance were found depending on the analytic approach used and whether the model was trained on a balanced training set. Table 1 summarizes the performance of the RF and LR models trained on balanced and imbalanced data. Considering the four models evaluated, RF trained on imbalanced data had the highest accuracy, specificity, precision, and AUC, whereas the RF trained on imbalanced data had the highest F1 an balanced accuracy. The LR model trained on balanced data had the highest recall. Predictive models trained on balanced data were more useful than those trained on imbalanced data for predicting a higher proportion of treatment completers, as indicated by higher recall. Models trained on balanced data also achieved higher balanced accuracy and F1 scores relative to models trained on imbalanced data. However, accuracy, specificity, and precision were higher for RF and LR models trained on imbalanced data.

OPEN IN VIEWER

Table 1.

Performance Metrics for Models Predicting Treatment Completion for Unhoused Young Adults in Substance Use Treatment Services Using Random Forest Imputed Dataset (N = 12,273).

Approach	Random forest		Logistic regression
Model	1	2	3	4
Training data	Imbalanced	Balanced	Imbalanced	Balanced
Metric
Accuracy	0.7406	0.7065	0.7043	0.6769
Specificity	0.9002	0.7159	0.8886	0.6656
Precision	0.6370	0.5216	0.5461	0.4868
Recall	0.4819	0.6856	0.2968	0.7019
F1	0.4819	0.5925	0.3846	0.5749
Balanced accuracy	0.6439	0.6856	0.5927	0.6838
AUC	0.7753	0.7665	0.7269	0.7234
95% CI	0.7561, 0.7958	0.7469, 0.7860	0.7051, 0.7491	0.7021, 0.7462
Tuning parameters	ntree: 300 mtry: 5	ntree: 500 mtry: 5	α: 1.0 λ: .0021	α: 0.1 λ: .0002

AUC = Area under the receiver operating characteristic curve; F1 = harmonic mean of precision and recall; 95%CI = 95% confidence interval; ntree = number of trees in each random forest; mtry = number of features tried at each split; α = elastic net mixing parameter; λ = regularization parameter; bolded metrics indicate highest value relative to other models.

Aim 2: Feature Importance

Variable Importance Plots

Figure 1 shows variable importance scores for both RF and LR trained on balanced data. The top two most important features for each model were ranked in the same order: receiving ambulatory treatment services and having private insurance. The next most important features for the RF model included first use of substances as a minor, methamphetamine use, and having any prior treatment. The next most important features for accurate classification in the LR model included other stimulant use, being referred from a criminal justice setting, and receiving residential services. Several top features were ranked similarly across models, such as receiving ambulatory services, having private insurance, and methamphetamine use, criminal justice referral, and prior treatment. However, some features were inconsistently ranked across models; for example, first use as a minor was ranked as the third most important feature in the RF model but was ranked second to last in the LR model.

OPEN IN VIEWER

Figure 1.

Variable Importance Plots for Models Trained on Balanced Data.

Partial Dependence Profiles

To understand the marginal effect of each of the top predictors on treatment completion, partial dependence profiles are presented in Figure 2. As indicated in Figure 1, the top two features were the same for the optimal RF and LR model. The PDPs visualized in Figure 2 show that each of these top three features had a similar predicted marginal effect on treatment completion. For the RF model, receiving ambulatory services was associated with 0.32 probability of treatment completion, relative to 0.59 probability of completing treatment if not receiving ambulatory services. For the LR model, receiving ambulatory services was associated with 0.29 probability of completing treatment, relative to 0.65 probability of completing treatment if not receiving ambulatory services. Similarly, probability of treatment completion was comparable across RF (0.62) and LR (0.68) models.

OPEN IN VIEWER

Figure 2.

Partial Dependence Profiles: Top 5 Features for Each Model.

Discussion

Our study contributes to the growing use of supervised ML approaches to advance behavioral health science that promotes positive mental health and substance use outcomes among young adults (Dell et al., 2022b; Han & Seo, 2022; Kundu et al., 2022; Rakovski et al., 2023). We found that models trained on routinely collected administrative data can yield moderately accurate predictions of treatment completion in unhoused young adults. We envision that the models developed in this study could be further refined and potentially integrated into substance use treatment services that already collect TEDS-D data elements, particularly those programs serving unhoused young adults. Models may be useful for facilitating individualized treatment planning between clinicians and clients or for administrators in developing programming to meet the needs of unhoused young adults. However, further evaluation is needed before findings can be implemented and scaled to individual settings, as it is unknown whether the models, trained on national data, might yield meaningful predictions for individual sites with smaller numbers of clients. In the present study, findings reveal multiple features important in the accurate classification of treatment completion, which may be useful for the development of individualized interventions to support clients’ engagement and retention in treatment. These two main findings are discussed in more depth below.

Recent research has found that RF has outperformed LR in classifying different clinical conditions (Dell et al., 2022b, 2022c). However, not one method will outperform under all conditions, as in the present study the penalized LR using elastic net and RF yielded similar results when both were trained on balanced data. In addition, the evaluation of each approach requires interpretation of multiple metrics, as no single metric can fully capture the performance of a model. Although models trained on balanced data had slightly lower overall accuracy, AUC, precision and specificity, higher recall may be prioritized over specificity when the goal is to identify a rare outcome (Byrne et al., 2019). Overall, our models showed similar AUC as other modeling approaches utilizing similar features from TEDS data on distinct populations in SUD treatment. Although a few other studies (discussed below) have used a wider array of modeling techniques, we evaluated our models with a comprehensive array of performance metrics that facilitate assessment of the tradeoffs between models (Acion et al., 2017; Baird et al., 2022;Stafford et al., 2022).

An additional consideration when selecting between models for use in any setting is how explainable each model is for different stakeholders who directly use or are impacted by the predictions generated by the model. Inspecting feature importances can provide insight into how each model operates and can influence stakeholders’ levels of trust in the model’s predictions and ultimately how or whether the model is used to guide practice. As was evident in the present study, several features were ranked of similar importance across the RF and LR models considered and these consistently important features may be considered as reflecting “genuine aspects of the data considered” (Saarela & Jauhiainen, 2021, p. 9). Similar levels of accuracy were found between LR and RF models despite having some inconsistencies in feature importance scores across models. Selection of an optimal model in practice settings requires consideration of both the technical performance of the algorithm to classify the outcome of interest and the degree to which stakeholders trust the model to advance the service setting’s goals, cohere with the setting’s people, culture, and processes, and the degree to which the use of a predictive algorithm can be supported by the agency’s technology and infrastructure.

As detailed below, our findings complement and build upon other research using predictive modeling approaches to classify treatment completion or non-completion for different populations receiving SUD treatment. Acion et al. (2017) sought to predict treatment completion among Hispanic adults in outpatient treatment who had no prior treatment episodes using the 2006-2011 TEDS-D. The range of AUC (0.793–0.820) found by Acion et al. (2017) was slightly higher than what we found (0.7234–0.7753) and might be attributable to differences in substance use and demographic patterns between each sample and that our study included different treatment settings rather than focusing solely on outpatient settings. We did not rely solely on AUC and included metrics such as precision, recall, and F1 score to assess how well each predictive model was at correctly identifying relevant cases rather than looking at the model’s overall ability to discriminate between classes. In addition, Stafford et al. (2022) generated several predictive models to identify reasons for OUD treatment discontinuation using more recent 2015-2019 TEDS-D data. The highest performing RF model generated by Safford et al. showed comparable accuracy (0.69) and AUC (0.73) relative to our models. In Stafford et al. (2022), the feature most important for accurate classification of treatment discontinuation among opioid use disorder (OUD) treatment episodes (service setting) was ranked highly in both the RF and penalized LR models considered in this study. Finally, prior analyses using decision trees to identify inequities in SUD treatment completion have found similar patterns, namely the absence of a mental health condition, having income, and being non-Hispanic white were associated with higher probabilities of treatment completion (Baird et al., 2022).

The lower probability of treatment completion in ambulatory settings found in our study is consistent with other studies using TEDS data to study other populations, such as those who use opioids and pregnant women receiving treatment for cannabis use (Kitsantas et al., 2023; Stahler & Mennis, 2018). Several implications could follow from this finding. On the one hand, we may question whether ambulatory settings are the most appropriate settings for unhoused young adults to engage in substance misuse treatment, as lacking secure and stable housing can create immense barriers to accessing treatment regularly. However, a limitation of the current study is that what constitutes treatment completion likely varies across detoxification, ambulatory, and residential services. Furthermore, TEDS-D does not contain data on the complete array of services offered during each ambulatory episode and whether harm reduction or abstinence-only approaches are being implemented. Person-centered care emphasizes client choice and requires flexibility in the types of services offered and how they are delivered. As most services are offered in ambulatory settings, strategies to retain unhoused young adults into are needed, and could include an increase in the availability of housing first services, for example, and intensive outreach approaches tailored to those clients who use methamphetamine and experience treatment barriers due to lack of commercial insurance.

Having no private insurance and receiving outpatient services was predictive of lower probability of treatment completion. This could be interpreted in light of the difficulties the unhoused population can face when engaging in ambulatory treatment services. Prior literature has found that unhoused clients’ previous negative experiences of care, including provider stigma or difficulty navigating complex administrative processes or keeping appointments, self-stigma, or meeting food or shelter needs taking priority over attending clinic services create substantial barriers to accessing and engaging in treatment services (Miler et al., 2021). Research on residential and outpatient settings have found mixed results for the relationship between prior treatment episodes and treatment completion (Baker et al., 2020; Cacciola et al., 2009; Mutter et al., 2015). Some evidence suggests that persons with multiple treatment episodes may have greater substance use severity and psychiatric comorbidities (Rash et al., 2008; Villagrana & Lee, 2020).

Use of methamphetamine (for each model), use of heroin (in the RF model), and use of other stimulants (in the LR model) were important substance use-related features for accurately predicting treatment completion. Co-occurring methamphetamine and opioid, particularly fentanyl, use is associated with greater risk of overdose and death (Han et al., 2021; Mattson et al., 2021). Although the unhoused population has higher risk of opioid and methamphetamine use disorder, they are less likely to engage and be retained in care, although low-threshold programs are associated with higher retention in SUD treatment services (Gaeta Gazzola et al., 2022; Wakeman et al., 2022). In addition, as psychiatric disorders are common among people experiencing homelessness relative to the general population, low-threshold programs that integrate mental health and substance use treatment may be advantageous to the health of unhoused people, especially as comorbid psychiatric conditions are risk factors for treatment non-adherence and non-completion (Krawczyk et al., 2017).

Although these features are associated with lower probability of completing treatment, several evidence-based and evidence-informed interventions can be delivered in outpatient settings that specifically address methamphetamine use, opioid use, and psychiatric comorbidity among young adults. Housing First models, which provide immediate access to housing and rehabilitative services, are considered crucial to address and treat SUD and co-occurring and mental health issues among unhoused young adults (Slesnick et al., 2023). Motivational interviewing (MI) is an evidence-based, person-centered practice that strengthens motivation for behavior change (Miller & Rollnick, 2023). MI strategies with unhoused young adults may promote both positive substance use behaviors as well as sexual health behaviors (Tucker et al., 2020, 2021). Next, contingency management interventions, which are informed by operant conditioning to reward and reinforce healthy substance use behaviors, have shown effectiveness for unhoused men who have sex with men with methamphetamine use disorders and for persons receiving medication for opioid use disorders (MOUD) (Bolivar et al., 2021; Brown & DeFulio, 2020; Reback et al., 2010). For psychiatric comorbidity, Assertive Community Treatment leverages the skills of a multidisciplinary team to assertively engage and support persons with comorbid serious mental illness and substance use disorders who have high rates of hospitalization, criminal justice system-involvement, or housing instability (Morse et al., 2006; Penzenstadler et al., 2019). In addition, although employment was not a highly important feature for classifying treatment completion, qualitative evidence from unhoused young adults suggests the importance of employment and educational opportunities as a potential pathway from homelessness (Sample & Ferguson, 2020). To this end, integrating supported employment and supported education into housing and behavioral health interventions may be supportive of recovery (Ferguson et al., 2012; Hofstra et al., 2023).

Supplemental Material