PROTOCOL: The effectiveness of skills training to increase employment among those experiencing and at risk of homelessness: A systematic review

Types of studies

We will include randomized control trials (RCTs) and quasi-experimental studies in our review. This inclusive approach acknowledges the limited number of studies employing random assignment in the pre-search phase for this topic. We will consider both individual random and cluster RCTs that meet the specified inclusion criteria. All included studies should involve the experimental group (participants receiving skills training courses or programs) and the control group (participants with placebo intervention, usual care, or a wait-list group). The quasi-experimental studies should contain the pre-test and post-test, and report the method used to control the selection bias (such as propensity score matching).

We will exclude studies that lack a control group, such as single-group before-after studies. Additionally, studies meeting the inclusion criteria but failing to provide sufficient information for the computation of means and standard deviations for continuous outcomes or event numbers for dichotomous outcomes will be reported for their characteristics but will be excluded from the meta-analysis due to incomplete data. All studies should be published in English. We will take studies or reports published in peer-reviewed journals, websites, and gray literature into consideration.

Types of participants

Individuals experiencing and at risk of homelessness will be eligible participants. Homelessness is defined differently across nations and throughout time. We will cite the definition used by Keenan (2021) and O'Leary (2022) in the Campbell Collaboration protocol: Homelessness refers to individuals who do not have access to safe, stable, and adequate housing. This includes those who are living in unsanitary or overcrowded environments, those who are sleeping in public places or on the streets, those who are staying in temporary accommodations such as shelters and hostels, and those who are in insecure living facing eviction or unsafe living conditions. It additionally involves individuals who are residing in inappropriate accommodations, such as tents or vehicles, or who are “sofa surfing” without a fixed address.

Additionally, participants also have to undergo a skills training program that offers comprehensive vocational services to people who are suffering or at risk of homelessness.

Types of interventions

The interventions will provide short-term or long-term systematic skills training courses or offer job training opportunities or certifications for individuals experiencing or at risk of homelessness. These courses or training programs aim to improve the following skills of homeless people, including work-related hard skills (technical skills and work experience), work-related soft skills (non-technical skills, such as communication, teamwork, emotional intelligence, etc.), and life skills (fundamental skills, self-sufficient living skills, and social skills). These courses and training programs could be provided by governments, charitable organizations, faith communities and the non-profit sector.

Types of outcome measures

We will measure multi-dimensional outcomes to evaluate the effect of the skills training. We considered employment status as the primary outcome. We regarded work and life skills improvement and housing stability as the secondary outcomes. All included studies must report the primary outcome. We will extract information on secondary outcomes from the included studies, but the absence of secondary outcome reporting will not be a reason for study exclusion.

Duration of follow-up

We will divide the follow-up time into short-term follow-up (≤1 month), medium-term follow-up (1–6 months), and long-term follow-up (>6 months).

Types of settings

This systematic review will cover all kinds of training courses and training programs provided by governments, charitable organizations, faith communities, and the non-profit sectors. Homelessness service settings can be transitional shelters, emergency shelters, respite sites, alternative space programs, or part-time shelters (Toronto Public Health, 2019). In our systematic review, We will mainly include the following intervention settings:

Shelter programs: many shelters offer skills training programs to their residents, which can include job skills training, life skills training, and mental health services. These programs can help homeless individuals build the skills they need to become self-sufficient and find employment.

Electronic searches

We will search the following electronic databases with no date, language, or location restrictions:

Applied Social Sciences Index and Abstracts (ASSIA)

We will also search the following clinic trial platforms:

The Cochrane Central Register of Controlled Trials (CENTRAL)

Searching other resources

We will search the following dissertations, conference proceedings, journals, websites, and gray literature:

Conference Proceedings Citation Index.

Description of methods used in primary research

The primary study should be designed with at least one intervention group and at least one control group. It will use the following study design: RCT or quasi-experimental design.

Selection of studies

First, we will remove the duplicate publication by Endnote 20 and Rayyan. Second, two reviewers (Z. J. and M. N.) will independently select eligible studies by screening the title and abstract through Endnote 20 and Rayyan software. Third, two authors will rescreen by searching the full text of these studies to determine the final included studies. Two reviewers will resolve the discrepancy by consensus.

Data extraction and management

Two reviews (Z. J. and Y. Q.) will independently extract the following information from each study: the name of the first author, publication year, country, study design (RCT or quasi-experimental study), inclusion and exclusion criteria, population type (experiencing homelessness or at risk of homelessness), the characteristics of enrolled subjects (number, age, gender), type of intervention group (work-related hard skills training, work-related soft skills training or life skills training), intervention goals, type of control group (placebo intervention, usual care or at wait-list), intervention settings, intervention duration, intervention intensity, intervention adaptation, intervention integrity, and the effect size for intervention. We also need to extract the baseline and final values or any changes in the focus outcomes.

Assessment of risk of bias in included studies

For the studies that met the inclusion of this review, RCTs will be assessed by the Cochrane Risk of Bias tool 2.0 (ROB 2.0) (Sterne, 2019). Quasi-experimental studies will be evaluated by the Cochrane Risk of Bias In Non-randomized Studies–of Interventions tool (ROBINS-I) (Sterne, 2016). Two reviewers (M. Y. and K. G.) will independently evaluate the quality. The discrepancies will be resolved by consensus. The ROB 2.0 tools have the following aspects: (1) bias arising from the randomization process; (2) bias due to deviations from intended interventions; (3) bias due to missing outcome data; (4) bias in the measurement of the outcome; (5) bias in the selection of the reported result. The studies will reach an overall judgment about the risk of bias as “low risk of bias,” “some concerns” and “high risk of bias.” The ROBINS-I tools have 7 domains including (1) bias due to confounding; (2) bias in the selection of participants into the study; (3) bias in the classification of interventions; (4) bias due to deviations from intended interventions; (5) bias due to missing data; (6) bias in the measurement of outcomes; (7) bias in the selection of the reported result. We can judge the overall risk of bias for every study as “low risk of bias,” “moderate risk of bias,” “serious risk of bias,” “critical risk of bias” and “no information.”

We will present the assessment results of RCTs and quasi-experimental studies separately. Additionally, we plan to perform sensitivity analyses distinguishing the role of low and high risk of bias, we will conduct separate sensitivity analyses for different study designs and distinct measures of treatment effect.

We will extract the standardized effect sizes from the study to perform the meta-analysis. For the studies without standardized effect sizes, we plan to use the Campbell Collaboration's effect size calculator. If there is insufficient information to calculate the effect size, we will contact the author to access the related data. If it is impossible to acquire detailed information, we will describe the characteristics but exclude the study in the synthesis of the results. For dichotomous outcomes, such as employment status, and housing status, we will use odds ratios (ORs) with 95% confidence intervals (CIs). For continuous outcomes, such as skills score, we will use the mean differences (MDs) with 95% CI if they are reported in the same measurement scale. With the continuous outcomes reported in the different scales, we plan to calculate Hedges' g to report standardized mean differences (SMDs) in order to avoid overestimation (Hedges, 1981).

Unit of analysis issues

We recognize that one or more interventions may be evaluated in a single study. In this case, the pair-wise comparison should be assessed separately. For trials with three-arms or multi-arms, we plan to use the three-level model or the Robust Variance Estimation method to incorporate several effect sizes within studies (Noortgate, 2014; Tipton, 2015). If there is more than one intervention duration, we will regard each duration as an independent group. For the cluster-randomized trials, we will assess the unit of analysis errors. If there is a unit of errors, we need to use an intra-cluster correlation coefficient collected from related trials to inflate the standard deviation (Li, 2022).

Criteria for determination of independent findings

There is a risk of overlapping results between studies. Therefore, to avoid counting data repeatedly, we will compare the studies that met the inclusion criteria for the authors, affiliation institution, inclusion criteria, and data details. We will only include the unique data from each study.

Dealing with missing data

We will try to contact the author to obtain all the data when we encounter a problem with missing data. If it is unable to access the necessary data of the included study, especially effect size data, we will report the characteristics of the study but state that it could not be included in the meta-analysis because of missing data.

We plan to use scatterplots and boxplots to identify outliers. If we detect an outlier, we need to analyze whether it is caused by a measurement error or a data entry error, then correct the error if possible. We plan to carry out a sensitivity analysis and report both the results with the outlier and without the outlier.

Assessment of heterogeneity

There are two ways to assess the heterogeneity of the included studies. We will use the I ² statistic (range: 0%–100%) and the likelihood ratio test to assess heterogeneity. The I ² in the range of 0% to 40% is regarded as not important; 30% to 60% represents moderate heterogeneity; 50% to 90% represents substantial heterogeneity; 75% to 100% is regarded as considerable heterogeneity (Higgins, 2023). The likelihood-based method has a consideration of between-studies variance. This method is preferred to the standard method in undertaking random effects meta-analysis when the value of between-trial components of variance has an important effect on the overall estimated treatment effect (Hardy, 1996).

Assessment of reporting biases

Funnel plots and Egger's test can be performed to search for the signs of asymmetry to examine the publication bias if the number of included studies is 10 or more. We will not evaluate publication bias when there are fewer than 10 studies with the same endpoint. Less than 10 studies will not properly distinguish chance from real asymmetry.

Data synthesis

All analyses will be conducted using the statistical programming language R, package metafor. We will pool data from the same endpoint. For dichotomous outcomes, we will use the DerSimonian-Laird inverse variance method and the random effect model to combine studies by ORs with corresponding 95% CIs. For continuous outcomes, we will combine studies by MD or SMD with a corresponding 95% CIs. If it couldn't be performed in a quantitative synthesis, study findings will be synthesized narratively (Campbell, 2020).

Subgroup analysis and investigation of heterogeneity

There are considerable heterogeneities between the interventions, the measurement of outcomes, and the measurement time points related to skills training. We will perform subgroup analysis by meta-regression in the following scenarios.

Demographics (age, sex, ethnicity)

Sensitivity analysis

We will perform sensitivity analyses to assess the robustness of our findings. These analyses will include the isolation of studies based on their study design, distinguishing RCTs and quasi-experimental designs in the pooling process. We will also conduct sensitivity analyses to differentiate the influence of low and high risk of bias. In cases where we identify an outlier, we will make separate data pools, one including the outlier and one excluding it (O'Leary, 2022).

Treatment of qualitative research

We do not plan to include qualitative research.

Summary of findings and assessment of the certainty of the evidence

We do not plan to include a summary of findings and an assessment of the certainty of the evidence.