Interventions to Improve the Economic Self-sufficiency and Well-being of Resettled Refugees: A Systematic Review

3.1.1 Types of studies

Studies were required to use a prospective, controlled methodology: a randomised controlled trial (RCT) design; a quasi-randomised controlled trial design (QRCT, where participants are allocated by means which may not be expected to influence outcomes, such as alternating allocation, birth date, the date of the week or month, case number or alphabetical order); or a nonrandomised controlled design (i.e. quasi-experimental design). Nonrandomised controlled studies were required to provide information on baseline comparability of the cohorts and use statistical tools to adjust for baseline differences. For all included studies, participants needed to be prospectively assigned to study groups or a control group (i.e. alternative intervention or ‘services as usual‘), and studies needed to measure control group outcomes concurrently with intervention group outcomes.

Additionally, studies were required to have been conducted or published since 1980 (see Section 1.1).

3.1.2 Types of participants

The review included studies of individuals who meet the domestic legislative definition of a refugee for the country of the intervention and:

a.

were formally assisted to resettle by the government (i.e. were resettled refugees but not asylum-seekers);

If for any reason a study's sample did not fall completely within these parameters, we agreed to contact the author in order to obtain disaggregated data for each of the populations meeting the criteria of a, b, and c. If we had been unable to obtain disaggregated data, we planned to use sensitivity analyses based on studies with mixed populations.

Although abstracts or titles sometimes used the term ‘refugee’ to cover asylum-seekers, we were able to discern through the full article or learn from the author if the population was resettled refugees.

3.1.3 Types of interventions

Interventions were eligible if they were designed to increase the economic self-sufficiency and well-being of resettled refugees. There was no restriction on the type or the duration of the intervention. Such interventions could include employment casework, therapy sessions, or broad community support, for example.

3.1.4 Types of outcome measures

Studies were required to report at least one primary or secondary outcome.

3.2.1 Search terms

The following key search terms were used:

Population: (resettle* OR re-settle* OR refuge* OR force* ADJ *migrant* OR asylum* OR humanitar* ADJ entrant* OR humanitar* ADJ settle*)

These terms were in line with other meta-analyses on refugees and the suggested terminology for limiting studies to those using quantitative methodology (Bollini, Pampallona, Wanner, & Kupelnick, 2009; Bronstein & Montgomery, 2011; Clark & Mytton, 2007; Fazel, Wheeler, & Danesh, 2005; Gagnon & Tuck, 2004; Lipsey & Wilson, 2001; Littell, Corcoran, & Pillai, 2008; Morton, 2011; Robjant, Hassan, & Katona, 2009). From the protocol, we added the terms ‘labor’ and ‘labour’ to cover interventions that may use terminology such as ‘labour market integration‘.

A full listing of how these search terms were used in each database is given in Appendix 8.1.

3.2.2 Databases

We searched the following databases and citation indices:

Applied Social Sciences Index and Abstracts (ASSIA)

These databases were chosen based on existing advice on systematic reviews and reviews in the area (Greenhalgh, 2010; Hammerstrøm, Wade, & Jørgensen, 2010). Databases are constantly developing, however; we therefore updated this list when carrying out the searches to reflect the most up-to-date knowledge. Thus:

Business Source Complete replaced Business Source Elite to ensure a more sensitive search. Business Source Complete contained more active, peer-reviewed, business related journals than any other database currently available.

3.2.3 Searching other resources

In attempt to capture unpublished reports, the above list includes some databases known to encompass grey literature: CPCI-SSH Conference Proceedings Citation Index, OpenGrey, PAIS International, PolicyFile, and Proquest dissertation and thesis.

Additionally, we hand searched the Journal of Refugee Studies for any relevant articles that may have been published since its inception (1988).

We also searched the following websites covering research and policy, and relevant governmental organisations:

Center for Migration and Refugee Studies, American University in Cairo: www.aucegypt.edu/gapp/cmrs

(This website was browsed, but a search was not possible due to technical difficulties which persisted through this research.)

Manpower Demonstration Research Corporation: www.mdrc.org

We also inspected the bibliographies of all studies reviewed in full text, and used personal contacts, listservs/discussion lists, Twitter, and Facebook groups in attempt to locate any relevant ongoing or unpublished studies.

3.2.4 Selection of studies

Records were screened and selected through a two-part process. Through each stage, the number of papers excluded and retrieved was noted in the flow diagram in the next section.

First, both review authors separately reviewed all titles and available abstracts obtained from the search. When there was any doubt about the relevance of the title or differences in opinion between the two review authors, the full text of the paper was obtained and reviewed.

Second, all studies retrieved were screened by the two review authors against the inclusion criteria using the screening guide (in the Appendices, Section 8.2).

If studies met inclusion criteria, we planned to use the data collection sheet for included studies (in the Appendices, Section 8.3) and the risk of bias sheet (Section 8.4), and plans for analysis (below). Evidence of effectiveness was to be examined only for studies meeting the screening criteria.

3.3.1 Data extraction

The review authors planned to independently extract and code studies in the data extraction sheet. The initial data extraction sheet included information on the context, study design, study sample, and outcomes. A separate risk of bias sheet can be found in the Appendices, Section 8.4 to assess and record the quality of included studies. The review authors planned to assess the appropriateness of the data extraction and would have determined if changes were necessary. Relevant primary investigators were to be contacted as necessary for missing or unclear information. Disagreement on extraction and coding would have been resolved through consulting an independent reviewer at the Centre for Evidence-Based Intervention at the University of Oxford or the Campbell Methods Group.

3.3.2 Risk of bias table

As previously stated, internal validity was ensured by including only studies with the following designs: prospective randomised controlled trials, quasi-randomised controlled trials, or nonrandomised controlled trials which adjust for baseline differences.

For included studies, additional risk of bias was planned to be assessed using both i) categories outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011: Section 8.6) and ii) predetermined criteria previously published in two systematic reviews and modified to adapt to the parameters of this study as shown in appendix 8.4 (Morton, 2011; Zief, Lauver, & Maynard, 2006). Review authors planned to independently complete both of the risk of bias sheets. Relevant primary investigators were to be contacted to request any missing information. If there had been any dispute between the two review authors, an arbiter from the Centre for Evidence-Based Intervention at Oxford University or the Campbell Methods Group would have been consulted.

i) Using the data extraction form, review authors planned to independently assess each study for risk of bias on the following criteria as outlined by the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011: Section 8.6):

1.

Sequence generation (was the allocation sequence adequately generated?)

One example of ‘other sources of bias’ could have been including participants who have received employment services from more than one agency. This could lead to bias, as participants in this subgroup may be more likely to demonstrate an effect after having received a higher dosage of employment services.

For each domain, review authors planned to independently assign each included study to one of the following categories:

(A)

High risk of bias

ii) Additionally, we planned to carry out a narrative assessment of bias based on the second risk of bias table. This table was adapted to this review to accommodate both randomised controlled trials and high-quality quasi-experimental designs. In particular, these criteria would have addressed how a study controls for baseline differences, reassignment, and attrition. Studies would have been considered to be of higher quality if they meet What Works Clearinghouse standards for overall and differential attrition (Mathematica Policy Research, Inc., 2011: 13-14), use statistical tools to adjust for all baseline differences that are statistically significant before the intervention, and use original assignment of intervention and other groups as the basis for analyses. These standards have been used in recent systematic reviews (Del Grosso, Kleinman, Esposito, Martin, & Paulsell, 2011; Mathematica Policy Research & Child Trends, 2012). This assessment of the risk of bias was not intended to be a ranked or quantitative exercise. Rather, the exercise was planned to facilitate discussion about the strengths and weaknesses of the research designs and their implementation for included studies.

3.3.3 Measures of treatment effect

3.3.4 Unit of analysis issues

3.3.5 Dealing with missing data

If a study had missing data, we would have contacted the primary author to request additional information. For trials reporting outcomes only for participants completing the trial, the primary author would be asked to provide additional information to permit intention-to-treat analyses. Studies in which participants were analysed as members of the groups to which they were originally assigned (intention-to-treat analysis), studies that included only those participants who were willing or able to provide data (available-case analysis), and studies that analysed participants who adhered to the study's design (per-protocol analysis) would have been analysed separately.

If obtaining missing data was not possible or investigators were unresponsive, we would have made assumptions regarding whether the data were ‘missing at random’ or ‘not missing at random’ and would have followed the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011: Section 16.1.2). We anticipated that data that were not missing at random were likely to be missing for reasons related to the outcomes of the missing data. For example, if a participant agreed to take part in an employment services intervention, but was unhappy with not finding a job or the job found, the participant may have been unwilling to complete any follow-up interviews or questionnaires on his or her experience. In such a situation, where dichotomous data were missing, we planned to impute data with the assumption that the participants experienced the less favourable outcome. We also planned to explore the possibility that those missing experienced the positive outcome (found work) and impute data based on this assumption. We planned to conduct sensitivity analyses to examine the impact on the results of changes in the assumptions made about missing data.

If studies had missing summary data, such as missing standard deviations, we planned to derive these where possible, using calculations provided in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011: Section 16.1.3).

We planned to specify the methods used to address any missing data in the results tables. If imputation was not possible, we planned to outline the reasons for this in the text.

3.3.6 Assessment of heterogeneity

We planned to record any heterogeneity in terms of study demographics, setting, programme characteristics, and study quality both in a narrative format and in a table.

If meta-analysis had been used, statistical heterogeneity would have been reported both using a Q statistic and its p value, the I² statistic, and by visual inspection of forest plots.

3.3.7 Assessment of reporting biases

Reporting bias could have been present both as a result of publication bias and because of selective reporting. Extensive searches were conducted to attempt to identify both published and unpublished studies. We planned to use funnel plots for information about possible publication bias if we found sufficient studies (e.g. at least ten studies, Higgins & Green, 2011: 10.4.1). However, asymmetric funnel plots are not necessarily caused by publication bias (and publication bias does not necessarily cause asymmetry in a funnel plot). If asymmetry had been present, we would have considered possible reasons for this.

We also planned to seek to identify any pre-published study protocols to check that all pre-specified items appear in the final reports. Additionally, review authors would have contacted researchers with regard to missing data and information, and all missing data and concerns about reporting biases would have been reported.

3.3.8 Subgroup analysis

We planned to subgroup results by:

gender of the participants (Beiser & Hou, 2003; Siraj, 2007);

If sufficient data had existed and comparison conditions had been similar, subgroup analyses in RevMan would have examined potential differential effects of interventions, dividing studies using meta-regression where appropriate (Higgins & Green, 2011: Section 9.6) according to:

programme content grouped if there were clear delineations in approach, intensity, and content (e.g. mediation between employers/employees, employment mentorship, and language training)

In the event of sufficient studies and data for subgroup analyses, these analyses would have been accompanied by a discussion of their potential pitfalls. No conclusions from subgroup analyses would have been drawn and interpretation of relationships would have been cautious, as they are based on subdivision of studies and indirect comparisons.

3.3.9 Sensitivity analyses

We planned to perform sensitivity analysis to assess whether the findings of this review were robust with respect to the decisions made in the process of obtaining them. For example, sensitivity analyses may have been performed by excluding studies according to study quality issues (including those with low sample size and high risk of bias) and by separating studies by design (i.e. randomised controlled trials from quasi-experimental designs). For methodological quality, we planned to consider sensitivity analysis for each major component of the risk of bias checklists. Sensitivity analysis would further be used to examine the robustness of conclusions in relation to the quality of data (outcome measures based on different time intervals).

3.3.10 Data synthesis

Meta-analysis would have been used if appropriate. If sufficient studies were identified, we planned to analyse the effects of differing types of interventions, time points, and/or comparison conditions separately. Meta-analyses would have been conducted for each outcome construct separately, combining dichotomous and continuous outcomes as appropriate as discussed in ‘Section 7.5.1 Measures of Treatment Effect‘. The standard for study design was fairly strict to be included in this review, so we did not plan separate studies by study design, but study design would have been explored through sensitivity analyses if there had been sufficient studies.

3.3.11 Statistical procedures and conventions

A random effects model would have been assumed given the expected level of heterogeneity across studies. If there were sufficient studies, analyses would have been performed using RevMan5, SPSS, or STATA.

3.3.12 Narrative synthesis

We planned to use a narrative synthesis of the results to present information on the strength of the study design, risk of biases, population differences, context of the intervention, and context of the results. Depending on the quantity of papers that met the inclusion criteria, we had planned follow the three step suggestion of Petticrew and Roberts (2005): ‘(i) organizing the description of the studies into logical categories; (ii) analyzing the findings within each of the categories; and (iii) synthesizing the findings across all included studies’ (p.170). As planned, we discussed the nature of excluded studies in the appendix since no studies met the inclusion criteria.

3.3.13 Treatment of qualitative research

Qualitative data from any included studies was viewed as useful in contextualizing the results and determining the risk of bias, and would have been reported in the review if it had been available.