Work Programmes for Welfare Recipients

Rationale for this review

We are not aware of any systematic reviews of the effects of work programmes for welfare recipients worldwide. Several overviews of controlled trials have been performed (mostly from the USA; Fischer 1995; Michalopoulos, Schwartz, & Adams-Ciardullo 2000; Cebulla et al. 2005; Grogger, et. al. 2002; Bloom, Hill, & Riccio 2003). In Europe, a recent collaborative overview (Cornwell et al. 2002) was performed in six European countries (Denmark, France, Germany, the Netherlands, Norway, and the UK). The results were not directly comparable because the countries have different ways of organising their social services, and they had somewhat different methodological approaches.

The group of individuals on welfare differs among countries. In the USA, welfare is mostly provided to poor families with children. In 1996, the Temporary Assistance for Needy Families (TANF) programme replaced the Aid to Families with Dependent Children (AFDC) as the nation's main welfare programme. Most TANF recipients are single mothers, but there are also some families with two unemployed parents. The Food Stamp Program is also an important transfer programme for the poor in the U. S. In addition there are EITC (Earned Income Tax Credit) and Medicaid. Some countries do not have a sharp distinction between unemployment benefits and welfare benefits, while other countries (for instance Norway and Denmark) distinguish between individuals who are entitled to unemployment benefit if they have been employed in the past, and individuals who are only entitled to welfare. In these countries, all citizens have the right to receive welfare benefit if they have no other source of income.

Included designs

We chose to limit the review to randomised controlled trials because this is the most rigorous design for studies aimed at establishing causal inference.

Criteria for inclusion and exclusion of studies for this review: Included designs: o

Randomised controlled trials (RCTs)

Excluded designs

Studies using the following designs were collected, listed, and described: o

Non-randomised concurrently controlled studies (two groups defined by the investigators and dependent variables measured on both groups before and after the intervention)

There were no restrictions regarding country of publication. There were no language restrictions.

Participants

Welfare recipients. This includes both persons who already receive welfare and people who applied for welfare. People who receive food stamps were also included. We excluded persons entitled to unemployment benefit and persons with pensions of any kind.

Interventions

Interventions intended to help welfare recipients become self-sufficient typically come as “packages” involving several components. Therefore, it is usually not possible to measure the effect of one component apart from the effect of the other components of the programme. We included interventions that offered one or more of the following: time limited work experience, job search assistance, remedial education, job clubs, financial incentives, time limits on financial support, or vocational training. The interventions could be either mandatory or voluntary. In a voluntary programme random assignment takes place after individuals volunteer, and in a mandatory programme volunteering does not take place at all; individuals who meet certain criteria are simply randomly assigned.

The term “time limited work experience” needs some clarification. The work experience is not a permanent job. The duration of the work must be explicitly stated. This duration was recorded for each study.

Control/comparison conditions

The main control or comparison condition was ordinary (passive) social economic assistance or the usual services available to welfare recipients.

Outcome measures

All outcomes involved a comparison between different groups. The comparisons were post intervention measures. Accepted sources of outcome data were state and county administrative records, recordings by programme personnel or employers and self-reports. The type of data source was recorded for each study.

Primary Outcomes: (1)

Work status

Secondary outcomes: (1)

Skills and satisfaction

Follow-up durations

Because duration of follow-up was expected to vary across studies, the exact duration of follow-up was recorded for each study. After data collection was finished, duration times were analysed based on the available data. Follow-up times were grouped into “one year”, “two years”, “five years”, “1-3 years” etc.

Search strategy for identification of relevant studies

Strategy for electronic database searches

We searched the following databases:

C2-SPECTR (11 December 2003),

The African Trials Registry (http://www.mrc.ac.za/ATR/) was searched on 18 January, 2006 for the years 1955-2002.

Search words in title and keywords were ‘work’, ‘employment’, ‘unemployment’, ‘welfare’, and ‘social assistance’.

We used the same text words across all databases (unless specifically stated) and used the specialised controlled vocabularies for each database. For details of the search strategy, see Appendix A.

Other sources of information

In addition, references from included primary reports were scanned. We also scanned the reference lists from reviews that we found through our search strategy. The following authors of included studies and other potential experts in the field were contacted: Espen Dahl, Ivar Lødemel, Mathematica Policy Research and Abt Associates were contacted by email. Specifically, Gayle Hamilton at MDRC provided comments on a draft version in March 2005. The following centres of the Cochrane Collaboration were contacted by e-mail on 12 January 2006 with a request for trials: Australasian, Brazilian, Chinese, Dutch, German, Italian, Russian (under Nordic branch), South African, and the one in Bahrain. We searched the book “The Digest of Social Experiments” (Greenberg & Shroder 2004). This book documents 240 completed and 21 ongoing social experiments updated until April 2003. Another source of study data is a database of US welfare-to-work programs constructed and maintained by Andreas Cebulla, Stacey Bouchet and David Greenberg (Cebulla et. al. 2005). This database is open to the public and is said to include all known random assignment evaluations of U. S. welfare-to-work programs through 2003 that were targeted specifically at AFDC/TANF recipients.

The following websites were searched:

Abt Associates (USA) (Searched 29 November, 2005)

( http://www.abtassociates.com )

Criteria for evaluating eligibility of retrieved studies

Two reviewers independently scanned the abstracts and titles of identified reports for eligibility, according to the inclusion criteria above. Full copies of all those deemed eligible by one of the reviewers were retrieved for closer examination. Consensus was reached by discussion and consultation with a third reviewer, if necessary. All studies which initially appeared to meet the inclusion criteria but, based on the full text reports, did not meet the inclusion criteria were detailed in a table of excluded studies.

Quality assessments

We assessed components that may contribute to the measured effectiveness of the included interventions. Two reviewers independently assigned each selected study to quality categories described below. Uncertainty or disagreement was solved by discussion with a third reviewer.

Generation of allocation sequence

MET = Resulting sequences are unpredictable (explicitly stated use of either computer-generated random numbers, table of random numbers, drawing lots or envelopes, coin tossing, shuffling cards, or throwing dice).

UNCLEAR = Vague statement that the study was randomised but not describing the generation of the allocation sequence or statement(s) indicating that random allocation was used in some but not all cases.

NOT MET = Explicit description of inadequate generation of sequence, (e.g., using case record numbers, alternation, date of admission, date of birth).

Concealment of allocation sequence

MET = Participants and investigators cannot foresee assignment, e.g. central randomisation performed at a site remote from trial location; or use of sequentially numbered, sealed, opaque envelopes).

UNCLEAR = Vague statement that the study was randomised but not describing the concealment of the allocation sequence.

NOT MET = Explicit statement that allocation was not concealed OR statement indicating that participants or investigators can foresee upcoming assignment (e. g., open allocation schedule, unsealed or non-opaque envelopes).

Control of initial difference in prognostic factors between groups

In a properly randomised study, all initial differences between groups will be caused by chance. But these initial differences may be important and can affect outcomes, especially if the study is not large (as N increases, the probability of a meaningful difference between randomised groups decreases). This applies to all prognostic variables, both known and unknown. But in improperly randomised designs and RCTs with differential attrition, there may be important differences between groups. These differences can be systematic, and they can appear in unmeasured variables as well as in the measured ones. It is generally possible to control for the latter but not the former. Matching can be used before the intervention to make groups more similar, and regression methods can be used after the intervention to control for initial differences, but all these methods may introduce bias in the results (Deeks et al. 2003).

Studies, in which both generation and concealment of allocation sequence are MET, were coded as MET below.

MET = Control for one or more prognostic factors. Also score MET when there is no control for prognostic factors because there was no imbalance (statistically significant difference between groups) in measured variables.

UNCLEAR = Sufficient information could not be obtained.

NOT MET = Imbalance in prognostic factors and failure to control for this imbalance.

Prevention of Performance Bias

MET = other interventions avoided or used similarly across comparison groups.

UNCLEAR = Use of other interventions not reported and could not be verified by contacting the investigators.

NOT MET = Dissimilar use of other interventions across comparison groups, i. e. differences in the care provided to the participants in the comparison groups other than the intervention under investigation.

Prevention of Detection Bias

MET = Assessor unaware of the assigned treatment when collecting outcome measures. This was also scored as met if the outcome was questionnaire data or data from administrative records.

UNCLEAR = Blinding of assessor not reported and could not be verified by contacting the investigators.

NOT MET = Assessor aware of the assigned treatment when collecting outcome measures.

Prevention of Attrition Bias

MET = Losses to follow up less than or equal to 20% and equally distributed between comparison groups (proportion of total loss to follow-up equal to or less than 20% in group with the highest loss to follow-up).

UNCLEAR = Losses to follow up not reported.

NOT MET = Losses to follow up greater than 20% or not equally distributed between comparison groups.

Intention-to-treat analysis

MET = Intention to treat analysis performed or possible with data provided.

UNCLEAR = Intention to treat not reported, and could not be undertaken by contacting the investigators.

NOT MET = Intention to treat analyses not done and not possible for reviewers to calculate independently.

This review has been graded using GRADE (GRADE Working Group, 2004). GRADE stands for Grading of Recommendations Assessment, Development and Evaluation. The evidence is graded separately for each outcome. Evidence from effect studies is downgraded if the design is not a randomised controlled trial. Limitations might be lack of allocation concealment, lack of blinding, or a large attrition. Consistency refers to the similarity of estimates of effect across studies. Finally, directness refers to the extent to which the people, interventions and outcome measures are similar to those of interest. The overall quality of the evidence is categorised as high, moderate, low, or very low. The reasons for downgrading the evidence are always made explicit in GRADE.

Data management and extraction

Two reviewers independently extracted information from the full text report on study characteristics using a specially designed, pretested data extraction form (Appendix B). For cases in which outcome information was missing from the original reports, attempts were made to retrieve the necessary data for the analysis from the original investigators or from the Internet. For example, missing unemployment rates were collected by going to the website of the U. S. Department of Labor ( http://data.bls.gov/PDQ/outside.jsp?survey=la ). Inter-rater agreement (i. e. coding reliability) was assessed and reported as percent agreement. Disagreements were resolved by meeting and discussing coded items. Data were entered into Comprehensive Meta-Analysis version 2 (Borenstein et al. 2004)

Details of study coding categories

Study characteristics: Country (or countries) of origin, year of publication, publication type, trial quality (see above), employment rate in the area.

Design: Randomised controlled trial, cluster-randomised controlled trial, or quasi-randomised controlled trial.

Participants: age, ethnicity, gender, education level, number of children, age of youngest child, previous work experience, welfare history.

Intervention: employment-focused or education-focused, job search first or varied first activity, mandatory or voluntary, duration of intervention, extent of work (e.g. working hours per week), funding agent and implementing agent.

Data synthesis

Continuous measures (e.g., earnings) were calculated as standardised mean differences (Hedges’ g). We reported the 95% confidence intervals for all effect estimates. We also reported the post-intervention earnings and welfare payments in US dollars converted to year 2005 equivalents using the following website:

http://oregonstate.edu/Dept/pol_sci/fac/sahr/sahr.htm#_Conversion_factor_tables_are_availa

For studies with other currencies, we first converted the amounts into US dollars and then converted to their 2005 equivalents.

Discrete outcomes

We expressed binary outcome measures (e.g., employed/unemployed) as risk ratios (relative risks). The odds ratio is not the correct outcome measure in prospective studies2. We also reported the number needed to treat (NNT, with 95% confidence intervals) for summary findings.

When computing summary statistics, the primary studies were weighted by the inverse of their variance.

Identifying heterogeneity

We used the advice described in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2005) for identifying statistical heterogeneity. If confidence intervals for the results of individual studies (generally depicted graphically using horizontal lines) have poor overlap, this generally indicates the presence of statistical heterogeneity. More specifically we first performed a Chi-square test for heterogeneity. If this approached statistical significance (p < 0.10), we looked at the results of an I-Squared test. If this was greater than 25%, we concluded that there was heterogeneity in the results.

Addressing heterogeneity

When heterogeneity was judged to be large, we examined potential sources using meta-regression, and stratified analyses using categorical moderators.

We performed meta-regressions with the following intervention and contextual moderators:

duration of intervention

The effects of participant characteristics are best measured by looking at separate subgroup results for each primary study (e.g. combining results only for Whites), but this was not feasible because subgroup results were only reported in some of the reports. Instead we entered aggregate data from the primary studies as predictors in meta-regressions:

mean age

One moderator was entered at a time, so we could not measure the effect of the moderators controlling for the others (this was not a multiple regression). We report zero-order correlations among the continuous moderators.

We performed separate analyses across the following intervention and setting related factors:

labour force attachment (employment-focused) or human capital development (education-focused) strategy

We performed separate analyses across different levels of the following design quality factors:

proper generation of the randomisation sequence

Publication bias

To explore possible publication bias, we report a funnel plot for each main outcome showing both observed studies and studies imputed to produce symmetric plots (how it would have looked with no evidence of publication bias).

Sensitivity Analyses

Sensitivity analyses were used to evaluate whether the pooled effect sizes were robust across different ways of calculating standard errors based on significance levels. Reporting of nonsignificant results was not only analysed using p = 0.1, but also using p = 0.55 and p = 0.99.

Sensitivity analyses were also used for exploring the effects of continuous moderators. For the main analyses, we excluded studies with missing data in the meta-regressions. In another scenario, we imputed the variable means for studies with missing values. We compared the patterns of significant slopes over the two ways of dealing with missing values.

Fixed vs. Random Effects Models

We used fixed effects models when heterogeneity was small (as defined above). We used random effects models when heterogeneity was large, and could not be explained by intervention/setting factors or meta-regression.

Criteria for determination of independent findings

In many instances, several different outcome data are measured on the same subjects in the primary studies (e.g. employment status and earnings). Sometimes the same outcome is measured at multiple points in time. Because these data are from the same sample of participants, and, therefore, are not independent estimates of treatment effect, we analysed the data in such a way that any one analysis contained a single outcome from a particular time period after random assignment. When two intervention groups shared the same control group, we avoided including both intervention groups in the same analysis. Specifically, we used four separate data files for employment, earnings, welfare payments, and proportion on welfare. Within each data file, we grouped the outcomes according to follow-up time. Intervention groups sharing the same control group were not analysed together. The only exceptions were the direct comparisons in Atlanta, Columbus, Grand Rapids, and Riverside, in which we compared two intervention groups sharing a common control group (not a meta-analysis). For the exploratory analyses (meta-regression and the meta-analytic analogue of ANOVA) we used the mean effect magnitude (an option in Comprehensive Meta-Analysis) for studies in which the same outcome was measured at different follow-up times.

Statistical procedures and conventions

Results were analysed using Comprehensive Meta-Analysis software version 2 (Borenstein, et. al., 2004). Prognostic variables were used in analyses if there were less than or equal to 20% missing data. Graphical presentations of effects (forest plots) were produced using Comprehensive Meta-Analysis.

Changes in this review from the published protocol

The main change is that we have extended the inclusion criteria to all programmes that the field perceives as “welfare-to-work” programmes. We previously excluded studies where the programmes did not offer time-limited work experience. But the external reviewers argued that work experience was only a main part of welfare-to-work programmes in the 1960s and 1970s. This component has become less and less important. Wider inclusion criteria are more meaningful and useful for policy purposes.

We have also assessed the quality of the evidence using GRADE (GRADE Working Group, 2004). For each of the four outcomes we graded the evidence for the 2 year follow-up time.

The screening process

Figure 1 is a flowchart of the inclusion process. After having excluded 191 citations from reading the fulltext reports, we were left with 113 citations. Another 45 citations (mainly from Europe) were excluded because they were either nonrandomised controlled studies or they did not include work experience. We included 68 citations when using the limited protocol inclusion criteria. After incorporating the extended inclusion criteria, we included another 34 citations, making the total 102 included citations.

Citations can be reports, journal articles, books, book chapters, etc. Each programme can be reported in several citations and/or several programmes can be reported in one citation. Finally, a programme can be implemented in several sites. Table 1 shows the 46 included programmes described in 102 citations. It was difficult to arrive at the number of programmes because many had several different names. For instance, the programme “Welfare Reform Indiana” is also labelled “Indiana Manpower Placement and Comprehensive Training (IMPACT) Program”. The programme “EWEP (Experimental Work Experience Program” is also called “San Diego Job Search and Work Experience Demonstration”. And “EMPOWER” also goes under the name “Arizona Works”. A further complication is that even if the same programme was conducted in different states, the programme operators were free to include or exclude programme components, and they had varying degrees of monitoring and sanctioning.

Multisite programmes with different intervention conditions: Many of the programmes were conducted in several sites. The highest number of sites was in the Food Stamp Education and Training Program (53 sites). Sometimes separate results for each site were reported for a programme, but at other times results were summed up for all sites. A few sites reported more than one intervention condition (Atlanta, Columbus, Grand Rapids, and Riverside). Atlanta, for instance, had one group randomised to receive a labour force attachment strategy and another group receiving a human capital development strategy.

Intervention conditions: Table 1 describes 46 programmes but it has 58 rows. This is because the GAIN evaluation had 6 conditions and the NEWWS evaluation had 8 conditions.

Different endpoints: For some of the 58 conditions there were more than one follow-up time. Because of this, there were e. g. 73 different employment endpoints.

Excluded studies

Table 2 (list of excluded studies) consists of 24 citations from 16 studies which were not randomised controlled trials (e. g. cohort studies or nonrandomised studies). Most of these studies were from the United States, but some were from Norway, Belgium, Canada, and the United Kingdom.

Ongoing studies

We identified two ongoing studies which possibly conform to our inclusion criteria:

The first one is a Dutch study (Blonk & Brenninkmeijer 2004) which investigated effects of the JOBS programme in Holland.

The other is the ERA Programme (The Employment Retention and Advancement Program; Anderson & Martinson 2003, 2005). It is conducted in several U. S. sites (Los Angeles County, Riverside County (2), Duval County, Leon County, Cook County, St. Clair County, Medford, Eugene, Hennepin County, New York City (2), and Portland). The programme is also being evaluated in the UK (Walker, et. al., 2006). Over the course of about 15 months, Jobcentre Plus randomly assigned over 16,000 people, making the ERA evaluation the largest random assignment test of a social policy's effectiveness in the UK to date.

Studies awaiting assessment

According to Vasiliy V. Vlassov, director of the Russian branch of the Nordic Cochrane Centre (personal communication, 24 January, 2006), there are a number of Russian databases with free internet access where trials may be described. (A list of databases is at http://www.inion.ru/product/db_2.htm .) We have not been able to access them because of the language barrier.

Raw data

We would like to draw attention to the four included Microsoft Excel files labelled “employment.xls”, “earnings.xls”, “welfare payments.xls”, and “welfare proportion.xls”. The files contain all the raw data collected for this review. There are two reasons for including these files, both having to do with transparency. Firstly, interested researchers will be able to transfer the data into their statistical software package of their likings and check our results, or they may do additional analyses, like multiple meta-regressions. Secondly, anyone having first-hand knowledge of the included primary studies might point out possible errors in the data files or add missing values. Appendix B (data extraction form) provides explanations of the variables in the attached Excel files.

Description of included studies

A main finding is that there are many randomised trials from the United States but few such trials from outside North America (two Canadian programmes were included; see Table 1).

Year when data collection started: Figure 2 shows that the first programme of this type started in the late 1960s. The number of studies was low in the 1970s. It increased sharply around 1980, and increased further in the late 1980s. After peaking in the period 1990-1994, it decreased somewhat in the late 1990s. The numbers refer to intervention sites (n= 59) with separately reported results.

Policy context and evaluators: In 1988, US Congress passed the Family Support Act, which offered states $1 billion a year in new federal money for welfare-to-work programmes, but only if they put up matching state resources. Most of the studies reported here were conducted by the MDRC (previously known as the Manpower Demonstration Research Corporation). In their studies, many background variables have been recorded in the same way across different sites and different programmes. Some other studies are evaluated by Mathematica Policy Research or Abt Associates. Finally, a few programmes have been evaluated by universities and federal agencies.

Program goals and components: Goals like preventing social exclusion and increasing quality of life were not emphasised in the included reports. No study had the aim of drug rehabilitation. Almost all programmes had multiple components. This could be, for instance, job search assistance, job clubs, educational classes, case management, time limits on welfare receipt, financial incentives, or child care or transportation services.

Data sources: Most studies had data on earnings and employment from state and county administrative records (register data).

Publication Type: Of the 102 included citations, only 11 were journal articles. The rest were reports, books, and book chapters.

Methodological quality of included studies

Inter-rater reliability

Six raters worked independently and then compared results in pairs. The results reported here were recorded after a pilot test period of the first 12 programmes during which we had meetings and negotiated how to code the studies consistently according to a version of the data extraction form (Appendix B is the latest version). We then coded agreement for 11 programmes4. In total, we coded agreement for 55 variables 11 times, summing up to 605 possible disagreements (Table 3). Note that agreement means different things depending on the type of question. For variables coded “Met”, “Unclear” and “Not Met”, there are six possible ways of disagreeing. For variables such as mean age, there are in principle an infinite number of ways of disagreeing. In such cases, raters had to report exactly the same numerical value in order to get an agreement. The data extraction form (Appendix B) was edited several times during the pilot period, but ultimately all 46 programmes were coded in a consistent way.

For all pairs of raters and all variables we achieved the proportional agreement P_o= .82. Table 3 shows in detail how the disagreements were distributed over the 41 variables with disagreement. For the remaining 14 variables5 there was perfect agreement.

Participants

Our best estimate of the sample size in this systematic review is 412,045 participants, of which 245,509 were randomised to an intervention group and 166,536 were randomised to a control group. Almost 9 out of 10 were women (Table 7). This is because lone parents are usually women. The few males are either unemployed men in two-parent families with small children or heads of single-parent families.

In 1990, the U.S. Bureau of the Census (U.S. Census Bureau 1990) classified race into four major groups: (a) White; (b) Black; (c) American Indian, Eskimo, and Aleut; and (d) Asian and Pacific Islander. According to the 1990 Census, the White population accounted for 82.5 % and the Black population for 12.4 %. The remaining 5.1 percent were in categories (c) and (d). According to Table 7, Black citizens are over-represented and Whites are under-represented in the studies in this review.

Also according to the Census, the percent of people 25 years and over who have completed four years of high school or more rose steadily from 69 percent in 1980 to 84 percent in 2003. Among the participants in this review (some of whom were under age 25), 43 percent had a general educational diploma (GED) and 46 percent had a high school diploma. A GED is received by high school dropouts who pass a qualifying test.

Study participants can also be classified either as recent applicants of welfare when they enter a study, or people who already receive welfare (recipients). Also, they can be divided into single-parent families or two-parent families. We have not taken these distinctions into account in this version of the review, but we might consider doing this in future updates.

Intervention

Table 5 shows that of 73 endpoints6 we classified 46 as employment-focused. Another name for employment-focused is labour force attachment. This means that we judged that the aim was to try and get participants as quickly as possible into jobs, even if that meant accepting the lowest paying jobs. The rationale behind such a focus is that the best place to acquire jobrelevant skills and build competence is a real workplace. The education-focused approach (also called the human capital development approach) reasons that if you start by educating the participants, they will eventually be qualified for better paying jobs, and not the low paying ones which can not sustain a family economically. For 16 endpoints, the intervention focus was classified as education-focused.

Nine endpoints (labelled “other”) had a detailed screening process at the outset. The results of this screening influenced whether the participant started job search immediately or began with some kind of educational activity. Finally, in two instances we could not decide what type of intervention focus was in place.

The first activity is closely related to the intervention focus in that employment-focused interventions tend to start with job search assistance, job clubs or work placement, while the education-focused ones usually have more varied first activity, often depending on the job-readiness of the individual client. For 24 endpoints, we coded the intervention as job-search first. For 20 endpoints, we coded varied first activity, and for 26 we coded it as other (Table 5).

Sanctioning: If participants did not comply with their assignments in the mandatory programmes, programme managers could sanction them. This could mean denying them their welfare payments. Across all the 34 mandatory programmes (74 percent of the programmes were classified as mandatory), we estimated that about 17 percent of participants were sanctioned.

Reporting of outcomes

As shown in Table 6, most of the 73 endpoints (N = 46, 63%) had data on earnings and employment from state and county administrative records (register data).

Meta-analysis

Figures 3 through 6 show meta-analytic summaries of effects grouped according to follow-up time. For each follow-up time, there are two rows that show the fixed-effects and random-effects estimates. A few studies have results for more than one follow-up. These are included in the rows labelled “Combined”. The effect magnitudes with lower and upper bounds of the confidence intervals are shown in the middle part. The schematic on the right is a summary forest plot. The middle of the (orange-coloured) diamonds is located at the estimated effect magnitude, and their widths indicate the length of the 95 percent confidence interval around the estimate. At the bottom, there are two summary rows which give an impression of the overall estimates (with green diamonds). Note that these summaries are meaningful only if the effects do not vary systematically with follow-up time. For proportional outcomes (proportion employed and proportion on welfare) we use risk ratios. The risk ratio is e.g. the relative “risk” of becoming employed. If, for instance, 55 out of 100 in the intervention group become employed and 50 out of 100 in the control group become employed, the risk ratio is 0.55/0.50 = 1.1. A risk ratio of 1.00 indicates no effect. A number greater than 1 indicates that the intervention group is becoming employed to a greater degree, and a number smaller than 1 indicates that the control group is becoming employed to a greater degree.

For continuous outcomes (such as earnings) we use Hedges’ g. A g of zero means no difference between the intervention group and the control group. Large samples usually have more precise estimates than small samples. This precision is measured by the standard error of the estimate.

Each study is weighted by its precision. This means that large and precise studies are given more weight in the analyses. While the confidence intervals for an individual study can be wide, the meta-analysis confidence intervals are much narrower, signalling a more precise estimate.

Heterogeneity is a measure of the spread of results. Statistical tests of homogeneity (i.e. lack of heterogeneity) assess whether the individual study results are likely to reflect a single underlying effect, as opposed to a distribution of effects. If this test fails to detect heterogeneity among results, then it is assumed that the differences observed between individual studies are a consequence of sampling variation and simply due to chance.

Impact on employment, earnings, welfare payment, and proportion on welfare

The differential effect of employment focus and education focus

As part of the JOBS (NEWWS) evaluation, in three sites (Atlanta, Grand Rapids and Riverside), subjects were randomised into either a HCD (human capital development) condition, a LFA (labour force attachment) condition, or a control condition.

LFA versus HCD

Employment: A random-effects meta-analysis at the three sites showed that the LFA approach had a slightly better effect on employment (risk ratio: 1.094, 95% CI: 1.056-1.133) than the HCD approach (risk ratio: 1.049, 95% CI: 1.003-1.098), but the confidence intervals have a large overlap. Overall, the results were heterogeneous (Q = 25.7, df = 5, p = .000, I2 = 80.5). The subgroups were also heterogeneous (HCD: Q = 8.7, df = 2, p = .013, I2 = 76.9 / LFA: Q = 4.8, df = 2, p = .087, I2 = 59.0)

No direct comparisons on the effects on earnings, welfare payments or welfare proportion were available for the LFA and HCD approaches.

Integrated or traditional case-management

At Columbus, Ohio the participants were randomised into either a traditional case management approach or an integrated approach. In the traditional case management model, welfare eligibility and employment programme functions were performed by separate staff members. In the integrated case management model, these two functions were performed by the same staff member. At two years the two approaches generated almost exactly the same earnings that were also close to the control group (integrated: Hedges’ g = 0.076, 95% CI: 0.018-0.133, traditional: Hedges’ g = 0.075, 95% CI: 0.018-0.132. At five years, the approaches did not have a statistically significant effect on employment, and neither did they differ (integrated: risk ratio = 1.022, 95% CI: 0.999-1.045, traditional: risk ratio = 1.012, 95% CI: 0.989-1.035.

Publication bias

Figures 7–10 are funnel plots showing for each of the four outcomes the relation between each study's effect size and its standard error. In the absence of publication bias the studies will be distributed symmetrically about the combined effect size. By contrast, in the presence of bias, the bottom of the plot would tend to show a higher concentration of studies on one side of the mean than the other. This would reflect the fact that smaller studies (which appear toward the bottom) are more likely to be published if they have larger than average effects, which makes them more likely to meet the criterion for statistical significance. We have used a “trim and fill” function which builds on the key idea behind the funnel plot; that in the absence of bias the plot would be symmetric about the summary effect. If there are more small studies on the right than on the left, the concern is that studies may be missing from the left. The trim and fill procedure imputes these missing studies, adds them to the analysis, and then re-computes the summary effect size. Figures 7–10 indicates that there is some asymmetry in the plots for employment, earnings, and welfare payments, but not for welfare proportion.

Exploratory moderator analyses

Because the results were heterogeneous, we attempted to explore this heterogeneity using moderator analyses. Note that by doing this, randomisation is lost, and, hence, the following are quasi-experimental results. Also note that they are exploratory hypothesis-generating analyses. They are not theory-driven, hypothesis-testing analyses.

Data on a number of variables were collected, but not used as possible moderators. Country where the study was conducted was e.g. not used because all but two studies came from the USA. For the same reason, publication type was not used because the majority of citations were reports. Publication year was not used because we used the year when the data collection started. Design was not used, since all studies used RCT. Some potentially interesting variables were not used because of a large number of missing values (number of children, age of youngest child, previous work experience, previous welfare history, extent of work experience). Finally we did not use data on funding and implementing agents. Studies were mostly funded and implemented by federal and state authorities.