Abstract
2. BACKGROUND
The first paper identifying a condition called autism was published in 1943 (Kanner, 1943). Today, the condition is referred to as autism spectrum disorders (ASD) and refers to a range of mild to severe forms of neurological disorders marked by impairment in social functioning, communication, and repetitive and unusual patterns of behavior.
According to the Centers for Disease Control and Prevention (CDC), the rate of autism is increasing among children and now indicates that 1 in 150 children are identified with ASD (CDC, 2008). Functional limitations caused by ASD continue into adulthood and often create barriers to independent living and stable long-term employment (Autism Society of America, 2008).
As the number of individuals diagnosed with ASD rises and the number of students with ASD exiting public school settings rises, attention is increasingly focused on potential employment outcomes for individuals with ASD. Individuals without severe disabilities are eight times more likely to be employed than individuals with very severe disabilities (National Organization on Disability, 2000). Individuals with ASD are among those least likely to be employed (Cameto, Marder, Wagner, & Cardoso, 2003; Dew & Alan, 2007).
Although economic conditions and employer attitudes affect employment opportunities, employment outcomes for individuals with ASD can be improved by appropriately addressing specific behaviors common among individuals with ASD (Schaller & Yang, 2005). Studies indicate that people with ASD are more likely to lose their employment for behavioral and social interaction reasons rather than their inability to perform work tasks (Dew & Alan, 2007; Hurlbutt & Chalmers, 2004; Unger, 1999). Behavior management challenges affecting employment must be understood and addressed consistently by employment support service providers in order to effectively facilitate obtaining and maintaining employment by individuals with ASD. Relatively few employment support service providers including state vocational rehabilitation counselors have an in-depth understanding of services that are associated with developing successful employment outcomes for individuals with ASD (Dew & Alan, 2007).
Each state in the United States has at least one state agency that is charged with the provision of vocational rehabilitation services to facilitate employment outcomes for eligible individuals with disabilities. Data accumulated through the national network of state vocational rehabilitation agencies indicate that few individuals with autism are requesting services and of those that do most are not successfully employed as an outcome of the service. However, in its latest report, the national vocational rehabilitation service system indicates in fiscal year 2007 it provided services that successfully placed 1,774 individuals with ASD into employment situations that continued for 90 or more days (Rehabilitation Services Administration, 2009).
There has been research done in the area of autism and employment, but such research has not been reported in a comprehensive review format before. Thus, a systematic review on the effectiveness of adult employment assistance for individuals with ASD would not only provide ideas for further research but also show findings that can inform implementation of such research.
More needs to be known about strategies that are successful in securing employment outcomes with individuals with ASD. Enhanced awareness of effective approaches is likely to increase the availability of more effective employment support services for individuals with ASD, such as job coaching, vocational rehabilitation, or supported employment. A review of studies on adult employment assistance would not only synthesize the effects and techniques of successful or unsuccessful interventions, but also call to attention the gaps in experimental research in this area.
3. OBJECTIVE OF THE REVIEW
To determine the effectiveness of adult employment assistance in securing and maintaining employment for adults (18 years and older) with autism spectrum disorders.
4. METHODS
A. Criteria for Inclusion and Exclusion of Studies in the Review
A two-stage process will be used to determine inclusion or exclusion of studies: (1) title and abstract stage and (2) full text stage.
1. Title and Abstract Stage
Studies will be assessed and selected for advancement to the next stage of inclusion if evaluation meets at least one of the two following criteria:
a. Participant
The participant sample of study must include only adults 18 years or older and no longer enrolled in a school-to-work program or secondary-level education program, with a diagnosis of Autism Spectrum Disorder. The review is intended to identify effective interventions that could be implemented by adult service program agencies; therefore, secondary school and other employment programs conducted in school settings with students will be excluded. The authors plan to conduct a second review on the effectiveness of employment assistance for youth and transition-age participants currently enrolled in an educational setting context.
b. Intervention
The focus of the study intervention must center on the topic of employment. The types of employment to be included are competitive, supported, or integrated employment. Sheltered work or non-integrated work will be excluded from the review. Studies that report a range of effectiveness standards from high to low will be included in the review to determine the overall effectiveness of adult employment assistance. Studies that exclusively address the acquisition of job-related skills/behaviors without demonstrating actual employment placement and maintenance of employment will not be included in this review. Studies focusing on job skill acquisition and subsequent employment will be included in this review.
If these criteria are not clear from the title or abstract, the study may be advanced for retrieval of a full text to determine eligibility.
2. Full-Text Stage
A full-text of all citations/abstracts advanced from Stage 1 will be retrieved for a final determination for inclusion in the review and analysis. All of the following criteria must be met by each study in order to be included in the review and analysis.
a. Research Design
Studies will use an experimental (RCT), quasi-experimental design (QED), or single subject experimental design (SSED) design to report the effects of the intervention.
In addition, qualitative studies (primarily case studies) and correlational design studies (COR) will be retrieved and reported to enrich description of intervention effects.
b. Participants
The recipients of the intervention must be individuals with an ASD who are 18 years of age or older and are voluntarily seeking assistance in obtaining employment. Study participants with ASD that are eligible for inclusion are individuals with Asperger Syndrome, Autism, Rhett Syndrome, Childhood Disintegrative Disorder, and Pervasive Developmental Disorder-Not Otherwise Specified, as defined in the DSM-IV-TR and diagnosed by a physician, psychiatrist, or neurologist.
Participants who are not employed at the time of the study intervention are the focus of this review. Authors will not differentiate between participants who have had an employment history from those who do not. Individuals who are employed prior to an intervention studied and continue to be employed during and after the study intervention will not be included in this review.
Study participants with ASD and other disabilities will be included; however, study participants with solely mental retardation, schizophrenia, attention deficits or other non-autism related conditions will not be included. Persons with ASD and secondary conditions will be included.
c. Intervention
The treatment under investigation must be directed towards the provision of adult employment assistance intended to produce employment outcomes for individuals with ASD. The intervention may address social, behavioral, and/or cognitive dimensions related to the acquisition and maintenance of employment among the study participants. The intervention should be focused and involve relatively specific and structured experiences designed to support employment placement, for example, providing guidance in completion of applications, resumes, and engaging in interviews; shaping of work skills and appropriate employment setting social skills; teaching of work-related communication skills; or working directly with employers in the structuring of work and work setting features for individuals with ASD.
d. Outcome Measures
The study must report subsequent attainment of an employment placement and provide specific data about the duration and/or retention of that placement. Gainful employment includes competitive, integrated, or supported employment; sheltered work or non-integrated work will not be considered as an outcome measure. Ten (10) or more hours of employment per week will be considered as gainful employment. Employment will be considered to encompass full or part-time (25% time or more) placements. Interventions of any length duration will be included provided adequate description is provided.
e. Publication Status
Eligible studies may be either published or unpublished.
f. Country of Origin and Language of Publication
Studies may be conducted in any country. We will not exclude studies reported in languages other than English, but we will not specifically search for non-English literature. Assistance in reading non-English studies will be obtained as needed.
B. Search Strategy for Identification of Relevant Studies
The search strategy for identification of relevant studies is highlighted below.
1. Electronic Search
Computerized database searches will be conducted. We will consult database thesauri, where they are available to assure that the universe of appropriate synonyms have been included in the intervention and outcome search term categories. Search terms and search strategies will be modified to fit individual databases.
Databases searched will include the following databases (and others may be added): PubMed NARIC REHABDATA ERIC CIRRIE (Center for International Rehabilitation Research Information and Exchange Database) PsycINFO ARD (Autism Research Database) Sage Family Studies Abstracts Dissertation Abstracts International Cochrane Central Registry of Controlled Trials British Education Index Canadian Research Index Australia Education index CBCA Education Sociological Abstracts Academic Search Complete CINAHL Plus with Full Text Professional Development Collection PsycCRITIQUES PsycEXTRA Psychology and Behavioral Sciences Collection Web of Science Academic One File WorldCat [for monographs] FRANCIS ABI Inform Global Science and Technology Collection PsycARTICLES Business Source Complete CBCA Business ProQuest European Business
2. Search Terms:
The keywords to be used in the computerized bibliography searches will be divided into three categories: population, treatment, and domain. The searches will cover the period from 1943 through 2008. All search terms will be truncated using the DIALOG Database conventions so that they will include variations in endings of words and in spelling. Terms from the three categories will be connected with “or” within each category and will be connected with “and” between categories.
autis? childhood(w)disintegrative(w)disorder? pervasive(w)developmental(w)disorder? pervasive(w)developmental(w)disorder(w)not(w)otherwise(w)specified? spectrum(w)disorder? Asperger?
treatment? intervention? model? program? practice? instruction? training? service? supported employ?
employ? adult? rehabilitation? vocational? workplace?
3. International Contacts
Our efforts to find studies from outside the United States will include corresponding with researchers in countries known for their efforts in autism research (e.g. United Kingdom, Canada, and Australia). Correspondence will ask for their suggestions and help in locating eligible studies, particularly unpublished and fugitive literature. Recent reviews on autism research outside the U.S. will provide additional leads and addresses of contact persons.
4. Grey Literature
Grey literature that is identified through electronic searches will be submitted to the same inclusion criteria as other studies. A time range for these types of studies has not been specified in order to maximize consideration of all relevant grey literature. Reference lists from other systematic reviews and individual studies will be searched for potential studies to consider for inclusion.
5. Cross-referencing of Bibliographies
The references in relevant journal articles, systematic reviews, and other reports of research results will be scanned for new additions to our literature for review.
6. Conference Programs
Recent conference programs and conference syntheses will be reviewed for leads about eligible literature for review. Professional organizations that will be reviewed include: Autism Society of America National Association of Rehabilitation Research and Training Centers International Society for Autism Research National Alliance for Autism Research Autism Research Institute National Autistic Society (UK) Autism Research Centre (Cambridge)
Conferences that would be reviewed include, for example, Asia Pacific Autism Conference 2009 and the PENN Autism Network Conference.
C. Coding Procedures and Categories
Studies will be coded for inclusion/exclusion decisions at two stages, Stage 1: citation and abstract and Stage 2: full-text. Two coders will serve as independent reviewers at both stages. A third party will be used in the case that the reviewers disagree and cannot resolve a final coding value.
1. Citation and Abstract Stage
At Stage 1, the decision for advancing the retrieved citations and abstracts to the full text stage retrieval will be made by a single reviewer based on meeting two items from a, b, and c of the following questions or a designation by the reviewer of ‘unsure’ (item d): Are the participants identified, described, and defined under the Autism Spectrum Disorder category? Are the participants 18 years of age or older? Are the participants no longer enrolled in a school-to-work transition program or secondary-level education/program? Is this abstract/citation about employment? Unsure of meeting inclusion criteria?
If the reviewer is ‘Unsure’ the citation/abstract will be advanced to the Full-Text for a final inclusion decision.
2. Full-Text Level
At the Full-Text Stage 2 level, a full text of all citations advanced from Stage 1 will be obtained and coded for an inclusion/exclusion decision. The decision for advancing the retrieved full-text studies to an inclusion status will be made by two reviewers for each study, independently evaluating each study. An inclusion decision for advancement to the coding stage of the process will require that a study meets all the criteria presented earlier. Inter-rater reliability will be established prior to initiating coding activities, thus minimizing coding disagreements. Any disagreements in the determination of the presence of these criteria will be resolved by discussion of the two reviewers. In the event reviewer differences cannot be resolved via consensus building, a third reviewer will be asked to render a final decision.
At the Full-Text Stage 2 level, two reviewers will record all excluded studies and the reasons for exclusion independently.
Multiple studies that use the same sample or data will not be included. The most rigorous study focusing on our desired intervention outcome will be selected for inclusion.
D. Assessment of Methodological Quality
Included studies will be coded by two independent reviewers for methodological quality on dimensions such as: Design type RCT Individual Randomized Design RCT Group Randomized Design Quasi-Experiment: Equivalent Comparison Design (individuals) Quasi-Experiment: Equivalent Comparison Design (groups) Quasi-Experiment: Nonequivalent Comparison Design (individuals) Quasi-Experiment: Nonequivalent Comparison Design (groups) Quasi-Experiment: Regression Discontinuity Single Group Quasi-Experiment: Interrupted Time Series Design Single Group Pretest/Posttest design ABA Single Subject Design
1
MBL Single Subject Design Survey: Cross Sectional Survey: Longitudinal Unit of assignment (e.g., individual vs. group/class) Unit of analysis (e.g., Intention to Treat, Test only, Treated) Attrition from pretest to posttest Fidelity of implementation (e.g., following replicable program of intervention) Blinding of assessors/interventionists
These dimensions will be analyzed as moderator or sub-group analyses to assess their potential explanatory impact on the effect of the intervention.
Other data that will be extracted from the primary studies include: publication source, subject characteristics, sample source, employment setting, intervention characteristics, type of employment, and outcome measurement (see Appendix B for Coding Form).
E. Calculating Effect Size
The magnitude of the intervention effect will be calculated using the commonly accepted statistical formulae and dedicated programs available. While the outcome data may be available in different formats, we believe that most of the data will be presented in such a way as to require one of the following statistical approaches.
1. Standardized Mean Difference Statistic (d-index)
Studies that report means and standard deviations for the experimental and control groups, the standardized mean difference statistic (d-index) will be used with the pooled standard deviation being the variance statistic for all calculations. For studies that report statistics such as t, F, or p-values and the accompanying sample sizes only, conversion formula will be used to calculate the d-index for the effect size estimate.
In addition, the calculated d value will be corrected for sample size using Hedge's g, which accounts for studies with small sample sizes.
2. Confidence Interval
All effect sizes will be calculated using a 95% confidence interval.
3. Odds Ratio Family
Studies reporting binary data in which mean outcomes are compared in the experimental and control (or comparison groups) will be summarized using the odds ratio derivative statistic. A 95% confidence interval for the odds ratio, risk difference, or risk ratio statistics will be used to report all effect sizes.
4. Percentage of Non-overlapping Data (PND).
The most commonly used statistic for calculating an effect size using single subject design data is the PND. The PND provides for the calculation of the non-overlapping difference of measured data points when comparing baseline and subsequent treatment (or generalization) phases. The procedure involves identifying the highest baseline data point and calculating the proportion of data points in the intervention phase that are higher than the data points in the baseline. The PND metric is a simile percentage value that scales the magnitude of the intervention impact as follows: 90%+ = highly effective, 70% to 90% = moderately effective, 50% to 70% =questionable effectiveness, and less than 50% = ineffective (Scruggs, Mastropieri, Cook, & Escobar, 1986). Authors will report a PND calculation as not being possible if study data are inappropriate for such calculations.
5. Percentage of Zero Data (PZD)
The PZD statistic is another statistic used for analysis of single subject experimental design effect size calculation. This metric will be used when the goal of the intervention is to decrease performance, which might be in the case when challenging behaviors prevent employment placement/outcomes. Hence, the reduction in challenging behaviors would be assessed with the PZD. PZD is calculated by identifying the first data point to reach zero during the intervention phase of the program and calculate the percent of zero treatment data points from that point onwards. The PZD scores are interpreted as follows: above 80% is considered highly effective, 55% and 80% is fairly effective, 18% and 54% is questionable effectiveness, and below 18% reflects unreliable treatment (Scotti et al., 1991).
F. Synthesis of Effect Sizes
Only studies using a two group experimental or quasi-experimental design will be included in the data synthesis. Studies reporting either a pre-experimental single group pre- posttest or single subject experimental design will be analyzed separately and not combined with other design types. Studies that utilize PND and PZD will be included in the synthesis efforts.
1. Within Study Synthesis
In anticipation that some studies will report multiple outcomes using the same experimental and control group comparisons, we will calculate an effect size for all comparisons individually. The synthesis of effect sizes within a study will be conducted by the aggregation of all effect sizes in order to produce an overall measure of treatment effect for that particular study. Because extracting multiple effect sizes from a single study may result in a violation of independence among the data, if a single study reports multiple outcomes, then the overall intervention effect size will be represented by the mean value of all study outcomes in a single study effect size.
Individual multiple effect sizes from the same study will not be synthesized in order to avoid the violation of the principle of data independence in the analysis.
2. Across Study Synthesis
The synthesis of effect sizes across conceptually similar constructs will be conducted in order to determine the magnitude of the effect when combining similar outcome effects from several studies. However, all cross study synthesis will utilize only one comparison effect size per study for any summary synthesis so that no single study outcome is represented more than one time in any analysis. For any group study (e.g., RCT, QED) any odds ratio effect sizes will be converted to a d-index effect size so that all effect sizes for all group studies will be presented in the same metric. For the single subject design studies, the PND will not be combined with the d-index metric but will be presented and analyzed separately.
3. Heterogeneity Analysis
For the analysis of dichotomous and continuous data, an assessment of heterogeneity will be conducted. Both fixed and random effects models will be calculated and reported in order to assess the difference in the magnitude of the intervention effect under different sampling error assumptions. Interpretation of the results can then be gauged in the context of the underlying model assumptions sampling homogeneity.
4. Sensitivity Analysis
A sensitivity analysis will be conducted to assess the impact of a single study on the magnitude of an overall observed effect size. The sensitivity analysis will be conducted for overall study effect size and as appropriate, the impact of moderating variables (e.g., attrition, type of treatment, missing data, sample size, study design).
5. Publication Bias
Publication bias will be assessed for published vs. unpublished included studies through the visual inspection of a funnel plot and/or a moderator analysis.
6. Incomplete Reporting of Study Data
For studies reporting incomplete outcome data, we will first contact the senior author of the study and request the missing information to include in the analysis. If we are unable to obtain the needed information, we will attempt to use a method of imputation in which we will set the effect for the missing data to zero and then calculate the outcome effect size. Reported results will be presented both with and without the imputed data in order to assess the magnitude of the impact of the imputation process.
7. Subgroup and Moderator Analyses
Subgroup/Moderator analyses will be conducted for a limited number of independent variables. The approach to the moderator analyses will be dependent on the available data. Meta-regression will be used if a sufficient number of studies are present to reasonably regress multiple factors on to the treatment effect size. At a minimum, the following categorical variables will be considered for moderator analyses if appropriate: Length of Employment Job Types Length of Employment Assistance Type of Intervention Costs of the Intervention Participant Severity Level Intention to Treat vs. Active Treatment only analysis models h. Age of Participant Educational Attainment of Participant
G. Treatment of Qualitative Research
Qualitative research will not be used in the analysis of the intervention research. However, we will collect and summarize relevant qualitative research such as survey, correlational studies, and case studies to provide a context for the interpretation of the quantitative results of any data synthesis or meta analysis.
5. TIMEFRAME
Once the protocol is approved, the authors anticipate submitting a first draft of a completed review to the Education Coordinating Group within six months.
6. PLANS FOR UPDATING THE REVIEW
The authors will examine the review every three years for update.
7. ACKNOWLEDGEMENTS
The concept for this systematic review was included in a proposal submitted to and funded by the National Institute on Disability and Rehabilitation Research in the Office of Special Education and Rehabilitative Services in the U.S. Department of Education.
8 STATEMENT CONCERNING CONFLICT OF INTEREST
No conflicts of interest are known.
Footnotes
Appendix A
Appendix B
1
Single subject experimental designs will be included when examining intervention that focus on skills related to the direct acquisition of an employment outcome.