Scaling Out Virtual Interview Training for Transition-Age Youth: A Quasi-Experimental Hybrid Effectiveness-Implementation Study

Abstract

In a previous randomized controlled trial (RCT), Virtual Interview Training for Transition Age Youth (VIT-TAY) enhanced employment for autistic youth receiving transition services. For this study, a nonrandomized hybrid effectiveness-implementation trial evaluated whether VIT-TAY maintained its core implementation and effectiveness functions when scaled out to students with disabilities in 32 schools. Also, we compared the implementation and effectiveness between VIT-TAY and Virtual Reality Job Interview Training (VR-JIT; developed for adults and previously evaluated in students with disabilities). Core implementation functions did not differ between VIT-TAY and VR-JIT. Greater engagement with either training was significantly associated with employment (both p < .05). Overall, VIT-TAY was feasibly implemented with high adherence and may be effective at enhancing employment for students with disabilities. Limitations and implications for research and practice are discussed.

Keywords

virtual reality job interview training employment transition services youth with disabilities

Pre-employment transitions services (Pre-ETS) consist of workplace readiness training (e.g., job-seeking social skills), work-based learning experiences, job exploration counseling, postsecondary education counseling, and self-advocacy to help prepare students with disabilities for the transition from secondary education to employment. Pre-ETS is federally mandated by the Workforce Innovation and Opportunity Act (WIOA, 2014) to serve 14- to 21-year-old special education students commonly referred to as “transition-age youth” (WIOA, 2014).

A recent study of 656 youth receiving Pre-ETS revealed that 88.8% of employed youth completed an interview prior to obtaining their job (M. J. Smith, Sherwood, Blajeski, et al., 2021), suggesting the importance of navigating job interviews for employment. Relatedly, job interview skills were identified as a critical component of job readiness training by the U.S. Department of Education, Office of Special Education and Rehabilitative Services (2017). Despite research on job interview training in the disability and rehabilitation fields since the 1970s (e.g., Barbee & Keil, 1973; Parsons, 2006), a recent review of secondary transition practices does not list any job interview interventions as evidence-based practices (Rowe et al., 2021). However, a single study focused on interview skills was identified as a research-based practice. This study used a prototype video modeling application that was preloaded with seven interviews to train job interview skills (Hayes et al., 2015). This pilot randomized controlled trial (RCT) with 15 participants revealed improved job interview skills, but did not evaluate whether these skills were associated with obtaining employment (Hayes et al., 2015). Thus, signaling a critical need for rigorous testing of job interview interventions.

Meanwhile, emerging literature of mostly pilot studies highlighted the initial efficacy of multiple job interview interventions at improving interview skills for individuals with a range of disabilities. Some of these interventions used resource-intensive, in vivo mock interview role-play methods with teachers (e.g., Rosales & Whitlow, 2019), while others used virtual learning environments (VLEs; e.g., virtual reality, computerized simulations; e.g., Burke et al., 2018; M. J. Smith et al., 2015). Specifically, VLEs (including augmented reality) have been embraced by schools to support the study of science, technology, engineering, mathematics, and vocational social skills within special education (e.g., Gregg et al., 2017; Kellems et al., 2019). Also, VLEs have enhanced engagement, interactivity, student motivation, and teacher efficiency to provide more individualized instruction (Politis et al., 2017; Thorsteinsson & Shavinina, 2013). Furthermore, VLEs have greater scalability than non-VLE interventions, create nonthreatening environments to make mistakes and receive judgment-free feedback, and are more acceptable and preferable by students with disabilities (e.g., Spencer et al., 2019).

Recently, two VLEs teaching interview skills moved beyond pilot studies and were scaled up in different settings. First, Burke et al. (2021) used a nonrandomized, single-arm open design to scale up Virtual interactive Training Agent (ViTA; clinicians facilitated mock job interviews through an avatar) among 153 trainees with disabilities recruited from secondary or postsecondary education settings. The study did not further characterize how ViTA was implemented. Although the results suggested the intervention was associated with improved job interview skills, the association with employment was not tested (Burke et al., 2021).

Second, a different approach evaluated taking Virtual Reality Job Interview Training (VR-JIT) to scale. This tool was originally developed and tested in lab studies among adults with serious mental illness (e.g., schizophrenia; M. J. Smith, Ginger, Wright, Wright, Humm, et al., 2014) and autism (M. J. Smith, Ginger, Wright, Wright, Taylor, et al., 2014) and later piloted in high school settings for students with autism (Genova et al., 2021). VR-JIT is an internet-delivered interview simulator that includes (a) a didactic eLearning review of interview skills and interviewing tips, (b) a job application that informs the simulated interview, and (c) a virtual interview to repetitively practice eight interview skills with a White female interviewer. VR-JIT provides four levels of feedback: (a) real-time nonverbal cues from an automated coach; (b) a color-coded transcript with statement-level feedback; (c) a numerical score (0–100); and (d) a qualitative performance assessment. The virtual interviews have three difficulty levels (i.e., easy [friendly interviewer], medium [professional interviewer], and hard [inappropriate interviewer]).

Specifically, this study of VR-JIT used a hybrid effectiveness-implementation design (Curran et al., 2012) to focus primarily on learning how VR-JIT was implemented in 15 schools among 279 students with disabilities receiving Pre-ETS while secondarily evaluating the impact of VR-JIT on employment (M. J. Smith, Smith, et al., 2021). The results indicated that 28 teachers implemented VR-JIT with a few modest and infrequent adaptations (e.g., implementing at home or in the community) to a prospective implementation strategy. Teachers and students reported high acceptability, and students reported high VR-JIT usability. Improved student performance on the VR-JIT interviews over time translated into greater odds of obtaining competitive employment.

Although the results of the aforementioned study were promising (M. J. Smith, Smith, et al., 2021), VR-JIT was originally designed to meet the needs of adults with serious mental illnesses (M. J. Smith, Ginger, Wright, Wright, Humm, et al., 2014). Thus, we adapted VR-JIT to meet the needs of transition-age youth with autism and others engaged in Pre-ETS—now called Virtual Interview Training for Transition Age Youth (VIT-TAY; M. J. Smith et al., 2020). VIT-TAY was evaluated in an RCT across five schools with 71 students with autism (M. J. Smith, Sherwood, Ross, et al., 2021). The results showed that participants randomized to Pre-ETS with VIT-TAY, compared with Pre-ETS only, had increased interview skills, reduced interview anxiety, and a higher rate of competitive employment by follow-up. We then sought to scale out VIT-TAY to students with disabilities beyond autism.

Scaling-out, an implementation science term defined by Aarons et al. (2017), reflects a deliberate effort to adapt an intervention and broaden its delivery to (a) a different setting with the same population (b) a different population within the same setting; or (c) both a different setting and a different population simultaneously. We used a hybrid effectiveness-implementation study design (Curran et al., 2012) to evaluate whether scaling out the adapted VIT-TAY in the same type of school settings would differ in effectiveness on employment and implementation strategies used when delivered among a different population (i.e., a sample of all students with disabilities engaged in Pre-ETS). As with any adaptation, ensuring that effectiveness remains comparable with the original intervention is critical (J. D. Smith, Berkel, Rudo-Stern, et al., 2018), alongside salient implementation processes and outcome variables (e.g., acceptability, engagement) that affect overall intervention impact (Glasgow et al., 2019). Thus, this study used a single-arm, nonrandomized hybrid design to compare the implementation process and employment outcomes of youth with disabilities in Pre-ETS from two cohorts: one receiving the original VR-JIT (i.e., adult version; M. J. Smith, Smith, et al., 2021) and one receiving the adapted VIT-TAY.

We hypothesized that the implementation processes (e.g., feasibility) and effectiveness outcomes (employment) would be comparable (noninferior) or better for the adapted VIT-TAY, compared with VR-JIT, given that VIT-TAY was designed to meet the needs of this population. Finally, we explored the relationship between the implementation process data and employment outcomes. Accordingly, three research questions guided this study comparing VIT-TAY to VR-JIT:

Research Question 1: Are the implementation processes comparable?

Research Question 2: Are the effectiveness outcomes comparable?

Research Question 3: What relationships are there between implementation processes and employment outcomes?

Method

The present study compared the implementation and effectiveness between the published evaluation of VR-JIT in 15 schools with 279 students with disabilities and our recent delivery of VIT-TAY in 32 schools with 356 students with disabilities. The Institutional Review Board (IRB) reviewed and designated this study as exempt human subjects research. No identifying student information was collected by the research team as the study primarily focused on evaluating the implementation of VIT-TAY. Informed consent was not required by the IRB.

Recruitment

School-level recruitment

Schools were recruited in Illinois, Michigan, and Florida. First, the Illinois Division of Rehabilitation Services (I-DRS) connected us with administrators for the Illinois Secondary Transitional Experience Program (STEP) network of (approximately 700) individual schools, school districts, regional educational cooperatives, and special education therapy and learning centers serving approximately 12,000 youth receiving secondary Pre-ETS. Second, we partnered with administrators from Chicago Public Schools (CPS) Office of Diverse Learner Supports and Services to obtain approval from the CPS Research Review Board to recruit schools. The CPS review board required their local teachers to obtain consent for student participation, which was regulated by CPS. Third, Michigan Rehabilitation Services’ administrators recommended partnering with the Michigan Career and Technical Institute (https://www.michigan.gov/mcti), whom they sponsor as a public postsecondary transition program annually serving more than 1,000 transition-age youth with disabilities that implements a standardized transition curriculum focused on vocational and technical training in 13 trades (e.g., electronics, retail). Fourth, Project SEARCH is an evidence-based, pre-employment training program integrated with state vocational rehabilitation services at 400 sites in the United States (Rowe et al., 2021). We recruited Project SEARCH sites and schools not yet implementing Project SEARCH from their annual conferences in Michigan and Florida. Finally, additional schools in Illinois and Michigan were referred through their own networks to approach the study team to participate.

We initially corresponded with 82 schools, and 52 schools completed training to implement VIT-TAY. However, 20 schools that completed training dropped out, citing shifting priorities or a lack of resources; or failed to respond to follow-up correspondence. Overall, 32 schools (14 Illinois, 12 Florida, six Michigan) participated. The National Center for Education Statistics Search for Public Schools database determined the locale subtype (city, suburban, town, or rural) for each school using 2016–2018 school-year data (Geverdt, 2015).

Staff-level recruitment

School-level administrators approached teachers leading Pre-ETS training with an opportunity to participate in the study. Teachers were free to participate or decline, and, to our knowledge, they were not unduly influenced by administrators. Overall, eight administrative leaders and 47 teachers (22 teachers also served as administrative leaders) agreed to participate. Demographic characteristics for these 55 staff members can be found in Supplemental Table 1.

Student-level recruitment

Participating teachers offered VIT-TAY to all their students ages 15 to 26 years at baseline (defined as the first day of the semester during which VIT-TAY was implemented) who were receiving Pre-ETS. Although Pre-ETS begins at age 14, this age group was excluded as they would not be eligible for competitive employment, whereas 15-year-old students can obtain a work permit. Students ages 22 to 26 years were included due to a Michigan state law that allows provision of the Individuals with Disabilities Education Improvement Act (IDIEA, 2004) to serve students with disabilities up to 26 years of age (Michigan Administrative Rules for Special Education, 2021). Students also had to be designated with at least one of the 13 disability categories according to IDIEA, including autism, deafness, emotional disturbance, and intellectual disability. Overall, 356 students chose to use VIT-TAY during the 2018–2019 academic year. Teachers did not report the number of students who declined to use VIT-TAY or did not meet eligibility criteria. Notably, the aforementioned VR-JIT evaluation included students ages 16 to 22 years. Students through age 21 were served by IDIEA and state-level vocational rehabilitation, while students age 22 were served by state vocational rehabilitation because the upper age limit for IDEA funding is 21.

Students were eligible to participate whether they were currently employed part-time or were not job-seeking at enrollment, as their status could change during receipt of Pre-ETS, and our goal was to evaluate the implementation of VIT-TAY within Pre-ETS. Teachers used student records to report students’ demographics, intelligence quotient (IQ) and reading level, job-seeking status, and employment history at baseline and at 3- and 6-month follow-ups. IQs were generated via standard methods (Schrank et al., 2014; Wechsler, 2014). Comparisons of demographics between students using VIT-TAY or VR-JIT are in Table 1.

Table 1.

Characteristics of Students Participating in the VR-JIT/VIT-TAY Comparison Study

Variables	VIT-TAY (n = 356)	VR-JIT (n = 279)	Test statistic (t or χ²)	p value
Mean age (SD)	18.7 (2.2)	18.6 (1.5)	0.55	.58
Male	65.4%	64.2%	0.12	.74
Race
White	51.4%	68.1%	41.57	<.001
African American	26.4%	11.8%
Latinx	12.4%	15.4%
More than 1 race	4.2%	2.2%
Asian American	3.9%	0.7%
Other	1.7%	1.8%
Mean overall IQ (SD)	74.8 (14.2)	77.4 (15.3)	1.58	.12
Grade
Freshman or sophomore	8.7%	6.8%	96.97	<.001
Junior	10.4%	9.7%
Senior	29.2%	65.9%
Adult transition	51.7%	17.6%
Reading level
Below fourth grade	25.9%	23.0%	2.08	.56
Fourth grade	13.0%	11.9%
Fifth grade	7.9%	10.8%
Sixth grade and higher	53.2%	54.2%
Disability categories
Specific learning disability	36.5%	36.2%	0.01	.93
Autism	22.8%	25.1%	0.47	.49
Other health impairment	19.4%	28.7%	7.52	.006
Intellectual disability	16.9%	14.7%	0.55	.46
Emotional disturbance	13.5%	19.4%	4.00	.046
Speech and language impairment	0.0%	7.5%	27.71	<.001
Geographic locale
City	43.8%	4.7%	123.58	<.001
Suburban	43.0%	69.5%
Town	11.5%	22.6%
Rural	1.7%	3.2%

Note. IQ = intelligence quotient; VIT-TAY = virtual interview training for transition-age youth; VR-JIT = virtual reality job interview training.

Standard Pre-ETS in Participating Schools

All schools facilitated Pre-ETS in the areas of social communication; work readiness (e.g., résumé review, job skill development, and mock job interviews); self-advocacy; disability awareness; and independent living skills (e.g., self-care and money management) via classroom instruction. The Illinois STEP schools facilitated work-based learning experiences. Students received job exploration counseling to find employment opportunities, on-the-job support and coaching, and remained connected with Illinois-Department of Rehabilitation Services (I-DRS) to receive postsecondary counseling after graduation (n = 68 students). Chicago Public Schools in Illinois facilitated community-based training to provide Pre-ETS and employment supports prior to graduation and links with I-DRS after graduation (n = 99 students). Project SEARCH facilitated the practice of classroom learned skills during a series of internships (i.e., work-based learning) organized and managed by the public education system, state vocational rehabilitation services, community service providers, and community employers (n = 61 students; Wehman et al., 2019). Job exploration counseling or postsecondary counseling may or may not be provided within Project SEARCH as it depends on the qualifications of the job developer. That said, job exploration and postsecondary counseling are not required in the Project SEARCH fidelity manual. Michigan Career and Technical Institute facilitated intensive technical and hands-on training in 13 trades with intensive supports from Michigan Rehabilitation Services after graduation (n = 37 students), while other schools (including some students who only attended STEP class) did not include (or receive) these additional trainings or state-level supports (n = 91 students). Overall, teachers reported on Pre-ETS related to directly accessing employment.

To assess possible differences between the implementation of VIT-TAY (current cohort) and the VR-JIT (prior cohort), teachers were surveyed on services provided for all students. No significant differences were found between conditions (VIT-TAY vs. VR-JIT, all p > .09) on teacher engagement in classroom-level resume review (48.9% vs. 58.6%), job skill development (61.0% vs. 71.2%), mock job interviews with teachers (45.1% vs. 40.3%), or mock job interviews with community employers (25.1% vs. 12.3%).

Procedures

As an adaptation of VR-JIT, VIT-TAY was designed to be an internet-delivered, scalable, individualized learning experience that students can use independently with minimal orientation and supervision. VIT-TAY used similar features as described earlier, while tailored adaptations included diverse interviewers (African American male and Latinx female), a token economy system, three additional learning goals across interview difficulty levels, screen reader capabilities, and social story-telling with the use of video and audio to reduce cognitive load. Additional VIT-TAY details can be found here (M. J. Smith et al., 2020; M. J. Smith, Sherwood, Ross, et al., 2021). For this study, we trained teachers to implement VIT-TAY using the strategies aligned with the VR-JIT cohort (M. J. Smith, Smith, et al., 2021). VIT-TAY delivery procedures and both implementation and effectiveness data collection methods were the same as the prior VR-JIT study, except when noted. Effectiveness-implementation hybrid trials are defined by a shared and simultaneous examination of the effects of the intervention on patient outcomes (e.g., employment) and the implementation processes and outcomes (Curran et al., 2012). This study used the Type 3 variant, which means a greater focus on implementation relative to the effectiveness of the intervention.

Recommended VIT-TAY curriculum

Prior VR-JIT efficacy studies found that completing 15 virtual interviews was associated with improved interview performance and increased access to jobs (e.g., M. J. Smith et al., 2015). Thus, we recommended that teachers encourage students to complete 15 virtual interviews while they facilitated students’ transitions from easy to hard difficulty levels via a progress tracking form (see online supplemental materials Figure 1). Moreover, we recommended that students complete approximately three 45- to 60-min VIT-TAY sessions per week over 4 to 6 weeks to achieve the recommended 15 virtual interviews. We advised teachers to focus at least two sessions on reviewing 10 interview skills highlighted in the eLearning content. Overall, we encouraged teachers to adapt the aforementioned implementation plan to fit within their everyday teaching and the needs of their students to optimize adherence. Teachers logged these adaptations in a post-session survey.

The VIT-TAY rubric required students to try and obtain a score of at least 90 (out of 100 possible points). The scores were based on points awarded for performance (i.e., selected interview responses) that map onto the interview skills highlighted in the eLearning content. Easy interviews targeted four interview skills, medium interviews targeted an additional three interview skills (seven total), and hard interviews targeted an additional three interview skills (10 total; see Table 2). We recommended that students have a maximum of five interview attempts at the easy (or medium) level before transitioning to medium-level (or hard-level) interviews. Teachers were responsible for facilitating the progression between difficulty levels. Students were not incentivized by teachers or the research team to engage in VIT-TAY. Teachers were not masked to study condition as this was a nonrandomized trial.

Table 2.

Scaffolded eLearning Interview Skill Lessons for VIT-TAY.

Interview skills	Points awarded on easy	Points awarded on medium	Points awarded on hard
Easy: “Are you someone they want to work with?”
Be confident	25	15	10
Be positive	25	15	10
Be professional	25	15	10
Show interest	25	10	10
Medium: “Will you be a good worker?”
Be honest	—	15	10
Be dependable	—	15	10
Work well with others	—	15	10
Hard: “Will you do a good job?”
Share strengths and skills	—	—	10
Share past experiences	—	—	10
Share past limitations	—	—	10
Total Points Scored	100	100	100

Note. VIT-TAY = virtual interview training for transition-age youth.

VIT-TAY implementer orientation and training

School staff attended a 60-min orientation, led by the research team, on how to implement VIT-TAY with the use of an electronic, comprehensive adherence checklist (Sherwood et al., In Press; M. J. Smith, Smith, et al., 2021). Notably, four administrative leaders attended the orientation but did not return the evaluation, and five teachers attended the orientation as backups, but did not implement VIT-TAY. After the orientation, teachers used VIT-TAY for 90 min themselves to obtain practical knowledge of the tool. This time was monitored and validated by the research team using the VIT-TAY administrative portal (i.e., a website for monitoring teacher and student engagement with VIT-TAY). Independent of the videoconference, teachers completed two role-plays with a peer (i.e., one as teacher and one as student) to practice using the fidelity checklist. After orientation, teachers informed their school-level supervisor of their readiness to implement VIT-TAY. School-level supervisors reported role-play completions and teacher preparedness to the research team. Adherence to training was met as all teachers completed the VIT-TAY practice, role-plays, and reported preparedness to deliver VIT-TAY. Teachers were trained to implement VIT-TAY within Pre-ETS over 6 to 10 weeks and used the checklist to report implementation adherence. Adherence was reported to the order in which students were taught each aspect of VIT-TAY, based on the a priori implementation plan. Finally, teachers marked the checklist for the items they taught to students and submitted their checklists to the research team.

Data Collection and Study Measures

We captured all data from teachers and students via electronic surveys completed at the schools. Leader and teacher surveys were sent via an online data capture tool compliant with the Family Education Rights and Privacy Act of 1974 called Research Enterprise Data Capture (REDCap; Harris et al., 2009). Links to student surveys were sent electronically to teachers via Qualtrics (2005), and teachers forwarded the links directly to the students.

VIT-TAY implementation process outcomes

VIT-TAY automates the capture of completed virtual interviews (at each difficulty level and total); scores for each interview (at each difficulty level); mean and highest score attained (range: 0 to 100); and minutes talking with the virtual interviewers (at each difficulty level and total). We conducted a principal component factor analysis using a high score, total interviews completed, and total minutes interviewed to compute a composite measure of “VIT-TAY engagement” to reduce the number of contrasts and reduce measurement error. Factor analysis (no rotation) results revealed a good fit to a one-factor solution (factor loading range: 0.597–0.936). Intercorrelations (Pearson’s r) ranged from .32 to .85 (all p < .001). A table of intercorrelations between all variables is available upon request.

VIT-TAY scale-out implementation outcomes

VIT-TAY orientation acceptability (pre-implementation)

This survey (seven items; 0 [not at all] to 4 [very]; α = .95) evaluated the acceptability of the orientation we provided to train teachers and leaders to teach students how to use VIT-TAY. Sample items reflected satisfaction with the orientation and satisfaction with the opportunity to practice VIT-TAY.

VIT-TAY appropriateness and expected implementation feasibility (pre-implementation)

We evaluated teachers’ and leaders’ perceptions of VIT-TAY as an appropriate tool for Pre-ETS prior to implementation (five items; 0 [not at all] to 4 [very]; α = .87; summed for a total score ranging from 0 to 20). Sample items reflected VIT-TAY appropriateness for training interview skills and whether VIT-TAY fit with students’ learning goals. We evaluated teachers’ and leaders’ confidence in the expected feasibility of implementing VIT-TAY (nine items; 0 [not at all] to 4 [very]; α = .84; summed for a total score ranging from 0 to 36). Sample items reflected the perceived ease of VIT-TAY delivery.

VIT-TAY implementation context

We used the Stirman et al. (2017) adaptation coding taxonomy for teacher reports of implementation context (six items; sample items reflect the level of assistance students needed to complete the training) and for strategies actually used during delivery (four items; sample items reflect how VIT-TAY was delivered). Teachers completed this survey at the delivery midpoint and endpoint.

VIT-TAY acceptability and sustainability (post-implementation)

We evaluated leaders’ and teachers’ (n = 49) perceptions of VIT-TAY acceptability after implementation (10 items; 0 [not at all] to 3 [very much]; α = .85; summed for a total score ranging from 0 to 10). Sample items reflected satisfaction with VIT-TAY as a service and confidence about their delivery of VIT-TAY. Leaders and teachers (n = 49) evaluated the sustainability of delivering VIT-TAY with three items evaluated independently (0 [not at all] to 3 [very]): “How motivated are you to continue to deliver VIT-TAY to students?”; “How disruptive will it be to your daily work routine to continue to use VIT-TAY with your students?” (reverse coded); and “How equipped is your school to support the continued delivery of VIT-TAY?”

The student acceptability survey (five items; 1 [not at all true] to 5 [very true]; α = .75; summed for a total score ranging from 5 to 25) included items with content focused on the enjoyment of use. We evaluated student-reported usability using the total score from six items (1 [not at all] to 5 [very much]; α = .82). Sample items reflected readiness to use VIT-TAY after orientation. We created an analogous six-item version of the original seven-item scale assessing VR-JIT usability for comparison (α = .80).

VIT-TAY effectiveness outcomes

Employment reflected a competitive, integrated community job. Job-seeking status was coded as 0 = unemployed, actively seeking employment; 1 = unemployed, not seeking employment; or 2 = employed, part-time. The employment outcome variable was coded as 0 for youth who remained unemployed between baseline and follow-up, or who were employed at baseline and then, became unemployed and remained unemployed at follow-up. A code of 1 reflected youth (who were either unemployed or employed at baseline) who obtained new jobs between baseline and follow-up. Youth who were employed at baseline and maintained that employment through follow-up were also identified. Teachers completed 3- and 6-month follow-up surveys for 160 youth; 115 youth had a 3-month follow-up only; 74 youth had a 6-month follow-up only; and seven youth did not have follow-up data. Thus, we obtained follow-up data on 349 (98.0%) of 356 youth. For youth whose teachers only reported the 3-month follow-up, we carried forward their outcome to their 6-month follow-up.

Data Analysis

The processed data from the prior VR-JIT study were used for the current evaluation, thus allowing statistical comparisons between the studies. All analyses were conducted with SPSS 26.0. Given the variability in school locale (city vs. suburban vs. town vs. rural) and transition program type (STEP vs. Michigan Career and Technical Institute vs. Chicago Public Schools vs. Project SEARCH vs. others), we conducted a design-effect analysis (the ratio of the operating variance to the sampling variance if a simple random sample were conducted [Muthen & Satorra, 1995]) on the employment outcome to evaluate the amount of variance corresponding to nesting within locale and transition program type. This variation could influence estimates of standard error and require multilevel analyses if statistically significant. Design-effect test statistics that are ≤2.0 reflect nonsignificant variation in the nested data structure and do not require multilevel analyses (Muthen & Satorra, 1995). Both locale and program type had design-effect estimates below 2.0. Thus, we did not analyze the multilevel nature of the study design.

Descriptive analyses characterized the process outcomes (i.e., VIT-TAY performance), demographics, IQ, and employment history. We evaluated the core determinants, processes, and outcomes germane to comparing adapted interventions during scale-out. Specifically, we compared VIT-TAY and VR-JIT implementers’ perspectives on training orientation; initial tool acceptability; appropriateness and expected implementation feasibility; implementation context and adaptation; and post-implementation acceptability and sustainability (Proctor et al., 2011; J. D. Smith, Berkel, Jordan, et al., 2018). Finally, we adapted the student-level acceptability and usability measures from our evaluation of VR-JIT to assess VIT-TAY (M. J. Smith, Smith, et al., 2021) and rescaled them for equivalence. We conducted independent-samples t tests to evaluate between-group differences in implementation outcomes.

We used logistic regression to conduct an intent-to-treat analysis (i.e., students who completed at least one virtual interview were included in the analysis) on whether VIT-TAY engagement explained significant variation in students’ new employment (compared with unemployment) by 6-month follow-up. The model included sex assigned at birth, IQ, and grade level as covariates as they are known contributors to employment among youth (e.g., Southward & Kyzar, 2017) and is the same model that previously evaluated VR-JIT in special education. We evaluated the model for multicollinearity; all variance-inflation factors were below 2.0. Of note, we recoded freshmen and sophomores into the same grade level due to small sample sizes. To compare whether two statistically significant odds ratios (ORs; i.e., VIT-TAY engagement, VR-JIT engagement) differed from each other, we calculated the standard error of the log odds and performed a one-tailed z test. Pearson correlations explored the relationship between VIT-TAY engagement and employment.

Missing Data

Overall IQ data were systematically missing from 53.6% of the sample. To covary for IQ in the regression analyses, we used a multiple imputation method to pool results from a series of five multiple imputations (Graham, 2012). Employment outcomes were not obtained for seven students. An additional 17 students were employed at baseline and sustained their jobs through follow-up. These participants were excluded from the logistic regression.

Results

VIT-TAY Implementation Process Outcomes

Table 3 compares the process data for the VIT-TAY and VR-JIT cohorts. The VIT-TAY and VR-JIT cohorts did not differ with respect to the overall number of completed virtual interviews (p > .10) or the proportion of students who completed at least 15 interviews (p > .10). However, significantly more VIT-TAY students than VR-JIT students transitioned from easy interviews to medium-difficulty interviews (i.e., by completing ≥3 easy interviews, p < .001), whereas there were no differences between programs in the proportion of students transitioning from medium to hard interviews (i.e., by completing ≥3 medium-level interviews, p = .52). However, fewer VIT-TAY students, compared with VR-JIT students, completed ≥3 hard interviews (29.2% vs. 42.3%, p = .001). Overall, VIT-TAY students, compared with VR-JIT students, completed fewer minutes of virtual interviews (p < .001). At easy interviews, VIT-TAY students completed significantly fewer minutes than VR-JIT students (p < .001), but at medium, VIT-TAY students completed significantly more minutes than VR-JIT students (p = .008). Groups did not differ with respect to minutes on hard interviews (p = .78).

Table 3.

Comparison of VIT-TAY and VR-JIT Process Data.

Engagement and performance	VIT-TAY	VR-JIT	Test statistic (χ² or t)	p value
% progressed from easy to medium	88.2%	77.1%	13.96	<.001
% progressed from medium to hard	54.2%	51.6%	0.43	.52
% completed at least 3 hard interviews	29.2%	42.3%	11.77	.001
% completed at least 15 interviews	28.1%	25.4%	0.56	.46
Total virtual interviews completed, M (SD) and range	10.1 (7.9), 1–50	10.7 (7.3), 1 to 37	0.96	.34
Easy interviews
Completed, M (SD) and range	4.7 (3.1), 1–30	4.8 (3.7), 0–28	0.35	.73
Min completed, M (SD) and range	49.5 (31.9), 1–299	91.9 (66.4), 0–395	9.83	<.001
Medium interview
Completed, M (SD) and range	3.1 (2.9), 1–21	2.7 (2.4), 0–14	1.52	.13
Min completed, M (SD) and range	63.5 (79.9), 0–1,020	50.2 (44.7), 0–244	2.65	.008
Hard interviews
Completed, M (SD) and range	2.4 (4.4), 0–38	3.2 (4.2), 0–25	2.30	.02
Min completed, M (SD) and range	51.1 (92.5), 0–542	52.9 (69.4), 0–428	0.27	.78
Min engaged in virtual interviews, M (SD) and range	158.9 (145.6), 1–984	198.7 (130.6), 14–676	3.57	<.001
High score, M (SD) and range	90.1 (10.0), 37–100	90.6 (11.7), 30–100	0.56	.58
Mean score, M (SD) and range	76.2 (11.7), 36.79–97.00	77.5 (14.1), 22.5–100	1.22	.22

Note. VIT-TAY = virtual interview training for transition-age youth; VR-JIT = virtual reality job interview training.

VIT-TAY Scale-Out Implementation Outcomes

VIT-TAY orientation acceptability, appropriateness, and feasibility (pre-implementation)

Staff reported VIT-TAY orientation as acceptable, while reporting that the VIT-TAY tool was appropriate and would be feasible to implement within Pre-ETS. These means did not significantly differ from those reported for VR-JIT (all p > .10; see Table 4).

Table 4.

Pre- and Post-Implementation Outcomes for VIT-TAY and VR-JIT

Implementation variables	VIT-TAY (n = 61), M (SD)	VR-JIT (n = 50), M (SD)	t	p value
Pre-implementation outcomes
Orientation acceptability^a	21.6 (5.4)	21.6 (4.8)	0.06	.95
Appropriateness^a	16.1 (2.8)	16.0 (2.6)	0.06	.96
Expected implementation feasibility^a	23.4 (5.7)	24.6 (4.6)	1.12	.26
Post-implementation outcomes
Intervention acceptability
Teachers and leaders^b	23.9 (4.2)	25.6 (3.9)	1.91	.06
Students^c	13.5 (3.6)	14.2 (3.0)	1.89	.06
Intervention usability
Students^c	17.7 (4.3)	19.5 (3.8)	4.10	<.001
Intervention sustainability
Teachers and leaders^b
Motivated to continue	2.3 (0.7)	2.4 (0.7)	0.20	.87
Equipped to continue	2.3 (0.8)	2.6 (0.6)	1.80	.09
Disruptive to routine	1.1 (1.1)	0.8 (0.9)	1.21	.23

Note. VIT-TAY = virtual interview training for transition-age youth; VR-JIT = virtual reality job interview training.

The 61 VIT-TAY staff members included 50 of the 55 participating leaders and teachers as well as 11 teachers who trained to participate but withdrew after completing their evaluation. Similar to the VR-JIT study, we included these additional pre-implementation data as they were collected in good faith and optimize variation in the measures collected. ^bn = 41 teachers and leaders for VIT-TAY; n = 31 teachers and leaders for VR-JIT. ^cn = 280 students for VIT-TAY; n = 115 students for VR-JIT.

VIT-TAY implementation context and adherence

VIT-TAY implementation primarily took place at school during Pre-ETS class, which is consistent with the prior VR-JIT implementation (M. J. Smith, Smith, et al., 2021). The means presented are between midpoint and endpoint. As a result, item means may not add up to 100%. Reports from 44 (of 47) teachers indicated classroom-level VIT-TAY implementation primarily occurred (a) in a group setting where students individually used devices (60.7%); (b) in private or semi-private space where students individually used devices (34.7%); and (c) in a group setting where students shared devices (6.9%). This pattern was consistent with the VR-JIT implementation (i.e., 72.5%, 22.3%, and 3.7%, respectively; all p > .10). The level of guidance provided by teachers to students using VIT-TAY, compared with VR-JIT, did not differ (all p > .10): no guidance (17.9% vs. 24.6%, respectively); a little guidance (27.1% vs. 25.9%, respectively); some guidance (24.9% vs. 26.3%, respectively); and a lot of guidance (26.2% vs. 21.6%, respectively).

Forty-two (of 47) teachers submitted adherence checklists of which 32 (76.2%) teachers reported high adherence (defined as at least 90% of boxes checked), which did not significantly differ from the 85.2% of VR-JIT teachers who reported high adherence (χ² = 0.8, p = .37; data not shown). Adherence did not significantly differ between the VIT-TAY (M = 89.2, SD = 0.18) and VR-JIT (M = 91.5, SD = 0.17) implementation (t = 0.5, p = .61).

VIT-TAY acceptability and sustainability (post-implementation)

As reported in Table 4, staff reported VIT-TAY was acceptable, which did not significantly differ from staff reports of VR-JIT (p > .05). They also reported that they were motivated and equipped to continue VIT-TAY implementation while also noting that VIT-TAY caused minimal disruption to their ongoing routines. Upon comparison, these sustainability assessments did not significantly differ (p > .08) from those obtained from VR-JIT.

Regarding student-reported acceptability and usability, students reported VIT-TAY as acceptable, which did not significantly differ from students’ reports of VR-JIT (p = .06). At the same time, while students reported that VIT-TAY was usable, the level of usability was significantly lower (p < .001) than students’ reports of VR-JIT.

Administrator and teacher perspectives on sustaining VIT-TAY did not differ from VR-JIT on factors such as cost (61.2% vs. 45.8%; χ² = 1.6, p = .21, respectively); burden to train new teachers (30.6% vs. 41.7%; χ² = 0.9, p = .35); inflexibility to adapt implementation (40.8% vs. 29.2%; χ² = 0.9, p = .33); prioritizing the training (85.7% vs. 75.0%; χ² = 1.3, p = .26); and perceived student satisfaction with the training (79.6% vs. 91.7%; χ² = 1.7, p = .19).

VIT-TAY Effectiveness Outcomes

Teachers reported 48.1% of students using VIT-TAY obtained new jobs (n = 168) between baseline and follow-up, which was significantly higher than the 32.7% job attainment rate of students who used VR-JIT. In addition, 4.9% of VIT-TAY students and 17.8% of VR-JIT students were employed at baseline and sustained their employment through follow-up. Meanwhile, by follow-up, the unemployment rate was 47.0% in the VIT-TAY group and 49.5% in the VR-JIT group (χ² = 33.4, p < .001). When comparing students who remained unemployed, 25.6% of the unemployed VIT-TAY students obtained either a new paid or unpaid internship by follow-up, compared with 17.0% of VR-JIT students (χ² = 3.2, p = .07).

We used logistic regression to assess whether VIT-TAY engagement was associated with employment (Table 5). Students had greater odds of obtaining employment (OR = 1.73, p < .001) for each additional unit of VIT-TAY engagement while covarying for IQ and sex (both nonsignificant covariates, p > .10) and grade level (youth at lower grade levels were less likely to obtain employment compared with youth receiving post-senior Pre-ETS; all p < .05).

Table 5.

VIT-TAY and Revised VR-JIT Logistic Regression Results.

Effectiveness variable	Predictor variables	OR	95% CI		p value
Effectiveness variable	Predictor variables	OR	LL	UL	p value
VIT-TAY
Obtained new employment by 6-month follow-up	VIT-TAY engagement	1.73	1.37	2.18	<.001
	Overall IQ	1.02	0.99	1.04	.09
	Sex^a	1.20	0.79	1.82	.23
	Freshman or sophomore^b	0.40	0.19	0.84	.02
	Junior^b	0.45	0.23	0.87	.02
	Senior^b	0.51	0.32	0.81	.009
Revised VR-JIT
Obtained new employment by six-month follow-up	VR-JIT engagement	1.48	1.09	2.00	.012
	Overall IQ	1.05	1.02	1.08	<.001
	Sex^a	1.32	0.72	2.41	.371
	Freshmen or Sophomore^b	0.07	0.01	0.38	.002
	Junior^b	0.18	0.05	0.67	.011
	Senior^b	0.53	0.24	1.15	.109

Note. VIT-TAY model fit statistics were χ²(6) = 44.4, p < .001; Nagelkerke R² = .167; Revised VR-JIT model fit statistics were χ²(6) = 36.1, p < .001; Nagelkerke R² = 0.200 (χ² and R² values represent the mean across the imputations for IQ). VIT-TAY = virtual interview training for transition-age youth; VR-JIT = virtual reality job interview training; OR = odds ratio; CI = confidence interval; LL = lower limit; UL = upper limit; OR = odds ratio.

Females as reference group. ^bStudents receiving transition services after completing senior coursework or senior year as reference group.

We used a z test to determine whether the ORs differed between VIT-TAY engagement and VR-JIT engagement as predictors of employment. To facilitate such a comparison, we first needed to recompute VR-JIT engagement using the same methods described above to compute VIT-TAY engagement. Notably, the VR-JIT study included minutes engaged with e-Learning when estimating VR-JIT engagement. However, minutes engaged in VIT-TAY e-Learning was not tracked due to a technical problem. Thus, it was not included in the estimate of VIT-TAY engagement. Therefore, we recomputed the VR-JIT engagement variable to remove e-Learning engagement (minutes) from the VR-JIT engagement variable to make both versions of the engagement variable equivalent. Then, we conducted a logistic regression using the revised VR-JIT engagement variable. The revised VR-JIT engagement variable was still associated with greater odds of employment (OR = 1.48; 95% confidence interval [CI] = [1.09, 2.00]; p = .012). Table 5 includes the revised regression results. Notably, the VIT-TAY OR reflecting employment did not differ from the VR-JIT OR (z = 0.35, p = .36).

Overall, 56 (47.9%) of 117 non-job-seeking VIT-TAY students obtained employment by follow-up compared with 18 (22.0%) of 82 non-job-seeking VR-JIT students (χ² = 13.9, p < .001). Notably, three VIT-TAY non-job-seekers and one VR-JIT non-job-seeker had missing employment data. Follow-up logistic regressions revealed VIT-TAY engagement was associated with greater odds of employment for non-job-seeking students (OR = 2.03, 95% CI [1.23, 3.35], p = .006), but VR-JIT did not explain significant variation in employment among non-job-seeking students (OR = 1.38, 95% CI [0.69, 2.78], p = .37). Finally, unemployed and job-seeking students at baseline who used VIT-TAY (n = 160) or VR-JIT (n = 107) did not differ with respect to their employment rate at follow-up (51.2% vs. 45.8%, χ² = 0.8, p = .38, respectively). VIT-TAY process outcomes (e.g., scores, completed interviews, minutes spent interviewing) were significantly correlated with employment compared with VR-JIT process outcomes where this relationship was primarily nonsignificant (see online supplemental materials Table 2).

Discussion

Scaling-out is an important concept in implementation research to more rapidly spread the impact of evidence-based interventions by shifting focus to a new population, a different service delivery context, or both. Given the focus of Pre-ETS on facilitating job readiness and work-based learning for students with disabilities, and the lack of evidence-based practices currently implemented within Pre-ETS for job interview skill development (Rowe et al., 2021), a hybrid effectiveness-implementation design is particularly germane to evaluating scale-out efforts of VIT-TAY in Pre-ETS given that both implementation processes outcomes and effectiveness outcomes are relevant (Aarons et al., 2017).

We used a hybrid effectiveness-implementation design to evaluate the scaling-out of VIT-TAY from youth with autism in Pre-ETS to all students with disabilities in Pre-ETS. We used the same prospective implementation strategies for VIT-TAY as were used in the original evaluation of VR-JIT in Pre-ETS (M. J. Smith, Smith, et al., 2021). Overall, teachers and administrators reported that VIT-TAY implementation was feasible and appropriate; orientation methods to prepare for delivery VIT-TAY were acceptable; VIT-TAY was implemented with high rates of adherence and required minimal adaptation; and teachers reported VIT-TAY was acceptable and sustainable. We now discuss the comparison of these and other implementation process/outcome variables and employment effects between VIT-TAY and VR-JIT.

We recommended that teachers use a flexible strategy to adapt their implementation, when necessary, which is consistent with current perspectives from the field of implementation science and often necessary for sustainability (Chambers et al., 2013). Using a “recommended” implementation protocol, which allows for some adaptation to the local context, is arguably preferred when implementation is being studied for the first time in its intended delivery setting. This was also done to characterize the type and amount of adaptations that were made to the recommended strategy to inform future adopters. Administrative leaders had a prospective plan for teachers to implement VIT-TAY at school during transition class time (86.2%) or study hall (10.2%), which was consistent with VR-JIT implementation (M. J. Smith, Smith, et al., 2021). Also, teacher reports indicated that students primarily used VIT-TAY on individual devices in their class or within semi-private spaces, and the level of guidance teachers provided to facilitate students using VIT-TAY was consistent with prior VR-JIT implementation.

Also similar to the original VR-JIT implementation, teachers using VIT-TAY reported the orientation was acceptable, and our data showed it could be done with full adherence, and that the technology was feasible and appropriate within Pre-ETS. Also, teachers using both interventions reported adherence at rates consistent with those from an RCT in which the research team supervised the delivery to ensure high adherence of teachers using VIT-TAY with students with autism (M. J. Smith, Sherwood, Ross, et al., 2021). Delivery of the adapted intervention using the same evidence-supported implementation strategies (e.g., training, school leadership engagement, during school in transition class) used in the original implementation is important when evaluating scaling-out to isolate the effects of the adaptation and change in population or context. In some situations, the implementation strategies themselves would also require adaptation or changes (e.g., when or where delivery occurs), particularly when scaling out to a new service context. However, the data suggest that the change in population in this study did not warrant a modification to said strategies. This was likely in part due to lessons learned from our prior study with VR-JIT as to what works in Pre-ETS and the similarities between the samples (i.e., autistic students vs. all students in Pre-ETS). This consistency across cohorts increases our confidence in the validity of our observed differences between the original and adapted interventions in this study.

Although teachers and students reported VIT-TAY as acceptable and easy to use, students’ views on usability were significantly lower than those of VR-JIT (M. J. Smith, Smith, et al., 2021). Notably, 49% of students who reported below-median usability were from schools where teachers anecdotally described the students as having complex support needs (e.g., low practical and social skills needed to function independently). Although these results are less concerning given the effectiveness of and high level of student engagement in VIT-TAY, improving the usability of the program for students with complex support needs may improve initial uptake and program completion. Regarding sustainability, teachers were motivated and felt equipped to continue VIT-TAY implementation, similar to teachers using VR-JIT. Also, the rate at which teachers and administrators viewed potential barriers (e.g., cost) and facilitators (e.g., motivation) to sustaining VIT-TAY did not differ from those reported for VR-JIT. Overall, implementation outcomes are by design a multivariate problem such that the interpretation of success involves the examination of these metrics in their totality (Glasgow et al., 2019).

Despite some of the noted differences in VIT-TAY implementation and usage between cohorts, we observed a 48.1% employment rate for students who used VIT-TAY, which was significantly higher than the 32.7% of students who found jobs after using VR-JIT. Both rates were higher than the national employment rate of 18.4% among youth with disabilities (Bureau of Labor Statistics, 2020). Meanwhile, the odds of obtaining employment did not differ between VIT-TAY (OR = 1.73, p < .001) and VR-JIT (OR = 1.48, p = .012) after accounting for covariates. These findings are consistent with a recent RCT suggesting VIT-TAY significantly enhanced competitive employment for transition-age youth with autism (M. J. Smith, Sherwood, Ross, et al., 2021).

Further comparison of the cohorts revealed that students not seeking jobs at baseline who engaged in VIT-TAY had a significantly higher employment rate by follow-up compared with students who were not job-seeking and used VR-JIT (47.9% vs. 22.0%, p < .001). The observed differences held when evaluating each group using logistic regression (OR = 2.03, p = .006 vs. OR = 1.38, p = .37). The adaptations to VIT-TAY, including greater scaffolding and social story-telling (M. J. Smith et al., 2020), could be facilitating greater engagement and motivation to get jobs among students who were not initially interested in employment at baseline—although these hypotheses will need to be tested in the future. In summary, the results suggest VIT-TAY is similar to VR-JIT on implementation processes and may be more effective in helping students obtain employment—arguably the most important outcome for students engaged in Pre-ETS.

Limitations and Implications for Future Research

We must interpret our results while accounting for study limitations. First, this trial used a nonrandomized, single-arm, open design. Hence, future research is needed to evaluate the effectiveness of VIT-TAY in the context of services as a usual control group. Second, our evaluations were primarily conducted at the teacher level rather than on student-level data (e.g., IDEA categories) as effect modifiers. Although our design analysis revealed that student type (based on the four types of Pre-ETS) did not explain significant variation in employment outcomes, future studies can examine this issue more directly. Third, although there was greater diversity among students in this study compared with the VR-JIT evaluation, the results might have limited generalizability to racial and ethnic minority students (e.g., only 26.4% of students were African American). Future research will be needed to evaluate VIT-TAY in settings with greater diversity. Fourth, most partners were city- and suburban-based schools. The results have limited generalizability to town- and rural-based schools, which can be evaluated more deeply in future studies. Finally, participating schools were supported by state-level vocational rehabilitation services. Thus, the results have limited generalizability to schools without state-level support. Such schools could be the focus of future studies. Overall, the study results are promising in the context of the limitations, and VIT-TAY effectiveness need to be confirmed in a future fully powered RCT.

Implications for Research, Practice, and Policy

Job interviewing is one of many pre-employment transition skills identified by the U.S. Department of Education (2017), and 88.8% of employed youth with disabilities receiving state-supported Pre-ETS completed interviews to get their jobs (M. J. Smith, Sherwood, Blajeski, et al., 2021). Furthermore, job interviews are a critical component of obtaining a job and pose a major barrier to employment for individuals with disabilities. However, the evidence for most job interview interventions comes from small-scale or underpowered studies. Thus, the implementation of an evidence-based intervention to enhance job interview skills should be a critical focus of policies to strengthen Pre-ETS going forward. The larger scaled and more rigorously designed trials of VR-JIT and VIT-TAY are promising; however, a fully powered RCT needs to establish the interventions’ effectiveness as evidence-based practice. Nonetheless, the field is looking for more remote learning options in the wake of the coronavirus pandemic. This hybrid effectiveness-implementation scale-out study provides considerable support and a blueprint for a future RCT evaluating the integrated delivery of VIT-TAY within Pre-ETS.

The study results will inform special education administrators and Pre-ETS teachers looking for acceptable and easy-to-implement methods to train job interview skills given that traditional mock interviews are resource-intensive (e.g., Gresham et al., 2001). Specifically, administrators are likely to see VIT-TAY as a convenient alternative to mock interviews, allowing their teaching staff to focus time and attention on other duties. For example, VIT-TAY gives teachers the flexibility to work with multiple students at once and serve as a group facilitator in addition to a 1:1 instructor. Also, VIT-TAY provides teachers and administrators with automatically generated reports to enhance and expedite their data collection and reporting requirements, which is critical for identifying whether individualized education program goals and objectives have been met. Finally, VIT-TAY has the potential to be implemented by vocational rehabilitation counselors and may provide more flexibility in their schedules to work with students on individualized job exploration and work-based learning experiences.

In conclusion, VIT-TAY appears to be feasible, scalable, and sustainable for teachers delivering the tool to students receiving Pre-ETS. In addition, VIT-TAY effectiveness did not significantly differ from that of VR-JIT in general although it appeared stronger for students who were not job seekers. Although students and teachers appeared satisfied with both tools, VIT-TAY emerged as comparatively more difficult to use by students who were anecdotally characterized as having complex support needs. Future research is needed to validate the effectiveness of VIT-TAY among students with disabilities in a fully powered RCT.

Footnotes

Acknowledgements

We would like to acknowledge the administration and staff from the following entities: the Level Up: Employment Skills Simulation Lab within the University of Michigan School of Social Work; Division of Rehabilitation Services, Department of Mental Health, and Secondary Transitional Experience Program within the Illinois Department of Human Services; Chicago Public Schools Office of Diverse Learners Supports and Services; Michigan Rehabilitation Services; Project SEARCH administrative team at Cincinnati Children’s Hospital Medical Center; and Project SEARCH staff at our partnering sites in Illinois, Michigan, and Florida for their help with this project and their continued support in our work together. Also, we would like to acknowledge the administrators, educational staff, and students from our school partners for participating in this project.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The University of Michigan will receive royalties from SIMmersion LLC on the sales of the virtual interview training for transition-age youth tool that was the focus of this study. These royalties will be shared with Dr. Matthew Smith and the University of Michigan School of Social Work. Dr. Smith adhered to the University of Michigan’s Conflict Management Plan that was reviewed and approved by a University of Michigan Conflict of Interest Committee. No other authors report any conflicts of interest.

Funding

This study was supported by grants from the Kessler Foundation (1003-1958-SEG-FY2016, PI: Matthew Smith). Marc S. Atkins was supported by the National Center for Advancing Translational Sciences, National Institutes of Health Grant UL1TR002003. Justin D. Smith was supported by the National Institute on Drug Abuse, National Institutes of Health Grant P30DA027828. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Kessler Foundation and the National Institutes of Health. Elizabeth C. Danielson was supported by funding from the National Institute on Disability, Independent Living, and Rehabilitation Research (#90ARHF0003).

ORCID iDs

Matthew J. Smith

Kari Sherwood

Supplemental Material

Supplemental material is available on the Career Development and Transition for Exceptional Individuals webpage with the online version of the article.

References

Aarons

G. A.

Sklar

Mustanski

Benbow

Brown

C. H.

(2017). “Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implementation Science, 12, 111. http://doi.org/10.1186/s13012-017-0640-6

Barbee

J. R.

Keil

E. C.

(1973). Experimental techniques of job interview training for the disadvantaged: Videotape feedback, behavior modification, and microcounseling. Journal of Applied Psychology, 58(2), 209–213. https://doi.org/10.1037/h0035655

Bureau of Labor Statistics. (2020). Persons with a disability: Labor force characteristics 2019. U.S. Department of Labor. https://www.bls.gov/news.release/pdf/disabl.pdf

Burke

S. L.

Bresnahan

Epnere

Rizzo

Partin

Ahiness

R. M.

Trimmer

(2018). Using virtual interactive training agents (ViTA) with adults with autism and other developmental disabilities. Journal of Autism and Developmental Disorders, 48(3), 905–912. https://doi.org/10.1007/s10803-017-3374-z

Burke

S. L.

Grudzien

Garcia

(2021). Brief report: Improving employment interview self-efficacy among adults with autism and other developmental disabilities using virtual interactive training agents (ViTA). Journal of Autism and Developmental Disorders, 51(2), 741–748. https://doi.org/10.1007/s10803-020-04571-8

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, Article 117. https://doi.org/10.1186/1748-5908-8-117

Curran

G. M.

Bauer

Mittman

Pyne

J. M.

Stetler

(2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217–226. https://doi.org/10.1097/MLR.0b013e3182408812

Genova

H. M.

Lancaster

Morecraft

Haas

Edwards

DiBenedetto

Krch

DeLuca

Smith

M. J.

(2021). A pilot RCT of virtual reality job interview training in transition-age youth on the autism spectrum. Research in Autism Spectrum Disorders, 89, Article 101878. https://doi.org/10.1016/j.rasd.2021.101878

Geverdt

(2015). Education Demographic and Geographic Estimates Program (EDGE): Locale boundaries user’s manual (NCES 2016-012). U.S. Department of Education, National Center for Education Statistics. https://catalog.data.gov/dataset/locales-2018

10.

Glasgow

R. E.

Harden

S. M.

Gaglio

Rabin

Smith

M. L.

Porter

G. C.

Ory

M. G.

Estabrooks

P. A.

(2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, 64. https://doi.org/10.3389/fpubh.2019.00064

11.

Graham

J. W.

(2012). Multiple imputation and analysis with SPSS 17-20. In Graham

J. W.

(Ed.), Missing data: Analysis and design (pp. 227–241). Springer. https://doi.org/10.1007/978-1-4614-4018-5_5

12.

Gregg

Galyardt

Wolfe

Moon

Todd

(2017). Virtual mentoring and persistence in STEM for students with disabilities. Career Development and Transition for Exceptional Individuals, 40(4), 205–214. https://doi.org/10.1177/2165143416651717

13.

Gresham

F. M.

Sugai

Horner

R. H.

(2001). Interpreting outcomes of social skills training for students with high-incidence disabilities. Exceptional Children, 67(3), 331–344. https://doi.org/10.1177/001440290106700303

14.

Harris

P. A.

Taylor

Thielke

Payne

Gonzalez

Conde

J. G.

(2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010

15.

Hayes

G. R.

Custodio

V. E.

Haimson

O. L.

Nguyen

Ringland

K. E.

Ulgado

R. R.

Waterhouse

Weiner

(2015). Mobile video modeling for employment interviews for individuals with autism. Journal of Vocational Rehabilitation, 43(3), 275–287. https://doi.org/10.3233/JVR-150775

16.

Individuals with Disabilities Education Improvement Act, Pub. L. No. PL 101-476, 20 U.S.C. § 1400 (2004).

17.

Kellems

R. O.

Cacciatore

Osborne

(2019). Using an augmented reality–based teaching strategy to teach mathematics to secondary students with disabilities. Career Development and Transition for Exceptional Individuals, 42(4), 253–258. https://doi.org/10.1177/2165143418822800

18.

Michigan Administrative Rules for Special Education. (2021). Michigan administrative code (R 340.1702). https://www.michigan.gov/mde/0,4615,7-140-6598---,00.html

19.

Muthen

B. O.

Satorra

(1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. https://doi.org/10.2307/271070

20.

Parsons

L. D.

(2006). Using video to teach social skills to secondary students with autism. Teaching Exceptional Children, 39(2), 32–38. https://doi.org/10.1177/004005990603900206

21.

Politis

Robb

Yakkundi

Dillenburger

Herbertson

Charlesworth

Goodman

(2017). People with disabilities leading the design of serious games and virtual worlds. International Journal of Serious Games, 4(2), 63–73. https://doi.org/10.17083/ijsg.v4i2.160

22.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

23.

Qualtrics. (2005). Qualtrics. [Computer software]. https://www.qualtrics.com

24.

Rosales

Whitlow

(2019). A component analysis of job interview training for young adults with autism spectrum disorder. Behavioral Interventions, 34(2), 147–162. https://doi.org/10.1002/bin.1658

25.

Rowe

D. A.

Mazzotti

V. L.

Fowler

C. H.

Test

D. W.

Mitchell

V. J.

Clark

K. A.

Holzberg

Owens

T. L.

Rusher

Seaman-Tullis

R. L.

Gushanas

C. M.

Castle

Chang

W.-H.

Voggt

Kwiatek

Dean

(2021). Updating the secondary transition research base: Evidence- and research-based practices in functional skills. Career Development and Transition for Exceptional Individuals, 44(1), 28–46. https://doi.org/10.1177/2165143420958

26.

Schrank

F. A.

McGrew

K. S.

Mather

(2014). Woodcock-Johnson IV. Riverside.

27.

Sherwood

Smith

M. J.

Ross

Johnson

Harrington

Blajeski

Dawalt

Bishop

Smith

J. D.

(In Press). Mixed-methods implementation evaluation of virtual interview training for transition-age autistic youth in pre-employment transition services. Journal of Vocational Rehabilitation.

28.

Smith

J. D.

Berkel

Jordan

Atkins

D. C.

Narayanan

S. S.

Gallo

Grimm

K. J.

Dishion

T. J.

Mauricio

A. M.

Rudo-Stern

Meachum

M. K.

Winslow

Bruening

M. M.

(2018). An individually tailored family-centered intervention for pediatric obesity in primary care: Study protocol of a randomized type II hybrid effectiveness–implementation trial (Raising Healthy Children study). Implementation Science, 13, Article 11. https://doi.org/10.1186/s13012-017-0697-2

29.

Smith

J. D.

Berkel

Rudo-Stern

Montaño

St George

S. M.

Prado

(2018). The Family Check-Up 4 Health (FCU4Health): Applying implementation science frameworks to the process of adapting an evidence-based parenting program for prevention of pediatric obesity and excess weight gain in primary care. Frontiers in Public Health, 6, Article 293. https://doi.org/10.3389/fpubh.2018.00293

30.

Smith

M. J.

Fleming

M. F.

Wright

M. A.

Jordan

Humm

L. B.

Olsen

Bell

M. D.

(2015). Job offers to individuals with severe mental illness after participation in virtual reality job interview training. Psychiatric Services, 66(11), 1173–1179. https://doi.org/10.1176/appi.ps.201400504

31.

Smith

M. J.

Ginger

E. J.

Wright

M. A.

Humm

L. B.

Olsen

Bell

M. B.

Fleming

M. F.

(2014). Virtual reality job interview training in individuals with psychiatric disabilities. Journal of Nervous and Mental Disease, 202(9), 659–667. https://doi.org/10.1097/NMD.0000000000000187

32.

Smith

M. J.

Ginger

E. J.

Wright

M. A.

Taylor

J. L.

Humm

L. B.

Olsen

D. E.

Bell

M. D.

Fleming

M. F.

(2014). Virtual reality job interview training in adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 44(10), 2450–2463. https://doi.org/10.1007/s10803-014-2113-y

33.

Smith

M. J.

Pinto

R. M.

Dawalt

Smith

J. D.

Sherwood

Miles

Taylor

J. L.

Hume

Dawkins

Baker-Ericzen

Frazier

Humm

Steacy

(2020). Using community-engaged methods to adapt virtual reality job-interview training for transition-age youth on the autism spectrum. Research in Autism Spectrum Disorders, 71, Article 101498. https://doi.org/10.1016/j.rasd.2019.101498

34.

Smith

M. J.

Sherwood

Blajeski

Ross

Smith

J. D.

Jordan

Dawalt

Bishop

Atkins

M. S.

(2021). Job interview and vocational outcomes among transition-age youth receiving special education pre-employment transition services. Intellectual and Developmental Disabilities, 59(5), 405–421. https://doi.org/10.1352/1934-9556-59.5.405

35.

Smith

M. J.

Sherwood

Ross

Smith

J. D.

Dawalt

Bishop

Humm

Elkins

Steacy

(2021). Virtual interview training for transition-age autistic youth: A randomized controlled feasibility and effectiveness trial. Autism, 25(6), 1536–1552. https://doi.org/10.1177/1362361321989928

36.

Smith

M. J.

Smith

J. D.

Jordan

Sherwood

McRobert

Ross

Oulvey

Atkins

M. S.

(2021). Virtual reality job interview training in transition services: Results of a single-arm, noncontrolled effectiveness-implementation hybrid trial. Journal of Special Education Technology, 36(1), 3–17. https://doi.org/10.1177%2F0162643420960093

37.

Southward

J. D.

Kyzar

(2017). Predictors of competitive employment for students with intellectual and/or developmental disabilities. Education and Training in Developmental Disabilities, 52(1), 26–37. https://doi.org/10.2307/26420373

38.

Spencer

Drescher

Sears

Scruggs

A. F.

Schreffler

(2019). Comparing the efficacy of virtual simulation to traditional classroom role-play. Journal of Educational Computing Research, 57(7), 1772–1785. https://doi.org/10.1177/0735633119855613

39.

Stirman

S. W.

Gamarra

Bartlett

Calloway

Gutner

(2017). Empirical examinations of modifications and adaptations to evidence-based psychotherapies: Methodologies, impact, and future directions. Clinical Psychology, 24(4), 396–420. https://doi.org/10.1111/cpsp.12218

40.

Thorsteinsson

Shavinina

L. V.

(2013). Developing an understanding of the pedagogy of using a virtual reality learning environment (VRLE) to support innovation education. Routledge.

41.

U.S. Department of Education. (2017). A transition guide to postsecondary education and employment for students and youth with disabilities. Office of Special Education and Rehabilitation Services. https://sites.ed.gov/idea/files/postsecondary-transition-guide-may-2017.pdf

42.

Wechsler

(2014). Wechsler intelligence scale for children (5th ed.). Pearson.

43.

Wehman

Schall

McDonough

Sima

Brooke

Ham

Whittenburg

Brooke

Avellone

Riehle

(2019). Competitive employment for transition-aged youth with significant impact from autism: A multi-site randomized clinical trial. Journal of Autism and Developmental Disorders, 50(6), 1882–1897. https://doi.org/10.1007/s10803-019-03940-2

44.

Workforce Innovation and Opportunity Act, Pub. L. No. PL 113-128, 29 U.S.C. § 3164 (2014).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.02 MB

0.01 MB