Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities

Abstract

Virtual Interview Training for Transition-Age Youth and Virtual Reality Job Interview Training are job interview simulators with demonstrated effectiveness in randomized controlled trials. We evaluated their dose responses via secondary data analysis of 558 transition-age youth with disabilities in 47 schools where the simulators were implemented in quasi-experimental studies. Cut-point analyses determined dosing efficiency and efficacy to optimize competitive employment. The most efficient dose when accounting for the balance between dose and employment was completing nine virtual interviews. The most efficacious dose to maximize the likelihood of successful employment was 38, but it varied across race, IQ, Individuals with Disabilities Education Improvement Act (IDEA) categories, and employment history. This study provides a novel approach to inform implementation guidelines for virtual interview training in pre-employment transition services. Limitations and implications for research and practice are discussed.

Keywords

virtual reality job interview training employment transition services dosing efficiency and efficacy

The employment rate of approximately 1.8 million transition-age youth with disabilities is 19.1% for youth 16 to 19 years old and 39.0% for youth 20 to 24 years old. Meanwhile, the rates of employment for their peers with no disabilities are 32.6% and 65.7%, respectively (Bureau of Labor Statistics, 2021). Thus, unemployment has become an expected outcome for most of these youth for at least the first few years after graduation or completing a high school certificate. To try and prevent these disparities in employment, the Workforce Innovation and Opportunity Act (WIOA, 2014) federally mandated pre-employment transition services (Pre-ETS) for transition-age youth in special education prior to and shortly after exiting high school. Specifically, Pre-ETS targets five areas of service that include workplace readiness training (e.g., job seeking skills), work-based learning experiences, postsecondary education counseling, job exploration counseling, and self-advocacy. In particular, the Office of Special Education and Rehabilitative Services within the United States Department of Education indicated job interview skills are an important aspect of workplace readiness training (United States Department of Education, 2017). This indication is consistent with a recent study of 656 transition-age youth with disabilities engaged in Pre-ETS which found that 88.8% of the employed youth completed a job interview prior to getting their job (Smith, Sherwood, Blajeski, et al., 2021).

Thus, it is critical for Pre-ETS to include an evidence-based practice to facilitate job interview skill development. However, the two Department of Education repositories of evidence-based Pre-ETS practices do not include any job interview interventions within the top “evidence-based” tier (Cobb et al., 2013; Rowe et al., 2021). That said, only one program is listed that uses video modeling to remediate interview skills. However, this program has only been tested in a single pilot study with 15 participants and is only noted as having “promising” evidence (Hayes et al., 2015). Thus, there is a critical need to identify a rigorously evaluated job interview intervention to disseminate within Pre-ETS.

One way to potentially address this gap in services involves the use of technology-based job interview simulators that have demonstrated efficacy and effectiveness at improving interview skills and access to employment. Specifically, Virtual Reality Job Interview Training (VR-JIT; a computerized job interview simulator designed for adults with mental health concerns) improved job interview skills and increased employment rates for adults with mental health concerns in a series of randomized controlled trials (e.g., Smith, Fleming, Wright, Roberts, et al., 2015; Smith, Ginger, Wright, Wright, Humm, et al., 2014). VR-JIT facilitated stronger interview skills by enabling trainees to engage in repetitive practice with four levels of automated feedback to practice interviewing for eight different jobs at a fictional big box store. Trainees speak their response (via speech recognition) from a choice of 10 to 15 scripted responses that range from effective to ineffective. Trainees receive real-time nonverbal feedback on their response choices, process feedback on each statement they make, a performance assessment on eight job interview skills, and a numerical score. The virtual hiring manager has various moods and personalities that combine for over 15 hr of unique job interview practice.

Notably, the field has begun to evaluate the efficacy and initial effectiveness of VR-JIT among transition-age youth with disabilities. Specifically, one pilot study found that VR-JIT was associated with improved skills and interview self-efficacy primarily among transition-age youth with autism in a lab-based study (Smith, Ginger, Wright, Wright, Taylor, et al., 2014); results which were then independently replicated in community settings (Arter et al., 2018; Genova et al., 2021; Ward & Esposito, 2019). However, only the original pilot evaluated whether VR-JIT engagement was associated with vocational outcomes, which revealed that VR-JIT trainees, as compared with a control group, were more likely to get a job or volunteer position within 6 months (Smith, Fleming, Wright, Losh, et al., 2015). Meanwhile, a nonrandomized trial evaluated whether VR-JIT engagement (when delivered in 15 schools) was associated with employment outcomes among 279 transition-age youth with disabilities. The findings revealed that greater engagement with VR-JIT (using a composite score reflecting number of virtual interviews completed, minutes engaged with virtual hiring manager, and mean score of virtual interviews) was associated with greater odds (OR = 1.63, p < . 01) of competitive employment by 6-month follow-up (Smith, Smith, Jordan, et al., 2021). Concurrently, a second line of research adapted VR-JIT to specifically meet the needs of transition-age youth with autism and their peers with other disabilities (now called Virtual Interview Training for Transition Age Youth [VIT-TAY]; Smith et al., 2020). A few notable adaptations that differentiate VIT-TAY from the VR-JIT designed for adults focused on (a) increased accessibility (e.g., reduced reading level from sixth to fourth grade; reducing scripted responses from 10 to 15 on the screen to six to eight responses [to reduce cognitive load]); (b) scaffolding 10 interview skills across interview difficulty level (i.e., four skills on easy, seven skills on medium, and 10 skills on hard); and (c) increased diversity. Specifically, VR-JIT includes one White, female interviewer and VIT-TAY includes one Latinx, female interview; one African American male interviewer, and one White, female, immersive job coach that facilitates a social story throughout the training.

The initial randomized controlled trial evaluating VIT-TAY among 71 autistic transition-age youth engaged in Pre-ETS in five schools revealed the Pre-ETS + VIT-TAY group had greater odds of competitive employment (OR = 12.4, p < .01), significant improvements in job interview skills, and significant reductions in job interview anxiety as compared with a Pre-ETS only control group (Smith, Sherwood, Ross, et al., 2021). These findings led to an evaluation of scaling out VIT-TAY to 356 transition-age youth with disabilities (broadly across IDEA categories) in a nonrandomized trial (by the same team using the same implementation and evaluation procedures as in the study of 279 youth with disabilities using VR-JIT). This study revealed greater engagement in VIT-TAY (using the same type of composite measure in the aforementioned VR-JIT study) was associated with greater odds (OR = 1.73, p < .001) of competitive employment by 6-month follow-up (Smith, Sherwood, et al., 2022). Also, comparatively VIT-TAY, and not VR-JIT, was effective at enhancing employment outcomes among transition-age youth with disabilities who were not “job-seekers” at the beginning of the two studies (Smith, Sherwood, et al., 2022). Moreover, we compared the implementation strategies used to deliver VR-JIT in 15 schools and VIT-TAY in 32 schools and found that these strategies did not significantly differ from one another (Smith, Sherwood, et al., 2022). However, the aforementioned studies of VIT-TAY and VR-JIT found a high degree of variation in the number of completed virtual interviews and the relationship of this variation with employment outcomes is unknown.

Thus, in this study, we sought to increase our understanding of the dosing efficiency and efficacy of VR-JIT and VIT-TAY among transition-age youth with disabilities. Specifically, dosing efficiency reflects the optimum balance between the number of completed virtual interviews and obtaining a job, while dosing efficacy reflects the highest likelihood of obtaining a job regardless of the number of completed virtual interviews. Our primary analysis evaluated the dosing efficacies and efficiencies of the VIT-TAY and VR-JIT samples in a combined analysis, and then report the dosing efficacies and efficiencies individually for each intervention. We chose this approach given that VIT-TAY is an adaptation of VR-JIT and still maintains the same core functions of VR-JIT (e.g., reviewing standard job interview skills, completing an online job application, repeatedly practicing interviews with a virtual hiring manager [who has a matrix of the same nine personalities], receiving the same type of feedback [i.e., automated real-time feedback, transcript-level feedback on specific responses to interview questions, summary feedback on interview skills and a numerical score]). In addition, both interventions were associated with similar increases in employment outcomes and utilized similar teacher-level implementation strategies. Our secondary analyses estimated the independent dosing efficacies and efficiencies of VIT-TAY and VR-JIT.

Research Question 1 (RQ1): What is the dosing efficacy and efficiency of VIT-TAY and VR-JIT in a combined sample?

Research Question 2 (RQ2): What is the dosing efficacy and efficiency of VIT-TAY?

Research Question 3 (RQ3): What is the dosing efficacy and efficiency of VR-JIT?

Method

Study Design

Secondary analyses were conducted in the de-identified data from two nonrandomized hybrid type 3 effectiveness-implementation evaluations of (a) VR-JIT in 15 schools for 279 students from August 2017 through March 2018 (Smith, Smith, et al., 2021) and (b) VIT-TAY in 32 schools for 356 students from August 2018 through March 2019 (Smith, Sherwood, et al., 2022). Both studies were reviewed by University of Michigan’s Institutional Review Board and designated as exempt human subjects research.

Recruitment

School partners were recruited in Illinois, Michigan, and Florida. The Illinois Division of Rehabilitation Services (DRS) connected the study team with the Illinois Secondary Transitional Experience Program (STEP; a network of approximately 700 schools providing Pre-ETS with funding and support from DRS) and Chicago Public Schools’ Office of Diverse Learner Supports and Services (Chicago, IL). Michigan Rehabilitation Services helped the study team network with Michigan Career and Technical Institute, a post-secondary Pre-ETS program that annual serves more than 1,000 transition-age youth with disabilities. Finally, the Project SEARCH administration at Cincinnati Children’s Hospital Medical Center connected the study team with Project SEARCH sites in Michigan and in Florida. Additional recruitment details and descriptions of Pre-ETS within these networks can be found here (Smith, Sherwood, et al., 2022; Smith, Smith, et al., 2021).

Participants

Across the two studies of VR-JIT and VIT-TAY, teachers implemented the interviewing tools for their students (ages 16 to 21 years for VR-JIT and 15 to 26 years for VIT-TAY). Additional inclusion criteria were (a) engaged in Pre-ETS and (b) being designated with at least one of the 13 disability categories according to the Individuals with Disabilities Education Improvement Act (Individuals with Disabilities Education Improvement Act IDEA, 2004). Notably, students ages 15 through ages 21 were served by IDEA and state-level vocational rehabilitation, and students 22 through 26 were served by state vocational rehabilitation.

Data were also provided by school-level administrators and teachers using administrative records that were reported to state vocational rehabilitation agencies during the VR-JIT and VIT-TAY studies. Although 279 students used VR-JIT and 356 students used VIT-TAY (n = 635 students in total), the current secondary analysis will only analyze data for those students for whom employment data was available and who were unemployed or became unemployed during the course of the 2-month implementation and 6-month follow-up (N = 558 total; n = 226 for VR-JIT and n = 332 for VIT-TAY). Thus, students who were employed at baseline and sustained their employment through follow-up (n = 49 and n = 17, respectively) and students with missing employment outcome data (n = 4 and n = 7, respectively) were excluded.

Study Measures

Teachers or school administrators used administrative data to complete electronic surveys to report the below de-identified student-level data to the research team. Specifically, the research team sent each school partner a list of personal identification numbers (PINs) for their students. Then, teachers internally referenced these PINs to enter the baseline and 3- and 6-month follow-up data.

The research team collected study measures at baseline (i.e., prior to implementing VR-JIT or VIT-TAY) from teachers (via surveys) for each participating student. Demographic variables included age, sex, and race. Educational variables included IQ (via the Wechsler Intelligence Scale for Children or the Woodcock-Johnson IV [Schrank et al., 2014; Wechsler, 2014]), grade level (1 = Freshman or sophomore, 2 = junior, 3 = senior, 4 = transition year), reading level (1 = third grade or lower, 2 = fourth grade, 3 = fifth grade, 4 = sixth grade, and 5 = higher than sixth grade; which we recoded into 0 = fifth grade or lower and 1 = sixth grade or higher), and category of disability as defined by the Individuals with Disabilities Education Improvement Act (IDEA, 2004; 1 = specific learning disability, 2 = other health impairment, 3 = autism, 4 = intellectual disability, 5 = emotional disturbance, and 6 = speech and language impairment).

Employment history variables included whether the student was employed at the baseline visit (0 = no, 1 = yes), whether student completed a job interview during the 3 months prior to baseline (0 = no, 1 = yes), and whether the student has ever been employed (0 = no, 1 = yes). Employment at 6-month follow-up was coded as “0” for youth who either remained unemployed between baseline and 6-month follow-up or were employed at baseline and became unemployed for the duration of the study. Employment was coded “1” for youth who were either unemployed or employed at baseline (and became unemployed) who then obtained new employment between baseline and 6-month follow-up. The above employment reflected competitive, integrated employment that was not set aside for someone with a disability.

Notably, the VR-JIT and VIT-TAY interventions automatically tracked the number of virtual interviews that were completed. We used this number of completed virtual interviews to assess dosing efficiency and efficacy. The research team used the student PINs to link the total number of completed interviews with the student-level administrative data. The National Center for Education Statistics Search for Public Schools database determined the locale subtype (city, suburban, town, or rural) for each school using 2016–2018 school-year data (Geverdt, 2015).

Virtual Interviewing Interventions

Both VR-JIT and VIT-TAY are computerized job interview simulators with virtual hiring managers that are delivered over the internet via a website (but can also be downloaded to function independently of the internet). Both simulators are commercially licensed by SIMmersion LLC (www.simmersion.com) and consist of (a) a didactic eLearning review of interview skills and interviewing tips; (b) a job application that informs the simulated interview; and (c) a virtual interview where trainees listen to job interview questions, review suggested responses (ranging from very ineffective to very effective), and speak chosen responses to the virtual hiring managers via speech recognition software. Both simulators provide four levels of feedback: (a) real-time nonverbal cues from an automated coach, (b) a color-coded transcript with statement-level feedback, (c) a numerical score (0–100), and (d) a qualitative performance assessment of the targeted interview skills. Both simulators have virtual hiring managers with three difficulty levels (i.e., easy [friendly interviewer], medium [professional interviewer], and hard [inappropriate interviewer]).

As summarized in the introduction, the simulators have notable differences. VR-JIT uses a sixth-grade reading level (compared with a fourth-grade reading level for VIT-TAY). VR-JIT targets eight job interview skills (confidence, positivity, professionalism, interest in the position, honesty, dependability, working well with others, and asking for accommodations) during easy, medium, and hard interviews. Meanwhile, VIT-TAY targets 10 job interview skills that are scaffolded across easy (four skills: confidence, positivity, professionalism, interest in position), medium, (seven skills [three new]: honesty, dependability, working well with others), and hard (10 skills [three new]: sharing strengths and skills, sharing past experiences, sharing limitations) interviews. VR-JIT facilitates interviews for eight jobs (cashier, inventory, food service, maintenance, stock clerk, janitor, customer service, and security), while VIT-TAY has 14 jobs (six new: greeter, website technical support, web development, data entry, automotive, child care worker). VIT-TAY was also adapted to include new components such as social story-telling (e.g., Kendra, the job coach provides video-training on the 10 job interview skills and reviews the performance assessment after the virtual interview is completed) and a token economy system (e.g., earn tokens via your virtual interview score to cash-in toward receiving additional advice from Kendra or asking her questions). See Smith et al. (2020) for additional details.

VR-JIT and VIT-TAY Implementation Strategies

Teachers were trained by the research team to lead students through a 45-min orientation on how to navigate the VR-JIT or VIT-TAY user interface and an administrative portal to monitor their students’ performances with VIT-TAY and VR-JIT. The implementations of VR-JIT and VIT-TAY by teachers were conducted with strong adherence (monitored via adherence checklist; Smith, Sherwood, et al., 2022; Smith, Smith, et al., 2021). Subsequently, teachers facilitated approximately two sessions for students to review the job interview skills highlighted in the eLearning content and complete the job application. Then teachers supervised their students actively practicing interviews with the virtual hiring managers where they answered questions, provided technical guidance (if needed), and reviewed transcripts and performance summary feedback with their students. Teachers also monitored student performance via the administrative portal and provided additional feedback to students.

Based on six RCTs evaluating VR-JIT efficacy and VIT-TAY effectiveness where participants completed an average of 15 virtual interviews (e.g., Smith, Fleming, Wright, Jordan, et al., 2015; Smith, Ginger, Wright, Wright, Taylor, et al., 2014; Smith, Sherwood, Ross, et al., 2021), the intervention protocols for the two VIT-TAY and VR-JIT studies recommended that students complete a total of 15 interviews reflecting a maximum of five interview attempts at the easy (or medium) level before transitioning to medium-level (or hard-level) interviews. Students completed approximately three 45- to 60-min sessions per week over 4 to 6 weeks to try and achieve the recommended 15 interviews. Teachers were allowed to adapt the implementation plan to fit within their everyday teaching and the needs of their students to optimize adherence.

Specifically, it was recommended that students complete at least three interviews on easy and score at least 90 on one of those interviews before progressing to medium. Then at medium, the same approach applied, complete at least three interviews and score at least 90 on one of them before progressing to hard. Complete remaining virtual interviews (toward the recommendation to complete 15 interviews) on the hard level. Teachers facilitated the progression between difficulty levels by monitoring student scoring and informing them when to move from easy to medium and from medium to hard. Students were not incentivized to complete the virtual interviews. Of note, virtual interview scores were based on points awarded for performance (i.e., selected interview responses) that map onto the interview skills presented in the eLearning content).

Data Analysis Plan

To evaluate between-group student-level descriptive characteristics, we used Stata (StataCorp, 2021) modules for performing cluster-adjusted chi-square (with Cramer’s V effect sizes) and t tests (with Cohen’s d effect sizes) using a school-level clustering variable (Herrin, 2002). Since schools are clustered within geographic locale, the locale variable was reported as a fixed effect without adjustment. Notably, prior studies of these data conducted design effects analyses to account for clustering of students at the school and program-type levels, but were not found to be significant (both p > .10; Smith, Sherwood, et al., 2022; Smith, Smith, et al., 2021).

Descriptively, we visually inspected a histogram (see Supplemental Figure 1) of all VR-JIT and VIT-TAY virtual interviews completed and used prior dose completion data from VR-JIT efficacy studies (i.e., a mean of 15 completed virtual interviews) to characterize that 29.9% of the current combined sample completed a low dose of one to five virtual interviews; 40.9% of the sample completed a medium dose of six to 14 virtual interviews; and 29.2% of the sample completed a large dose of 15 or more virtual interviews. Across the total sample, the median number of interviews was nine and the mode was shared between four and 15 completed interviews.

We used a receiver operating characteristic (ROC) curve to generate cut-points, via R (R Core Team, 2013), that reflect dosing efficiency and efficacy (Supplemental Figure 2). The ROC curve as used in this study illustrates the accuracy with which an independent variable is able to predict a dichotomous outcome in terms of both specificity (i.e., the ability to correctly predict those who will not achieve competitive employment due to the intervention [VIT-TAY or VR-JIT], also known as “true negatives”) and sensitivity (i.e., the ability to correctly predict who will achieve competitive employment due to the intervention, also known as “true positives”). Each point on the ROC curve represents a sensitivity/specificity combination (or, more accurately, a combination of sensitivity and 1—specificity) corresponding to a particular value of the predictor (McNeil & Hanley, 1984; Zweig & Campbell, 1993). The ROC curve can be used to identify the cut-point for the predictor at which both specificity and sensitivity are maximized, which corresponds to the point on the curve closest to the upper left-hand corner of the graph. That is, one uses the cut-point that corresponds to the location of the peak of the ROC graph to determine the cut-point that maximizes both specificity and sensitivity when predicting a dichotomous outcome such as competitive employment. In this context, that cut-point is where we have the greatest chance of success with the least investment in terms of the number of sessions (i.e., maximizing efficiency). We also examine the curve for the point where we can guarantee that the intervention will be successful for everyone in the sample regardless of the number of sessions (i.e., maximizing efficacy). This work extends an existing literature from psychiatry, psychology, cancer, and chemistry that used ROC curves to optimize efficiency in clinical contexts (Fombonne, 1991; Hill et al., 2004; Karve et al., 2009; Zlobec et al., 2007).

Thus, in our analysis, our independent variable was the number of virtual interviews completed by each participant, and the dependent variable was a dichotomous indicator of the success of the intervention (i.e., whether or not the participant obtained competitive employment). Thus, the point on the ROC curve that maximizes sensitivity and specificity also reflects the point at which the likelihood of a successful outcome (i.e., competitive employment) is maximized for the fewest virtual interviews completed (i.e., efficiency). An example ROC curve using our data, with this point plotted on the curve, is presented in Supplemental Figure 2. The point on the curve that maximizes the likelihood of a successful outcome (i.e., competitive employment) regardless of how many virtual interviews are completed (i.e., efficacy) is the point at which the curve reaches a sensitivity of 1.0 (i.e., all successful outcomes are correctly predicted). The area under the curve (AUC) for our total sample is .598. This value indicates that the predictor (i.e., number of completed virtual interviews) provides some value in predicting the outcome of competitive employment (if the predictor provided no value, the area under the curve would equal .50 or less). By comparison, the medical diagnostic guidelines suggest AUCs at >.90 provide “outstanding discrimination” between predictor and outcome, .899–.800 provide “excellent discrimination,” .799–700 provide “acceptable discrimination,” and .699–.501 provide “poor discrimination” (Hosmer et al., 2013).

Results

Participant Characteristics

Table 1 displays the student characteristics. We observed that the two cohorts did not differ with respect to age, sex, race, reading level, and IQ (all p > .10). Although between-group differences for grade level were not significant, this difference was characterized by a large effect size (V = .61) with a greater proportion of transition year students in the VIT-TAY cohort. The cohorts also did not differ with respect to their employment history (i.e., current employment at baseline, whether they interviewed to get a job within the past 3 months, and whether they were ever employed). Meanwhile, they differed with respect to geographic locale where the VIT-TAY study had significantly greater representation of City schools than the VR-JIT study (p < .05; V = .45). Disability categories differed between groups for speech and language impairment (p < .001), but was characterized by a small effect size (V = .22).

Table 1.

Student Characteristics.

Variables	Total sample(N = 558)	VIT-TAY(n = 332)	VR-JIT(n = 226)	Intra-cluster correlation	Test statistic(t or χ²)	Effect sized or V	p
Age (M, SD)	18.6 (1.9)	18.7 (2.2)	18.5 (1.5)	.65	.17	.27	.868
Sex (% male)	63.6	63.9	63.3	.03	.01	.01	.914
Race (%)
White	58.1	52.4	66.4	.16	6.46	.28	.596
Black or African American	20.6	26.5	11.9
Latinx	13.4	11.7	15.9
More than one race	3.4	3.9	2.7
Asian American	2.5	3.6	.9
American Indian	.7	.0	1.8
Unknown	.7	.9	0.4
Arab American	.4	.6	.0
Native Hawaiian or Pacific Islander	.2	.3	.0
IQ^a	75.6 (15.1)	74.6 (14.4)	76.7 (15.7)	.09	–.70	.14	.494
Grade (%)^b
Freshman or sophomore	8.3	8.8	7.5	.69	4.68	.61	.197
Junior	10.4	10.9	9.7
Senior	44.2	29.0	66.4
Transition year	37.1	51.4	16.4
Reading level (%)^c
Fifth grade or lower	48.2	47.7	48.8	.30	.01	.01	.930
Sixth grade or higher	51.8	52.3	51.2
Disability categories (%)
Specific learning disability	35.1	35.8	34.1	.25	.03	.02	.873
Other health impairment	24.4	20.5	30.1	.05	3.00	.11	.083
Autism	24.2	22.6	26.5	.11	.31	.05	.581
Intellectual disability	16.1	16.3	15.9	.12	.00	<.01	.957
Emotional disturbance	15.9	13.9	19.0	.12	.68	.07	.411
Speech and language impairment	3.2	0.0	8.0	.05	12.60	.22	<.001
Employment history (%)
Current employment^d	10.6	8.5	13.7	.04	1.91	.08	.167
Interviewed to get job (past 3 months)^e	21.6	19.0	25.9	.33	.37	.08	.542
Ever employed^f	33.9	30.7	38.7	.17	.73	.08	.393
Geographic locale (%)
City	27.4	44.0	3.1	n/a	113.89	.45	<.001
Suburban	53.8	42.8	69.9
Town	16.1	11.4	23.0
Rural	2.7	1.8	2.7

Note. Disability category distributions do not sum to 100% as students could be assigned to more than one category. VIT-TAY = Virtual Interview Training for Transition Age Youth; VR-JIT = Virtual Reality Job Interview Training.

IQ data were missing for n = 187 VIT-TAY students and n = 91 VR-JIT students. ^bGrade-level data were missing for n = 1 VIT-TAY student. ^cReading-level data were missing for n = 1 VIT-TAY student and n = 9 VR-JIT students. ^dCurrent employment data were missing for n = 2 VIT-TAY students. ^eInterviewed to get job data were missing for n = 5 VIT-TAY student and n = 29 VR-JIT students. ^fEver employed data were missing for n = 2 VIT-TAY student and n = 2 VR-JIT students.

Across the three dosage groups of low (1–5 virtual interviews completed), medium (6–14 virtual interviews completed), and high (15 or more virtual interviews completed; we observed that across all n = 558 participants, the low dose group obtained 34.7% employment, the medium dose group obtained 43.9% employment, and the high dose group obtained 61.3% employment by 6-month follow-up (χ² = 24.4, p < .001). Although, this stepwise dose response was replicable in the VIT-TAY subsample (n = 332; χ² = 24.17, p < .001) at 36.9% employment (low dose), 47.7% employment (medium dose), and 71.0% employment (high dose); it was not replicable in the VR-JIT subsample (n = 226, χ² = 4.4, p = .113) at 30.4% employment (low dose), 39.0% employment (medium dose), and 48.6% employment (high dose).

Cut-Point Analyses

Virtual Interviewing Efficiency

With respect to our ROC curve analyses, we present the cut-points for the total sample in Table 2. We observed that across the total sample, the most efficient dose when accounting for the balance between cost (dose) and success rate (employment) was to complete nine virtual interviews. There were no notable deviations in efficiency from the whole sample to the low/medium/high subgroups as their means were all within two virtual interview completions of the overall efficient dose of nine virtual interviews.

Table 2.

Cut-Points by Total Sample (Number of Virtual Interviews Completed).

Group	n	Efficiency	Efficacy	AUC
Total sample	558	9	38	.598
Race^a
White	324	10	35	.595
African American or Black	115	7	38	.613
Latinx	75	7	27	.552
Cognitive characteristics
IQ > 75	140	9	29	.612
IQ ≤ 75	140	11	38	.543
Sixth grade or higher reading level	288	9	36	.632
Fifth grade and lower reading level	270	8	38	.573
IDEA categories^b
Specific learning disability	196	9	38	.598
Autism	135	9	29	.593
Other health impairment	136	9	31	.564
Emotional disturbance	89	9	35	.647
Intellectual disability	90	10	38	.555
Speech and language disability	18	8	19	.591
Employment history
Had a job at some point in life	188	9	32	.541
Never worked in life	366	9	38	.643
Employment status
Currently working	59	7	21	.698
Not currently working	497	9	38	.603
Recent job interview
Within past 3 months	113	9	36	.587
Not within past 3 months	411	8	38	.600
Transition year
Student in transition year	207	9	38	.606
Not in a transition year	350	9	35	.585

Note. “Efficiency” refers to maximizing the chance of employment for the fewest completed virtual interviews. “Efficacy” refers to maximizing the chance of employment regardless of how many virtual interviews are completed. AUC = area under the curve.

Other race groups had too small of a sample size to generate a reliable cut-point estimate. ^bParticipants may be characterized by more than one IDEA category.

Regarding the VIT-TAY subsample, we observed that the most efficient dose was to complete seven virtual interviews (Table 3). There were a few notable cut-point deviations in this subsample. First, youth with an IQ ≤ 75 had a higher efficient dose of 11 virtual interviews, and students in their transition year had a higher efficient dose of 10 virtual interviews. Meanwhile, students working at baseline had a lower efficient dose of four virtual interviews, (Table 3). For the VR-JIT sample, we observed that the most efficient dose was to complete nine virtual interviews (Table 4). Notable cut-point deviations from the VR-JIT subsample included youth with an IQ ≤ 75 or an intellectual disability, for whom both had an efficient dose of completing 12 virtual interviews. In contrast, the dosing efficiency for African American youth was five virtual interviews.

Table 3.

Cut-Points by VIT-TAY Subsample (Number of Virtual Interviews Completed).

Group	n	Efficiency	Efficacy	AUC
VIT-TAY sample	332	7	38	.641
Race
White	174	9	29	.654
African American/Black	88	7	38	.636
Latinx	39	7	27	.540
Cognitive characteristics
IQ > 75	66	9	23	.744
IQ ≤ 75	79	11	38	.549
Sixth grade or higher reading level	174	9	36	.640
Fifth grade and lower reading level	158	7	38	.617
IDEA categories
Specific learning disability	119	9	38	.659
Autism	75	7	23	.618
Other health impairment	68	6	29	.554
Emotional disturbance	46	8	16	.709
Intellectual disability	54	7	38	.628
Speech and language disability	0	––	––	––
Employment history
Had a job at some point in life	102	6	32	.619
Never worked in life	230	8	38	.676
Employment status
Currently working	28	4	11	.770
Not currently working	302	8	38	.654
Recent job interview
Within past 3 months	62	8	36	.591
Not within past 3 months	265	7	38	.663
Transition year
Student in transition year	170	10	38	.613
Not in a transition year	161	7	32	.620

Note. VIT-TAY = Virtual Interview Training for Transition Age Youth. “Efficiency” refers to maximizing the chance of employment for the fewest completed virtual interviews. “Efficacy” refers to maximizing the chance of employment regardless of how many virtual interviews are completed. AUC = area under the curve.

Other race groups had too small of a sample size to generate a reliable cut-point estimate. ^bParticipants may be characterized by more than one IDEA category.

Table 4.

Cut-Points by Subsample for VR-JIT Subsample (Number of Sessions).

Group	n	Efficiency	Efficacy	AUC
VR-JIT sample	226	9	35	.545
Race
White	150	10	35	.534
African American/Black	27	5	21	.543
Latinx	36	8	20	.555
Cognitive characteristics
IQ > 75	74	10	29	.453
IQ ≤ 75	61	12	29	.560
Sixth grade or higher reading level	114	9	35	.597
Fifth grade and lower reading level	112	10	29	.529
IDEA categories
Specific learning disability	77	10	26	.502
Autism	60	9	29	.563
Other health impairment	68	10	31	.584
Emotional disturbance	43	10	35	.529
Intellectual disability	36	12	23	.477
Speech and language disability	18	8	19	.591
Employment history
Had a job at some point in life	86	10	29	.470
Never worked in life	136	9	35	.591
Employment status
Currently working	31	8	21	.709
Not currently working	195	10	35	.530
Recent job interview
Within past 3 months	51	10	29	.605
Not within past 3 months	146	8	35	.490
Transition year
Student in transition year	37	8	12	.504
Not in a transition year	189	10	35	.563

Note. VR-JIT = Virtual Reality Job Interview Training. “Efficiency” refers to maximizing the chance of employment for the fewest completed virtual interviews. “Efficacy” refers to maximizing the chance of employment regardless of how many virtual interviews are completed. AUC = area under the curve.

Other race groups had too small of a sample size to generate a reliable cut-point estimate. ^b Participants may be characterized by more than one IDEA category.

Virtual Interviewing Efficacy

We found that the number of completed virtual interviews required to maximize the likelihood of successfully obtaining employment (i.e., efficacy) was 38 across the total sample, but varied across race, cognitive characteristics, and IDEA categories, with most subgroups requiring fewer virtual interviews than the overall sample. These numbers are influenced by particular individuals who remained in the intervention for the longest time, and thus the lower numbers in the subsamples likely are a reflection that these longest-tenured individuals do not belong to those groups. These patterns were similar in the VIT-TAY and VR-JIT subsamples (Tables 3 and 4) where the efficacy dose was 38 and 35, respectively.

Area Under the Curve (AUC)

Each cut-point analysis was complemented with an AUC analysis. Table 2 displays the overall sample AUC was .598, which reflected poor discrimination based on standards from the medical field (Hosmer et al., 2013). Meanwhile, the AUC subgroup analyses from the whole sample approached acceptable discrimination at .698 (among those currently working) and ranged down to poor discrimination at .541 (among those who had a job at some point in their life). As displayed in Table 3, the AUC for the VIT-TAY subsample approached acceptable discrimination at .641 with the subgroups ranging from acceptable discrimination (.770, among those currently working) to poor discrimination (.591, among those who had a job interview in the past 3 months). As displayed in Table 4, the AUC for the VR-JIT subsample had poor discrimination at .545 with the subgroups ranging from acceptable discrimination (.709, currently working) to discrimination at no better than chance (.453, among those with IQ > 75).

Discussion

This is the first study to evaluate the dosing of either VIT-TAY or VR-JIT. Based on the use of ROC curve cut-point analyses in the clinical literature (e.g., Karve et al., 2009; Zlobec et al., 2007), we used this approach to test the dosing efficiency and efficacy of virtual interview training. Given the emerging effectiveness of VIT-TAY and VR-JIT when implemented in special education Pre-ETS (Smith, Sherwood, et al., 2022; Smith, Smith, et al., 2021), these virtual interviewing interventions have been implemented in more than 70 special education Pre-ETS and community-based post-secondary pre-employment programs across 10 states in the United States. Thus, the results of this study can provide initial guidance toward recommended practices to deliver virtual interview training.

Specifically, we plotted ROC curves that computed cut-points that predicted obtaining competitive employment while balancing sensitivity and specificity for the samples of youth with disabilities who used either VIT-TAY or VR-JIT and in an exploratory manner within VIT-TAY and VR-JIT separately. We then averaged the cut-points identified in each of the ROC plots for each of the participant characteristics (e.g., race, IQ, reading ability, IDEA categories, and employment history). The average efficiency cut-point to obtain a competitive job across the whole sample was nine, which reflects the maximum chance of obtaining employment for the lowest cost (i.e., fewest possible virtual interviews completed). Notably, there were no deviations from this efficiency in the total sample with regard to any participant characteristic.

Meanwhile, each ROC curve had a corresponding AUC. Within the medical field, ROC with AUC analyses are typically used during the early stages of developing diagnostic tests (e.g., for cancer) where a minimum level of acceptable discrimination is .70 and a value of .50 reflects no better than chance (Hosmer et al., 2013). The AUC for the total sample in this study was .598 which was greater than chance but below the acceptable level. Meanwhile, the AUC for the VIT-TAY subsample approaches an acceptable level at .641 (with several subgroups near or above .70), while the AUC for the VR-JIT subsample was .545 (with few subgroups near .70). The observed difference in AUC between VIT-TAY and VR-JIT could be explained by VIT-TAY being intentionally adapted from VR-JIT (e.g., modified job interview skills, reduction in reading level, greater diversity and scaffolding) to meet the needs of transition-age youth with autism or other disabilities. Overall, there seems to be some value in the AUC results for the VIT-TAY subsample, though it is important to keep in mind that these values do not have the discriminative power of what is expected for a diagnostic test in a medical setting. Thus, these initial AUC results from exploring pragmatic virtual interview dosing can serve as a baseline for comparison by future studies intentionally designed to evaluate VIT-TAY dosing.

We observed that the dosing efficiency of seven virtual interviews among the VIT-TAY subsample did not differ from the efficiency of nine virtual interviews in the total sample. Notably, youth with IQs of 75 or lower who used VIT-TAY had an efficient dose of 11, which suggested they may require more virtual interviews for dosing efficiency. Meanwhile, youth who had a job at baseline had a dosing efficiency of four virtual interviews. However, this result should be interpreted with caution given that this subgroup only represented 8.4% of the VIT-TAY subsample. Youth employed at baseline who used VR-JIT had an efficient dose of eight virtual interviews. Thus, additional research could help elucidate whether VIT-TAY may be more effective than VR-JIT (resulting in a lower dose) among youth who were recently employed or if the present result is nuanced due to small subgroups within the sample.

The VR-JIT subsample’s overall efficient dose was nine virtual interviews, which was consistent with the total sample. Notably, youth with an IQ ≤ 75 or with intellectual disability in this subsample had an efficient dose of completing 12 virtual interviews, which was consistent with the VIT-TAY subsample who had similar IQs. Meanwhile, Black or African American youth had a lower dosing efficiency of five virtual interviews for VR-JIT, which is lower than the efficiency of VIT-TAY. However, the Black or African American youth only represented 11.8% of the VR-JIT subsample. Thus, the results should be interpreted with caution.

Overall, the efficient dose of nine in the total sample is consistent with a recent randomized controlled trial that evaluated the real-world effectiveness of VR-JIT for adults facing mental health challenges. In this study, participants were engaged in the evidence-based version of supported employment (i.e., Individual Placement and Support (Bond, 1998)) and found that adults engaged in community-based vocational rehabilitation services completed approximately nine virtual interviews over approximately 8 to 9 weeks which was associated with greater odds of competitive employment by 9-month follow-up (Smith, Smith, et al., 2022).

Meanwhile, the average efficacy cut-point to obtain a competitive job among the VIT-TAY sample was 38 and 35 in the VR-JIT sample. These cut-points reflect the maximum chance of obtaining competitive employment regardless of cost (i.e., complete as many virtual interviews as necessary). Overall, the balance between efficiency and efficacy should be considered carefully as fewer than 5% of participants completed the most efficacious doses which were time intensive (e.g., >600 min of virtual interview training) and likely not feasible for Pre-ETS programs to deliver.

Our results also suggested the presence of a stepwise association between dosing and competitive employment (i.e., completing one to five virtual interviews: 34.7% employment; completing six to 14 virtual interviews: 43.9% employment; completing 15+ virtual interviews: 61.3% employment). The significance of this finding was driven by the students using VIT-TAY. Notably, the distribution of students who completed each tier (low, medium, and high) of dosing did not differ (and characterized by a small effect size; p = .09, V = .09) between the VIT-TAY (33.4%, 38.6%, and 28.0%, respectively) and VR-JIT (24.8%, 44.2%, and 31.0%, respectively) groups. Although this stepwise view of the data suggests completing more interviews was associated with higher employment rates, one should not overlook the quality of the interviews (i.e., scores) needed to progress from the easy to hard interviews. Our recommendation with respect to a more individualized pattern of virtual interviewing exposure follows in the below study implications.

Implications for Research, Practice, and Policy

Although there has been prior experimental evidence that practicing interview skills with VIT-TAY and VR-JIT has been associated with competitive employment (Smith, Sherwood, et al., 2022; Smith, Smith, et al., 2021), the relationships between the number of completed virtual interviews with employment outcomes in this study is correlational in nature and not causal. Moreover, the previous studies of VIT-TAY and VR-JIT in special education were not designed to randomly assign participants to the doses received. Thus, we cannot assume any equivalence between the levels of dosing (i.e., low, medium, and high) and the estimated dosing efficiency and efficacy in the current study may change if certain confounding variables were controlled. For example, a VIT-TAY subgroup of youth who were employed at baseline had a noticeably smaller efficiency (cut-point of 4) than youth who were not employed at baseline (cut-point of 8). This finding could suggest that VIT-TAY students might have had stronger interview skills given their recent employment and may have required less training to optimize their access to new employment. Alternatively, students employed at baseline who lost their jobs and gained new jobs prior to follow-up may have engaged in less training due to the obtainment of new jobs. Thus, future studies of VIT-TAY and VR-JIT may consider intentionally evaluating randomly assigned doses of virtual interview training to make inferences about causality.

Given the context of the above noted challenge to this study’s external validity and AUC results, we recommend a conservative approach to the implications of our dosing efficiency and efficacy results. Thus, the observed cut-points could provide initial guidance for teachers or providers to consider when developing a flexible, individualized plan that accounts for students’ interview scores (i.e., a marker of interview skill quality) as they follow scoring guidelines to progress from easy to hard interviews. This flexible approach is necessary instead of using a prescribed number of virtual interviews given the study results are not from a definitive dosing study that can confirm a causal relationship.

Moreover, this approach could support teachers as they monitor their students’ virtual interviewing scores and difficulty level, and use that information to modify the students’ progressions in real-time as they progress from easy toward hard interviews. Notably, this progression will reflect interview quality (i.e., meeting a high score threshold) and quantity (i.e., engaging in repeated practice to reinforce learned concepts). For instance, although the current study revealed a cut-point of 11 virtual interviews using VIT-TAY was generated for students with IQs of 75 or lower, students with IQs at 75 or lower will have other strengths that could overcome a prescribed completion of 11 virtual interviews. As such, teachers can use the aforementioned 11 virtual interview cut-point to develop an initial engagement plan (e.g., complete three easy interviews [scoring 90 at least once] then three medium interviews [scoring 90 at least once] then five hard interviews) to help provide students with a structured approach to practice with VIT-TAY. Then teachers can monitor the students’ performance’s and adapt the delivery plans to meet the students’ needs. Here are two examples. First, a student scores 95 on their first two easy interviews and then the teacher asks the student to skip the third easy interview to start interviews at the medium level of difficulty. Second, a teachers may ask a student with lower scores on easy (e.g., scores of 65, 70, and 75) to complete additional interviews until they score 90; thereby adapting the protocol before the student transitions to interviewing at the medium level of difficulty. The need to complete additional interviews on easy in this example also reflects that the quality of one’s VIT-TAY performance is a critical element to progressing from easy to hard interviews. Hence, it is not just the number of completed interviews that likely helps facilitate access to employment, but also the quality of the student’s interview performance.

In terms of implications for future research, the results of the cut-point analyses for the two samples ranged from seven to nine, which is not a meaningful difference with respect to cut-point results. Thus, future studies may consider whether to use the cut-points as a way to guide future dosing studies. Meanwhile, nondosing studies could instead focus on a more general approach of asking participants to complete at least three interviews each on easy, medium, and hard to try and adhere to the overall dosing efficiency of nine virtual interviews. This approach is pragmatically adaptable (similar to the above examples) given that some participants may have stronger or more limited interview skills than others and may need a tailored approach to progress through the difficulty levels. Overall, the results of this analysis have identified some initial dosing strategies that can potentially be used as a starting point when preparing individualized plans to use virtual interviewing tools for transition-age youth with disabilities. Finally, future research can also inform a more individualized approach to VIT-TAY delivery through student-level interviews eliciting recommendations on implementation strategies.

Limitations

The study has important limitations to consider within the context of potential implications. First, the small subgroup sample sizes suggest these results should be interpreted with caution. Moreover, future research is needed to replicate the dose responses of VR-JIT and VIT-TAY. Specifically, the use of fully-powered samples of the different subgroups will improve the external validity of the results. For instance, the cut-point of some subgroups should be interpreted with caution as they each represented <12% of the overall sample (e.g., participants identifying as African Americans, IDEA categorizations of emotional disturbance). Second, although cut-point analyses are a novel technique to evaluate dosing efficiency and efficacy, the associated AUC results for the whole sample and VR-JIT subsample provide poor discriminatory power for employment outcomes as compared with medical diagnostic assessments (Hosmer et al., 2013). Thus, the current results should be interpreted as recommended guidance for implementation rather than prescribed cutoffs for virtual interview training. Moreover, future research could consider using a latent class analysis to evaluate naturally occurring clusters within this data that emerge from how youth respond to VIT-TAY and VR-JIT.

Fourth, students with visual impairment, deafness, hearing impairment, deaf-blindness, orthopedic impairment, traumatic brain injury, and multiple disabilities were underrepresented (i.e., fewer than 10 cases) or not represented in the current sample. Thus, future research is needed to evaluate the overall effectiveness of VIT-TAY and VR-JIT (including dosing efficiency and efficacy) for students with the above IDEA designations. Fifth, the requirement for students to score at least 90 out of 100 points to progress between levels was an arbitrary cutoff determined during the original VR-JIT development that was carried forward to VIT-TAY. Anecdotal feedback from teachers and students recommended lowering this thresholds to 85 or 80 as real-world interviews don’t require a perfect answer to every interview question to facilitate a job offer. Thus, future research should evaluate this recommendation more clearly as a reduction in this scoring requirement may help students progress more efficiently from easy to hard interviews. Sixth, the VIT-TAY sample consisted of significantly more students in their “transitional” year who focused more explicitly on Pre-ETS. However, the cut-point analyses for VIT-TAY and VR-JIT did not demonstrate a meaningful difference when comparing students (from either intervention group) who were in their “transitional” year as compared with students not in their transitional year.

Finally, school participants were connected with their state vocational rehabilitation offices and the findings have limited generalizability to schools and students who are not connected to state vocational rehabilitation. Notably, school involvement with state vocational rehabilitation helps drive employment and not all special education teachers have the skills, knowledge, and training that state vocational rehabilitation counselors can offer students. As a result, future studies may want to specifically evaluate the effectiveness and implementation of virtual interview tools within schools who are not connected to state vocational rehabilitation.

Conclusion

This study provides a novel approach to strengthen the field’s understanding of VIT-TAY and VR-JIT dosing efficiency and efficacy, which may assist Pre-ETS programs choosing to implement the virtual interview training tools. Overall, the results suggest that completing approximately nine virtual interviews may provide the optimal amount of training while trying to balance the amount of training completed with obtaining a competitive job. However, this recommendation should be considered within the limitations of the study design and analyses. Moreover, teachers are encouraged to weigh their students’ strengths and areas for growth against the recommended cut-points to individualize virtual interview training delivery, which can still be further tailored based on the students’ performances with the tools.

Supplemental Material

sj-docx-1-cde-10.1177_21651434231160532 – Supplemental material for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities

Supplemental material, sj-docx-1-cde-10.1177_21651434231160532 for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities by Matthew J. Smith, Mark Van Ryzin, Neil Jordan, Marc Atkins, Lindsay A. Bornheimer, Kari Sherwood and Justin D. Smith in Career Development and Transition for Exceptional Individuals

Supplemental Material

sj-docx-2-cde-10.1177_21651434231160532 – Supplemental material for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities

Supplemental material, sj-docx-2-cde-10.1177_21651434231160532 for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities by Matthew J. Smith, Mark Van Ryzin, Neil Jordan, Marc Atkins, Lindsay A. Bornheimer, Kari Sherwood and Justin D. Smith in Career Development and Transition for Exceptional Individuals

Supplemental Material

sj-docx-3-cde-10.1177_21651434231160532 – Supplemental material for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities

Supplemental material, sj-docx-3-cde-10.1177_21651434231160532 for Virtual Job Interview Training: A Dose Response to Improve Employment for Transition-Age Youth With Disabilities by Matthew J. Smith, Mark Van Ryzin, Neil Jordan, Marc Atkins, Lindsay A. Bornheimer, Kari Sherwood and Justin D. Smith in Career Development and Transition for Exceptional Individuals

Footnotes

Acknowledgements

The authors acknowledge the Administration and Staff from the following entities: the Level Up: Employment Skills Simulation Lab within the University of Michigan School of Social Work; Division of Rehabilitation Services, Department of Mental Health, and Secondary Transitional Experience Program within the Illinois Department of Human Services; Chicago Public Schools Office of Diverse Learners Supports and Services; Michigan Rehabilitation Services; Project SEARCH administrative team at Cincinnati Children’s Hospital Medical Center, and Project SEARCH staff at our partnering sites in Illinois, Michigan, and Florida for their help with this project and their continued support in our work together. Also, the authors acknowledge the administrators, educational staff, and students from our school partners for participating in this project.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The University of Michigan will receive royalties from SIMmersion LLC on the sales of the virtual interview training for transition-age youth tool that was the focus of this study. These royalties will be shared with Dr. M.J.S. and the University of Michigan School of Social Work. Dr. M.J.S. adhered to the University of Michigan’s Conflict Management Plan that was reviewed and approved by a University of Michigan Conflict of Interest Committee. No other authors report any conflicts of interest.

Funding

This study was supported by grants from the Kessler Foundation (1003-1958-SEG-FY2016, PI: Matthew J. Smith). Marc S. Atkins was supported by the National Center for Advancing Translational Sciences, National Institutes of Health Grant UL1TR002003. Justin D. Smith was supported by the National Institute on Drug Abuse, National Institutes of Health Grant P30DA027828. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Kessler Foundation and the National Institutes of Health.

ORCID iDs

Matthew J. Smith

Kari Sherwood

Supplemental Material

Supplementary material for this article is available on the Career Development and Transition for Exceptional Individuals website with the online version of this article.

References

Arter

Brown

Law

Barna

Fruehan

Fidiam

(2018). Virtual reality: Improving interviewing skills in individuals with autism spectrum disorder. In Langran

Borup

(Eds.), Proceedings of society for information technology & teacher education international conference (pp. 1086–1088). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/182660/

Bond

(1998). Principles of the individual placement and support model: Empirical support. Psychiatric Rehabilitation Journal, 22(1), 11–23. https://doi.org/10.1037/h0095271

Bureau of Labor Statistics. (2021). Persons with a disability: Labor force characteristics 2021 (USDL-22-0317). https://www.bls.gov/news.release/pdf/disabl.pdf

Cobb

R. B.

Lipscomb

Wolgemuth

Schulte

Veliquette

Alwell

Batchelder

Bernard

Hernandez

Holmquist-Johnson

Orsi

Sample McMeeking

Wang

Weinberg

(2013). Improving post-high school outcomes for transition-age students with disabilities: An evidence review (NCEE 2013-4011). Institute of Educational Science. https://ies.ed.gov/ncee/pubs/20134011/pdf/20134011.pdf

Fombonne

(1991). The use of questionnaires in child psychiatry research: Measuring their performance and choosing an optimal cut-off. Journal of Child Psychology and Psychiatry, 32(4), 677–693. https://doi.org/10.1111/j.1469-7610.1991.tb00343.x

Genova

H. M.

Lancaster

Morecraft

Haas

Edwards

DiBenedetto

Krch

DeLuca

Smith

M. J.

(2021). A pilot RCT of virtual reality job interview training in transition-age youth on the autism spectrum. Research in Autism Spectrum Disorders, 89, 101878. https://doi.org/10.1016/j.rasd.2021.101878

Geverdt

(2015). Education Demographic and Geographic Estimates Program (EDGE): Locale boundaries user’s manual (NCES 2016-012). National Center for Education Statistics, U.S. Department of Education. https://files.eric.ed.gov/fulltext/ED577162.pdf

Hayes

G. R.

Custodio

V. E.

Haimson

O. L.

Nguyen

Ringland

K. E.

Ulgado

R. R.

Waterhouse

Weiner

(2015). Mobile video modeling for employment interviews for individuals with autism. Journal of Vocational Rehabilitation, 43, 275–287. https://doi.org/10.3233/JVR-150775

Herrin

(2002). CLTEST: Stata modules for performing cluster-adjusted chi-square and t-tests (Statistical Software Components S424901). Boston College Department of Economics.

10.

Hill

L. G.

Coie

J. D.

Lochman

J. E.

Greenberg

M. T.

(2004). Effectiveness of early screening for externalizing problems: Issues of screening accuracy and utility. Journal of Consulting and Clinical Psychology, 72(5), 809–820. https://doi.org/10.1037/0022-006X.72.5.809

11.

Hosmer

D. W.

Lemeshow

Sturdivant

R. X.

(2013). Applied logistic regression (3rd ed.). Wiley.

12.

Individuals with Disabilities Education Improvement Act IDEA, 20 U.S.C. § 1400, et seq. (2004).

13.

Karve

Cleves

M. A.

Helm

Hudson

T. J.

West

D. S.

Martin

B. C.

(2009). Good and poor adherence: Optimal cut-point for adherence measures using administrative claims data. Current Medical Research and Opinion, 25(9), 2303–2310. https://doi.org/10.1185/03007990903126833

14.

McNeil

B. J.

Hanley

J. A.

(1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Medical Decision Making, 4(2), 137–150. https://doi.org/10.1177/0272989X8400400203

15.

R Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/

16.

Rowe

D. A.

Mazzotti

V. L.

Fowler

C. H.

Test

D. W.

Mitchell

V. J.

Clark

K. A.

Holzberg

Owens

T. L.

Rusher

Seaman-Tullis

R. L.

Gushanas

C. M.

Castle

Chang

W.-H.

Voggt

Kwiatek

Dean

(2021). Updating the secondary transition research base: Evidence- and research-based practices in functional skills. Career Development and Transition for Exceptional Individuals, 44(1), 28–46. https://doi.org/10.1177/2165143420958

17.

Schrank

F. A.

McGrew

K. S.

Mather

(2014). Woodcock-Johnson IV. Riverside.

18.

Smith

M. J.

Fleming

M. F.

Wright

M. A.

Jordan

Humm

L. B.

Olsen

Bell

M. D.

(2015). Job offers to individuals with severe mental illness after participation in virtual reality job interview training. Psychiatric Services, 66(11), 1173–1179. https://doi.org/10.1176/appi.ps.201400504

19.

Smith

M. J.

Fleming

M. F.

Wright

M. A.

Losh

Humm

L. B.

Olsen

Bell

M. D.

(2015). Brief report: Vocational outcomes for young adults with autism spectrum disorders at six months after virtual reality job interview training. Journal of Autism and Developmental Disorders, 45(10), 3364–3369. https://doi.org/10.1007/s10803-015-2470-1

20.

Smith

M. J.

Fleming

M. F.

Wright

M. A.

Roberts

A. G.

Humm

L. B.

Olsen

Bell

M. D.

(2015). Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment. Schizophrenia Research, 166, 86–91. https://doi.org/10.1016/j.schres.2015.05.022

21.

Smith

M. J.

Ginger

E. J.

Wright

M. A.

Taylor

J. L.

Humm

L. B.

Olsen

D. E.

Bell

M. D.

Fleming

M. F.

(2014). Virtual reality job interview training in adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 44(10), 2450–2463. https://doi.org/10.1007/s10803-014-2113-y

22.

Smith

M. J.

Ginger

E. J.

Wright

M. A.

Wright

Humm

L. B.

Olsen

Bell

M. D.

Fleming

M. F.

(2014). Virtual reality job interview training for individuals with psychiatric disabilities. Journal of Nervous and Mental Disease, 202(9), 659–667. https://doi.org/10.1097/NMD.0000000000000187

23.

Smith

M. J.

Pinto

R. M.

Dawalt

Smith

J. D.

Sherwood

Miles

Taylor

J. L.

Hume

K. A.

Dawkins

Baker-Ericzén

Frazier

T. W.

Humm

L. B.

Steacy

(2020). Using community-engaged methods to adapt virtual reality job-interview training for transition-age youth on the autism spectrum. Research in Autism Spectrum Disorders, 71, 101498. https://doi.org/https://doi.org/10.1016/j.rasd.2019.101498

24.

Smith

M. J.

Sherwood

Blajeski

Ross

Smith

J. D.

Jordan

Dawalt

Bishop

Atkins

M. S.

(2021). Job interview and vocational outcomes among transition-age youth receiving special education pre-employment transition services. Intellectual and Developmental Disabilities, 59(5), 405–421. https://doi.org/10.1352/1934-9556-59.5.405

25.

Smith

M. J.

Sherwood

Ross

Oulvey

E. A.

Atkins

M. S.

Danielson

E. A.

Jordan

Smith

J. D.

(2022). Scaling out virtual interview training for transition age youth: A quasi-experimental hybrid effectiveness-implementation study. Career Development and Transition for Exceptional Individuals, 45(4), 213–227. https://doi.org/10.1177/21651434221081273

26.

Smith

M. J.

Sherwood

Ross

Smith

J. D.

DaWalt

Bishop

Humm

L. B.

Elkins

Steacy

(2021). Virtual interview training for autistic transition age youth: A randomized controlled feasibility and effectiveness trial. Autism, 25(6), 1536–1552. https://doi.org/10.1177/1362361321989928

27.

Smith

M. J.

Smith

J. D.

Blajeski

Ross

Jordan

Bell

M. D.

McGurk

Mueser

K. T.

Burke-Miller

Oulvey

E. A.

Fleming

M. F.

Nelson

Brown

Prestipino

Pashka

Razzano

L. A.

(2022). An RCT of virtual reality job interview training for individuals with serious mental illness in IPS supported employment. Psychiatric Services, 73(9), 1027–1038. https://doi.org/10.1176/appi.ps.202100516

28.

Smith

M. J.

Smith

J. D.

Jordan

Sherwood

McRobert

Ross

Oulvey

E. A.

Atkins

M. S.

(2021). Virtual reality job interview training in transition services: Results of a single-arm, noncontrolled effectiveness-implementation hybrid trial. Journal of Special Education Technology, 36(1), 3–17. https://doi.org/10.1177/0162643420960093

29.

StataCorp. (2021). Stata Statistical Software: Release 17. StataCorp LLC.

30.

U.S. Department of Education. (2017). A transition guide to postsecondary education and employment for students and youth with disabilities. Office of Special Education and Rehabilitation Services. https://sites.ed.gov/idea/files/postsecondary-transition-guide-may-2017.pdf

31.

Ward

D. M.

Esposito

M. C. K.

(2019). Virtual reality in transition program for adults with autism: Self-efficacy, confidence, and interview skills. Contemporary School Psychology, 23, 423–431. https://doi.org/10.1007/s40688-018-0195-9

32.

Wechsler

(2014). Wechsler Intelligence Scale for children (5th ed.). Pearson.

33.

Workforce Innovation and Opportunity Act WIOA, 29 U.S.C. § 3164, Pub. L. No. PL 113-128 (2014).

34.

Zlobec

Steele

Terracciano

Jass

J. R.

Lugli

(2007). Selecting immunohistochemical cut-off scores for novel biomarkers of progression and survival in colorectal cancer. Journal of Clinical Pathology, 60(10), 1112–1116. https://doi.org/10.1136/jcp.2006.044537

35.

Zweig

M. H.

Campbell

(1993). Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39(4), 561–577. https://doi.org/10.1093/clinchem/39.4.561

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.88 MB

0.03 MB

0.01 MB