Abstract
Job coaches are not typically trained to implement systematic instructional strategies to teach vocational skills to students with intellectual and developmental disabilities (IDD). This study replicated and expanded the evaluation of a job coach behavioral skills training program designed by Brock et al. (2016) to teach participants to implement task analysis, simultaneous prompting, and system of least prompts to teach vocational tasks to students with IDD. We used a multiple probe design with probe conditions across strategies, replicated across three participants, to assess acquisition and generalization. Participants demonstrated mastery of the three instructional strategies in simulated assessments with actors and generalized use of the strategies to teach novel vocational tasks to student interns with IDD. Implications for future research and practice are discussed.
Securing community-based employment is central to the lives of people with and without disabilities; yet individuals with intellectual and developmental disabilities (IDD) too often experience disappointing employment outcomes following their exit from the educational system (Qian et al., 2018). School-to-work transition programs provide students with IDD opportunities to receive vocational skills instruction both in the classroom and in employment settings through work-based learning experiences (e.g., job shadowing, internships, and simulated work experience; Wehman et al., 2021). Work-based learning provides students with opportunities to gain vocational knowledge and skills through internships in authentic employment settings where they receive direct instruction to learn the vocational skills that they will need to achieve their desired job once they exit the educational system (National Technical Assistance Center on Transition, 2015; Workforce Innovation Technical Assistance Center, 2020).
Students with IDD often receive vocational skills instruction from job coaches who work directly with the student to provide or facilitate the supports needed for the individual to eventually be successful in competitive, integrated employment (Rogan & Held, 1999; Wehman et al., 2021). Most importantly, job coaches provide direct, systematic instruction in community-based employment settings to facilitate acquisition and independence in performing vocational tasks. To successfully help students with IDD acquire vocational skills and learn to work independently, job coaches must (a) use effective systematic instructional strategies to teach vocational skills, (b) provide workplace supports such as visual aids or assistive technologies, and (c) transfer from job coach prompting and instruction to cues in the natural environment to support independent workplace performance (Wehman, Schall, et al., 2014).
Systematic instructional strategies are focused intervention practices that address specific skills or goals by analyzing and assessing performance, teaching new skills, and fading support to build independence (Hume et al., 2021; Wehman, Chan, et al., 2014). In a recent systematic review of the instructional strategies most frequently used to teach vocational skills to students with IDD, Gilson et al. (2017) reported that, among 21 instructional methods used across 56 studies and 766 students, the most effective strategies were implemented in community-based employment settings and were behaviorally based, including task analysis and response prompting. The National Technical Assistance Center on Transition (2019) similarly reported that these behaviorally based strategies are considered effective practices (i.e., evidence-based, research-based, or promising) for vocational skills instruction.
To teach a vocational task, a job coach can develop a task analysis (i.e., a list of sequentially ordered steps, where each step is part of the overall task; Cooper et al., 2007) to teach the vocational task in smaller, easier to manage steps. Then the job coach can use response prompting (e.g., verbal, gestural, or physical cues or instructions given to support the student with IDD in acquiring the skill; Hume et al., 2021) to teach the steps in the task analysis to the student with IDD. Although there are several forms of response prompting, two strategies with evidence of effectiveness for teaching vocational tasks to individuals with IDD are simultaneous prompting and the system of least prompts (Hume et al., 2021). Simultaneous prompting is an instructional strategy in which a controlling prompt (i.e., the prompt that is most likely to result in a correct response) is immediately delivered after a task direction is provided or after completion of the previous step in a task analysis. Simultaneous prompting is advantageous to teach new tasks because it minimizes opportunities for incorrect responding (e.g., there are only opportunities for prompted responding) and only one type of prompt needs to be delivered throughout the training procedure (Tekin-Iftar et al., 2019). Once a skill is acquired, it is important to fade prompting. As such, the system of least prompts can be used to systematically fade instructor supports until the student with IDD can complete the task independently. The system of least prompts is the delivery of prompts from various levels of intrusiveness in a predetermined hierarchy, starting with the least intrusive prompt (e.g., gestural) and gradually working up to the most intrusive prompt (e.g., physical), if necessary (Shepley et al., 2019).
Despite the efficacy and use of task analysis and response prompting (including simultaneous prompting and the system of least prompts) to teach vocational tasks to students with IDD, job coaches rarely receive any meaningful degree of training in implementing instructional strategies, nor do they receive proper oversight and support (Gilson et al., 2017, 2021; Hall et al., 2018; Rogan & Held, 1999; Wehman, Schall, et al., 2014). This lack of training and oversight can limit the ability of job coaches to provide vocational skills instruction and effective support to students with IDD, thereby impacting vocational skill acquisition and opportunities for independent employment after high school.
In response to the need to prepare job coaches to implement effective instructional strategies with high levels of fidelity when instructing students with IDD in vocational settings, this study was designed to replicate and extend a job coach training program designed by Brock et al. (2016). Brock et al. (2016) used a behavioral skills training (BST) approach (see Kirkpatrick et al., 2019, for a review of BST in supporting instructional behaviors) to teach six participants to use task analysis, simultaneous prompting, and the system of least prompts to teach vocational tasks. The participants (who were employed to support individuals with IDD) demonstrated skill acquisition in a simulated role-play setting where they used each instructional strategy to teach two vocational tasks (e.g., washing a table) to actors without disabilities, and then demonstrated generalized use of the instructional strategies to teach a novel job task (bagging groceries) that they had not been trained to teach (again to actors; Brock et al., 2016).
Despite the positive effects of the BST job coach training by Brock et al. (2016), there were limitations to the study, which should be addressed. First, participants only used the instructional strategies to teach two vocational tasks and generalization was only assessed with a single novel task. Given that individuals with IDD may be assigned new job tasks at any time, it is important to ensure that job coaches apply the instructional strategies to several different job tasks with fidelity and without the need for additional training. Thus, this study was designed to train across a range of job tasks to promote generalization and to assess fidelity of implementation of instructional strategies across job tasks. The inclusion of a range of job tasks also reduced performance bias related to multiple exposures of the same task.
Second, Brock et al. (2016) only assessed implementation of the systematic instructional strategies during role-play scenarios with actors without disabilities, and data were not collected for implementation with individuals with IDD. Given that job coaches will be working with individuals with IDD, it is important to ensure that training in the instructional strategies will generalize to the population with whom they will be implemented. This study extended the findings of Brock et al. by examining generalized use of the instructional strategies for implementation with individuals with IDD. There were two primary research questions, as follows:
Method
Participants
Job coach participants
The study was advertised to undergraduate students who volunteered with a peer mentoring program for young adults with IDD at a major Midwestern university. Three students signed up for the study; each reported that, although they had previous experience interacting with individuals with IDD, they had never been employed as a job coach (or other service provider for individuals with IDD) and did not have prior training or experience with task analysis, simultaneous prompting, or system of least prompts. All job coach participants (hereafter referred to as job coaches) had a high school diploma and were in their final year of pursuing a bachelor’s degree. Jody was a 22-year-old White female pursuing a degree in human resources and labor relations. Ramaya was a 22-year-old Black female pursuing a degree in kinesiology. Isha was a 21-year-old South Asian female pursuing a degree in neuroscience. All job coaches provided informed consent and received US$100 at the conclusion of the study.
Actors
Four actors were recruited and trained to perform in simulation assessments to measure job coach acquisition and performance of the instructional strategies. Three actors were White female students (age 21–29 years) pursuing a master’s degree in applied behavior analysis (ABA); the fourth was a 24-year-old White male with a bachelor’s degree in aerospace engineering.
Student interns with IDD
For generalization assessments, three student interns with IDD were recruited from a school district’s 1-year school-to-work transition program that was based at the university. Any student intern attending the transition program could volunteer for the study if they met the following inclusion criteria: (a) school-based eligibility for special education services (e.g., active individualized education program) under the category of intellectual disability or autism spectrum disorder (ASD), and (b) had not previously received systematic instruction on the generalization tasks (making a milkshake and bagging groceries). From the six volunteers, three student interns were randomly selected and then randomly assigned to a job coach. All three student interns were White, 23 years old, male, and pursuing a special education certificate of completion. Intern 1 (working with Jody) had a diagnosis of ASD and mild to moderate intellectual disability. Intern 2 (working with Ramaya) had a diagnosis of a genetic syndrome and mild to moderate intellectual disability. Intern 3 (working with Isha) had a diagnosis of ASD and mild to moderate intellectual disability.
Setting and Materials
Simulation assessments and training sessions for job coaches took place in a university conference room. Generalization assessments occurred in the school-to-work transition program classroom housed on the university campus. Training session materials included a computer with internet access, a video camera, and a PowerPoint presentation for each strategy. During training and simulation and generalization assessments, we provided paper and a pen to write the task analyses and the necessary materials for conducting the various tasks. For example, materials for setting the table included a place mat, a plate, a cup, a napkin, and silverware, as well as a picture of how the final arrangement should look.
Dependent Variable and Data Collection
The primary dependent variable was the percentage of steps implemented correctly by the job coaches for the three systematic instructional strategies (task analysis, simultaneous prompting, and system of least prompts) during assessments with a “student” (i.e., simulation assessments with actors and generalization assessments with student interns with IDD). The first author video recorded all simulation and generalization assessments of job coach performance and coded them immediately following the assessment.
Task analysis
To measure performance on the task analysis, we provided the job coach with a description of a task (including the materials and final product) and asked them to generate a task analysis. We then coded the task analysis for whether it met the following general criteria: (a) each step was a broken-down part of the larger skill, (b) the steps were listed in an order where the task could be completed correctly, (c) the steps were written specifically enough to be followed correctly, (d) the steps were written concisely in direct language, and (e) extraneous steps were not included. To measure criterion (a), we compared the job coach’s task analysis with a master task analysis generated for that task at the start of the study (see “Procedures” section), receiving 1 point for each step of the task analysis that matched the master task analysis. We then coded criteria (b) to (e) for the task analysis as either met, coded as 1, or not met, coded as 0. We calculated the dependent measure for each task analysis assessment by adding the number of steps matching the steps on the master task analysis for criterion (a) plus the number of criteria (b) to (e) that were met (e.g., 5 correct steps + 3 criteria were met) and dividing by the total number of steps on the master task analysis plus 4 for the total number of criteria that could be met (e.g., 8 steps on the task analysis + 4 criteria) and multiplying the product by 100 to yield a percentage.
Simultaneous prompting
To measure simultaneous prompting, we provided the job coach with a task analysis and instructed the job coach to use simultaneous prompting to teach a student each step within the task analysis. We coded performance for whether the job coach met the following criteria: the job coach (a) gave the initial vocal task direction (e.g., “Set the table”) to begin the task, (b) provided a prompt within 1 s of the task direction or completion of the previous step (e.g., “set the napkin on the placemat”), (c) provided immediate reinforcement (e.g., vocal praise) within 1s after completion of each step (e.g., “great job”), and (d) provided specific reinforcement for completion of the entire task (e.g., “Great job, you set the table!”). We coded criteria (a) and (d) as either met, coded as 1, or not met, coded as 0, for the task as a whole. We coded criteria (b) and (c) as met, coded as 1, or not met, coded as 0, for each step within the task analysis. We calculated the dependent variable by dividing the score for simultaneous prompting by the total possible score for the specific task and multiplying the product by 100 to yield a percentage.
System of least prompts
To measure performance of the system of least prompts, we provided the job coach with a task analysis and instructed them to use the system of least prompts to fade support for each step in the task analysis. We coded performance for whether the job coach met the following criteria: the job coach (a) gave the initial task direction, (b) waited 5 to 7 s without providing prompts or reinforcement for the student to respond after providing the task direction, (c) gave a gestural prompt toward the required item or items for that step (e.g., pointing to the milk) if the student did not respond or responded incorrectly, (d) gave a vocal verbal prompt (e.g., “Pour the milk in the measuring cup up to the red line”) if the student did not respond within 5 to 7 s or responded incorrectly, (e) gave a model prompt (e.g., saying “like this” and pretending to pour milk) if the student did not respond within 5 to 7 s or responded incorrectly, (f) provided specific vocal praise after the student completed each step in the task analysis, and (g) delivered specific reinforcement for completing the overall task (e.g., “Great, you made a milkshake!”). Criteria (a) and (g) were coded as either met, coded as 1, or not met, coded as 0, for the whole task. We coded criteria (b) through (f) as met, coded as 1, or not met, coded as 0, for each step and required progression through the task analysis. Specifically, the job coach participant could receive 1 point for giving the task direction, (1–4) points each for correctly progressing through the prompting hierarchy for each step in the task analysis, 1 point each for reinforcing the steps of the task analysis, and 1 point for providing specific praise at the conclusion of the task. The dependent variable was calculated by dividing the score for system of least prompts by the total possible score for the specific task and multiplying the product by 100 to yield a percentage.
Interobserver Reliability and Procedural Fidelity
The first author, a second-year ABA master’s student, served as the primary coder for the dependent variables and determined, through visual analysis, when participants were ready to progress through assessment and training. The primary coder trained the reliability coder, a master’s level certified rehabilitation counselor, to evaluate accuracy of the primary coder’s data collection. The reliability coder practiced with sample videos and task analyses for each strategy until they met 90% agreement with the primary coder. We calculated interobserver agreement (IOA) for 34% of assessments for each strategy in each phase for each participant. We calculated point-by-point IOA by taking the number of steps in which coders agreed on a rating, dividing that number by the sum of the number of agreements and disagreements, and multiplying by 100 to obtain a percentage of agreement. IOA was 94% for task analysis, 95% for simultaneous prompting, and 91% for system of least prompts across observations.
The same reliability coder also completed procedural fidelity checklists for 33% of the recorded simulations to ensure that the actors followed the specified guidelines (e.g., made the correct number of errors, did not provide feedback or reinforcement). We calculated actor procedural fidelity (93% across observations) by dividing the number of steps the actors implemented correctly by the total number of steps, and multiplying by 100 to obtain a percentage.
Finally, the reliability coder assessed procedural fidelity on training implementation for 33% of training for each strategy. We developed a task analysis of the BST for each strategy and calculated procedural fidelity by taking the number of criteria implemented correctly, dividing by the total number of criteria, and multiplying by 100 to obtain a percentage of steps implemented correctly. Procedural fidelity was 100% across all observed trainings.
Experimental Design
We used a multiple probe design with probe conditions across instructional strategies, replicated across three participants (Ledford & Gast, 2018). We probed all instructional strategies prior to implementation of the job coach training for one strategy (e.g., task analysis) and then again each time a targeted strategy reached mastery criterion. Probing all instructional strategies in this manner met the What Works Clearinghouse (2020) single-case design standards with reservations, allowed for an examination of the extent to which any previously mastered strategy maintained following intervention, and assessed whether the remaining untaught strategies occurred at levels similar to baseline prior to intervention. Thus, we demonstrated experimental control if job coach implementation fidelity increased only when the training was implemented for each strategy.
The job coaches began the baseline simulation assessment for all instructional strategies at the same time. Following baseline, we provided the job coach training for task analysis until the job coach met the mastery criterion of performing the strategy with at least 85% accuracy on three simulation assessments. We then conducted a probe of all three strategies prior to beginning simultaneous prompting BST. This continued until all three strategies were mastered, with the same mastery criterion applied to all three strategies. We conducted generalization assessments with student interns with IDD for each job coach following completion of training on all strategies.
Procedures
Prior to the start of this study, we generated a list of tasks and then wrote a task analysis for each task. All tasks could be broken down into eight to 12 steps in a task analysis. A team of three ABA graduate students and one doctoral-level behavior analyst reviewed the steps for each task analysis and came to a consensus through discussion on the order and number of steps included for each task. The team then reviewed each task analysis again and collectively used clinical judgment to determine whether all tasks were similar in complexity and difficulty. We used the generated task analyses as the master coding sheet with which the job coaches’ task analyses were compared for scoring purposes. See Table 1 for the list of tasks, the number of steps in each task, and the condition in which the task was used.
List of Tasks Used in Training Sessions and Assessments.
Note. TA = task analysis; SLP = system of least prompts.
Baseline
At the start of the study, the instructor (the first author) conducted baseline simulation assessments for each strategy. During simulation assessments, the instructor provided the job coaches with a brief definition of a strategy (see Table 2) and asked them to write a task analysis, teach the task using simultaneous prompting, and then fade support using the system of least prompts with a student actor. The instructor did not provide any social praise or feedback during baseline. Only the job coach, instructor, and actor posing as a student were present for the assessment.
Definitions of Instructional Strategies Provided to Job Coaches During All Assessments.
When the job coach arrived in the room, the instructor provided them with the definitions of the instructional strategies and assigned them a novel task to teach. The instructor first asked them to write a task analysis to perform the task. The instructor gave them a paper and pen to write the task analysis, the materials needed to complete the task, and a list of parameters for the task, including a description or picture of the task materials and what the materials should look like once the task was completed. For example, to write a task analysis for setting the table, the instructor placed the materials used for the task on the table (not set up) and gave the job coach a picture of what the materials should look like once setting the table had been completed.
After the job coach completed the assessment for task analysis, the instructor asked them to use simultaneous prompting to teach the task to a student actor who was learning the task for the first time. The job coach used the task analysis they generated so that we did not provide them with a model of a task analysis before beginning intervention on task analysis. We randomly assigned each student actor with a controlling prompt and trained them to wait for that prompt from the job coach before completing any steps of the task.
Finally, once the job coach completed the assessment for simultaneous prompting, we asked them to use the system of least prompts to support a student actor who had been learning the task for a while (using the same task analysis). During the system of least prompts assessments, we instructed the student actors to make a predetermined number of errors on predetermined steps in the task analysis. We used a random number generator to determine the number of errors that would occur (between 2 and 5 total), at which steps the errors would occur, and how many prompts were required before the student actor would respond correctly.
Intervention
The first author served as the instructor and conducted training with each participant individually at least 2 times per week. Similar to Brock et al. (2016), all participants progressed through training of the three strategies in the same order (task analysis, then simultaneous prompting, and then system of least prompts). The instructor scheduled the first training session for each strategy for 2 hr (or until training was complete) and then each subsequent training session for 1 hr (or until training was complete). To promote generalization, the instructor used a new task but the same implementation steps for instruction and rehearsal during each training session for each instructional strategy.
The instructor used BST to teach the instructional strategies (i.e., task analysis, simultaneous prompting, and system of least prompts). BST (Miltenberger, 2012) involves a four-step procedure in which the strategy is taught through instructions, modeling, rehearsal, and feedback. For instruction, the instructor used a PowerPoint presentation to define the strategy and describe each step of implementation. Specifically, for teaching task analysis, the instructor taught the job coaches to use this strategy to break a larger task into more manageable steps by walking through a task and writing down each step. To teach simultaneous prompting, the instructor taught the job coaches to use vocal and model prompts and instructed them that the first prompt they used that was followed by a correct response was considered the successful prompt (e.g., controlling prompt) that should be used for all subsequent steps in that task for that student. The instructor taught the job coaches to give a vocal task direction, to then provide either a vocal or model prompt within 1 s of the task direction or completion of the previous step, and to provide immediate reinforcement in the form of vocal praise within 1 s after completion of each step and after completion of the entire task. Finally, for the system of least prompts, the instructor taught the job coaches to use a hierarchy of prompts if the student did not respond within 5 to 7 s or responded incorrectly, beginning with the lowest prompt in the hierarchy. In sequential order, the hierarchy included a gestural prompt toward the required items for that step, a vocal verbal prompt, and a model prompt. If the student did not respond within 5 to 7 s of the prompt or if the student made an error, the job coach then moved on to the next highest prompt in the hierarchy and waited 5 to 7 s for the student to respond again. This process was taught to be repeated until the student responded correctly. When the student responded correctly to a prompted step, the job coach provided reinforcement in the form of specific vocal praise. Finally, the job coach delivered specific praise for the overall task.
Once the instructor taught the specific strategy through instruction, he provided a model using a sample task to show an example of how a job coach could implement the strategy with a student with IDD. The instructor modeled the strategy by either writing a task analysis or by implementing simultaneous or system of least prompts with a student actor while the job coach watched. The job coach then rehearsed implementing the strategy by either writing a task analysis or practicing with a student actor and received brief in-the-moment feedback related to their performance during rehearsal and more detailed feedback after the rehearsal.
Finally, the instructor asked the job coach to rehearse the strategy one more time using a novel task (to avoid practice effects and to allow assessment of generalization of performance across novel tasks throughout the training process). After the job coach completed the rehearsal, the instructor provided reinforcement and/or corrective feedback. For task analysis, the instructor read the task analysis aloud and provided vocal praise and/or corrective feedback. For simultaneous and system of least prompts, the instructor recorded each performance. Following completion of the simulation, the instructor transferred the video to a laptop and the instructor and job coach watched the video together while the instructor provided vocal behavior-specific praise and/or corrective feedback. At the end of each training session, the instructor calculated and graphed the percentage of steps performed accurately during the final rehearsal of the strategy and visually analyzed the data. Training on one strategy continued until the job coach met mastery criterion of 85% or above on the final rehearsal for three sessions.
Probe Assessment
Once the job coach met mastery criterion for a strategy, the instructor conducted a probe assessment that was identical to baseline for all three strategies. This probe assessment was conducted to establish a steady state of responding for the untrained strategies and to probe that the acquired strategy maintained in the absence of BST and prior to assessing generalization. This continued until the job coach mastered all three strategies and then the job coach participant was moved to generalization.
Generalization
The instructor conducted generalization assessments to assess the job coaches’ implementation of the instructional strategies with novel tasks with a student intern with IDD. The instructor assessed generalization for each of the three strategies by measuring job coach implementation fidelity for teaching a student intern with IDD two novel job tasks that were similar to tasks they would complete at internship sites in their school-to-work transition program (making a milkshake and bagging groceries).
First, because task analysis always precedes the response prompting procedures, the instructor asked the job coach to write a task analysis for making a milkshake and bagging groceries (providing two opportunities to assess generalization of task analysis). Then, the instructor conducted three generalization sessions to assess the use of simultaneous prompting and system of least prompts (providing three opportunities to assess generalization of each prompting strategy). For the first generalization session, the instructor randomly assigned each strategy to each task, so that the job coach first used simultaneous prompting to teach one task to the student intern with IDD and first used the system of least prompts to teach the second task. The instructor then alternated the experimental sequence (i.e., order in which prompting strategies were evaluated) for the two tasks for the next session. For example, Isha began with bagging groceries for simultaneous prompting in the first session, so the order for bagging groceries across the three sessions was simultaneous prompting, system of least prompts, and then back to simultaneous prompting. Her order for making a milkshake was the inverse (i.e., started with system of least prompts).
Social Validity
We collected social validity from the job coaches before and after training. Before training, we asked job coaches whether they had heard of each strategy, and then asked to what degree they believed they could implement the strategy. Job coaches rated their ability to implement the strategies on a scale from 1 to 5 (1 = unable to implement the strategy correctly, 2 = unable to implement the strategy correctly most of the time, 3 = able to implement the strategy correctly sometimes, 4 = able to implement the strategy correctly most of the time, 5 = able to implement the strategy correctly all the time). After training, we asked the second question again, and compared responses with the pretraining scores to assess the degree to which the job coaches thought they had mastered the strategy. After the training, we also asked open-ended questions about the job coaches’ thoughts on the training sessions and procedures, and to what degree they found them acceptable and helpful.
Data Analysis
We used visual analysis to assess for a functional relation between the training and the performance of the three instructional strategies. We coded simulation and generalization probes and graphically displayed the data following each assessment to allow for ongoing evaluation of behavior change within and across instructional strategies (Ledford & Gast, 2018). Upon introduction of the training, we assessed for changes in variability, level, and trend for each instructional strategy for each job coach to determine the effectiveness of the intervention.
Results
We conducted a multiple probe experiment for each of the three participants across the three instructional strategies, providing nine opportunities to demonstrate a functional relation between the independent and dependent variables. See Figures 1, 2, and 3 for a visual depiction of changes in level, variability, and trend.
Percentage of steps Jody performed correctly across instructional strategies during probe, training, and generalization sessions.
Percentage of steps Ramaya performed correctly across instructional strategies during probe, training, and generalization sessions.
Percentage of steps Isha performed correctly across instructional strategies during probe, training, and generalization sessions.
Jody
Results of the intervention on acquisition of the instructional strategies for Jody are displayed in Figure 1. Jody displayed low levels of accurate performance on all instructional strategies during baseline probe sessions, with an immediate increase in performance in targeted strategies when the intervention was implemented. For task analysis, Jody’s performance during the first baseline assessment was 75%, before dropping to 50% for the two subsequent sessions. When Jody entered intervention for task analysis, she showed an immediate effect in her first intervention session and met mastery criterion within three intervention sessions, with scores of 100%, 90%, and 100%. For simultaneous prompting, Jody’s baseline data were stable and low near 30%. When Jody entered intervention for simultaneous prompting, she showed an immediate effect in her first intervention session with a score of 100% and performance remained at 100% for the remaining two sessions. For system of least prompts, Jody’s performance was somewhat variable during baseline as she scored 20% on the initial probe, 5% for the subsequent three probes, and then 20% on her final probe. There was an immediate effect following the first intervention session as she scored 92%, 90%, and 90%. Jody displayed generalized performance of all three strategies to working with a student intern with IDD.
Ramaya
Results of the intervention on acquisition of the instructional strategies for Ramaya are displayed in Figure 2. Ramaya displayed low levels of accurate performance on all instructional strategies during baseline assessments, with an immediate increase in performance in targeted strategies when the intervention was implemented. For task analysis, Ramaya’s performance in the first baseline assessment was 67%, before dropping to 14% for the two subsequent sessions. When Ramaya entered intervention for task analysis, she showed an immediate effect in her first intervention session and met mastery criterion within five sessions, with scores of 79%, 100%, 57%, 100%, and 92%. For simultaneous prompting, Ramaya’s baseline data were somewhat variable, with a slight downward trend from 25% to 5% in the first three baseline probes before rising back to 31% for the final baseline probe. When Ramaya entered intervention for simultaneous prompting, she showed an immediate effect in her first intervention session with a score of 94%. Performance maintained at this level and Ramaya met mastery criterion within three sessions. For system of least prompts, Ramaya’s performance during the initial baseline probe was 24%, before increasing to around 30% and remaining stable for subsequent baseline assessments. There was an immediate effect following the first intervention session as she scored 83%. After a slight decrease to 72%, Ramaya’s performance increased to 95% and maintained between 90% and 95% for subsequent sessions, meeting mastery criterion after five sessions. Ramaya displayed generalized performance of all three strategies to working with a student intern with IDD, with the exception of her first generalization probe for system of least prompts where her performance decreased to 74%, Ramaya’s performance was between 85% and 100% for generalization assessments for all three strategies.
Isha
Results of the intervention on acquisition of the instructional strategies for Isha are displayed in Figure 3. Isha displayed low levels of accurate performance on all instructional strategies during baseline, with an immediate increase in performance in targeted strategies when the intervention was implemented. For task analysis, Isha’s performance during the first baseline assessment was 83%, before dropping to 36% and 29% for the two subsequent sessions. When Isha entered intervention for task analysis, she showed an immediate effect in her first intervention session. She met mastery criterion within three intervention sessions, with scores of 86%, 100%, and 93%. For simultaneous prompting, Isha’s baseline data were stable near 37%. When Isha entered intervention for simultaneous prompting, she showed an immediate effect in her first intervention session with a score of 100%. Isha’s performance decreased slightly to 94% during the second intervention session before returning to 100%. For system of least prompts, Isha’s performance was relatively stable and low during baseline at around 25%. There was an immediate effect following the first intervention session as she scored 83% before increasing to 94% and maintaining between 90% and 100% for subsequent sessions, and met mastery criterion after four sessions. Isha displayed generalized performance of all three strategies to working with a student intern with IDD, with the exception of her first generalization probe for system of least prompts where her performance decreased to 76%, Isha’s performance was between 90% and 100% for generalization assessments for all three strategies.
Social Validity
Before training, all three job coaches indicated that they had never heard of any of the target strategies. All three job coaches rated their ability to implement simultaneous prompting and system of least prompts as 1 (unable to implement the strategy correctly), and Jody and Ramaya rated their ability to implement task analysis as 1 as well. Isha rated her ability to implement task analysis as 2 (unable to implement the strategy correctly most of the time). After training, Jody and Isha both rated their ability to implement task analysis and simultaneous prompting as 5 (able to implement the strategy correctly all the time), and rated system of least prompts as 4 (able to implement the strategy correctly most of the time). Ramaya rated her ability to implement task analysis as 5, simultaneous prompting as 4, and system of least prompts as 3 (able to implement the strategy correctly sometimes).
For the open-ended questions, job coaches indicated that the length of the overall training was appropriate although Jody and Ramaya indicated that the training lasted a long time (over a month) because of scheduling difficulties and the detail of the training. All three job coaches indicated that each training session was an acceptable length. They indicated that the teaching presentations, instructor demonstrations, opportunities for rehearsal, and instructor feedback contributed positively to the training and were helpful for them to learn the strategies. Job coaches also stated that they would likely use their knowledge of the strategies in their future careers (employment law, occupational therapy, and neuroscience for Jody, Ramaya, and Isha, respectively). The strategies were also viewed as being beneficial for students with IDD who might receive support from job coaches using systematic instructional practices. Finally, job coaches said that they felt the training was appropriate for new job coaches, and that the training would have a high success rate for teaching newly hired job coaches with no prior experience.
Discussion
Despite their important role in school-to-work transition programs and vocational skills instruction, job coaches rarely receive training in how to implement systematic instructional strategies (Rogan & Held, 1999; Wehman, Schall, et al., 2014). This study was a replication of a job coach training program designed by Brock et al. (2016) to teach job coaches task analysis, simultaneous prompting, and system of least prompts. We extended the study to assess acquisition across several vocational tasks and evaluate generalization of the strategies to teaching student interns with IDD. All job coaches showed an immediate increase in performance following the introduction of the intervention, mastering each strategy within three to five intervention sessions. Job coaches continued to display high rates of performance in the absence of the job coach training and they displayed generalized responding across all three strategies when teaching vocational tasks to a student intern with IDD.
In evaluating the first research question, the results indicate that, similar to Brock et al. (2016), implementation fidelity applying three systematic instructional strategies to teach vocational tasks during simulation assessments with student actors immediately increased in level only after introduction of the training for each strategy. Thus, the job coach training using BST was an effective method to teach systematic instructional strategies to job coaches. Based on these findings, similar procedures could be used by job coach employers to train newly hired staff on how to analyze a task, begin to teach a task with minimal student errors, and systematically fade prompting to promote student independence.
The results of the first research question further extend the findings from Brock et al. (2016) as the job coaches in this study demonstrated the ability to apply the systematic instructional strategies across a range of vocational tasks. Brock et al. (2016) showed that job coaches could successfully acquire and apply the three target strategies with two vocational tasks during training sessions and generalize the use of these strategies to one untrained task. In this study, a new vocational task was used for each training and assessment, allowing for exposure to more than 18 different vocational tasks. All three job coaches mastered the three strategies and used them with a new task each session, indicating that these training procedures may be effective at programming for generalization across tasks. These findings have important implications for organizations providing training to job coaches and indicate that practicing instructional strategies with a variety of different stimuli and tasks may help improve job coaches’ ability to implement these strategies with various vocational tasks.
In evaluating the second research question, job coaches displayed generalization of implementation of the three systematic instructional strategies to teaching novel vocational tasks to student interns with IDD. These findings extend those from Brock et al. (2016) and highlight the importance of assessing generalized performance to working with students with IDD. Although all three job coaches were able to generalize the strategies to teaching student interns with IDD, Ramaya and Isha’s performance on the system of least prompts initially fell below mastery criterion levels (during the first generalization assessment). The previous job coach training only assessed implementation through simulations with actors (Brock et al., 2016) and the decrease in performance during the generalization probes in this study highlights the importance of assessing generalization to the population with whom the job coaches will most likely be working. Given that both Ramaya and Isha showed a decrease in performance for system of least prompts during their first generalization probe before performance improved and returned to mastery levels, future research should further plan for generalization. For example, similar to other implementations of BST (Miltenberger, 2012), future research should examine whether job coaches can be taught to use the strategies in situ with students with IDD (rather than in simulations first).
Limitations
There are limitations of this study related to participant selection, dependent variables, and assessment of generalization. First, we recruited the job coaches through advertisements to undergraduate students rather than through job coach employment agencies. As such, the participants in this study likely did not match the population of individuals who are typically hired as job coaches. Although it would have been ideal to assess the procedures with newly hired job coaches, this was not feasible.
Second, although we made several efforts to control task difficulty (e.g., keeping the number of steps within each task analysis between eight and 12 steps, reviewing all tasks for consensus regarding difficulty), the tasks used in each session may not have been as similar in difficulty and complexity as intended and may have inadvertently affected performance. For example, all three job coaches performed significantly higher on developing a task analysis for the first baseline assessment (setting the table) compared with the next two baseline assessments. This suggests that the first baseline assessment was an outlier and the task may have been less difficult or complex than the subsequent tasks. We could have mitigated this limitation by randomly counterbalancing the tasks across the baseline probes and across participants.
Furthermore, although we evaluated to ensure that each step was written specifically enough to be followed correctly, we did not rate the task analyses for whether the steps were written in objective and measurable terms. As a result, it is possible that the job coaches did not produce task analyses that could be used to accurately assess job skill performance. To keep the procedures simple for simultaneous prompting, we did not teach job coaches to conduct probe trials before prompting trials, and it is therefore not known whether they could incorporate this component. Finally, we did not randomize the order in which simultaneous prompting and the system of least prompts were taught across participants. It is possible that acquisition of simultaneous prompting impacted acquisition of the system of least prompts. Given the consistently low baseline levels, however, this risk of bias is low.
Third, we only conducted generalization probes with student interns with IDD post-intervention. As a result, although performance was low during all baseline assessments with student actors, we cannot definitively conclude that the job coaches would not have been able to perform the instructional strategies with student interns with IDD without training. We did not conduct a pre-intervention generalization probe because it was unethical to ask the student interns to complete a new job task while receiving potentially improper support from untrained “job coaches.” A final limitation is that, because the generalization probes took place in a controlled classroom environment, they did not fully resemble the conditions under which job coaches will be expected to implement the instructional strategies. Because job sites vary considerably and require nuanced considerations (e.g., cafeterias will most likely be loud, busy, and chaotic, and job coaches and their students may be easily distracted in these employment environments), it is not known whether skills would have generalized to job settings.
Implications for Future Practice and Research
Despite these limitations, the job coach training analyzed in this study could have implications for employers of future job coaches and students with IDD who are transitioning from school to an employment setting, as well as implications for future research. Students with IDD deserve the most appropriate and high-quality vocational skills instruction and preparation available, and job coaches deserve to be given the appropriate tools and training to provide that instruction. Given the lack of training that is currently provided to most newly hired job coaches (Hall et al., 2018), the procedures and findings of this study provide a feasible option for those who hire and train job coaches. Although the training sessions in this study were fairly long (1–2 hr across 3–5 sessions for each strategy), the implications of providing job coaches with effective instructional strategies may mitigate the intensive training provided up front. Effective training and supervision are empirically shown to lead to increased skill acquisition and independence, better educational and employment outcomes, and improved quality of life for individuals with IDD (Brock & Anderson, 2021; Butterworth et al., 2017; Novak et al., 2014). Furthermore, job coaches and other providers with high-quality training and supervision in instructional strategies report lower levels of stress and burnout (Graber et al., 2008), which may lead to improved job performance, treatment fidelity, and the use of systematic instructional strategies with students with IDD (Greenway et al., 2013).
Future research could evaluate the rate of student intern learning and performance for the target vocational tasks when taught by a job coach who has not received the BST job coach training on systematic instructional strategies, compared with those who are taught by a job coach who has received such training. Given that the three instructional strategies are evidence-based and supported by a large research literature (Hume et al., 2021), there is reason to hypothesize that students with IDD will more efficiently and effectively acquire job skills when taught by trained job coaches using these systematic instructional strategies, which could reduce the length of time job coach supports would be needed. Thus, results could be used to advocate for more widespread adoption of behaviorally based trainings in instructional strategies for job coaches.
Finally, the results indicate that individuals with no prior experience or training in systematic instructional strategies can be effectively taught how to implement these strategies with high levels of fidelity. Whereas in Brock et al. (2016) job coaches had between 3 and 30 years of experience providing direct support to individuals with IDD, the participants in this study had no prior experience job coaching or providing systematic instruction. Despite the differences in experience, participants in both studies demonstrated low levels of performance of all three instructional strategies in baseline, highlighting the critical need for a systematic training and further indicating that the existing training for job coaches is woefully inadequate. The results of this study, combined with Brock et al. (2016), indicate that not only can job coach training effectively teach participants with years of experience (Brock et al., 2016), but the procedures may also be appropriate for individuals with entry-level skills and experience. Based on these findings, future research should examine the effectiveness of the training on teaching newly hired job coaches to analyze a task (i.e., task analysis), begin to teach a task with minimal student errors (i.e., simultaneous prompting), and systematically fade prompting to promote student independence (i.e., system of least prompts).
Footnotes
Authors’ Note
Portions of this article—mainly in the Results discussion—were included as part of the master’s thesis of the first author, under the direction of the second author, at Michigan State University. The thesis is on the ProQuest site.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This work was supported in part by the Center for Research in Autism, Intellectual, and Other Neurodevelopmental Disabilities at Michigan State University.
