Using data-enabled performance feedback and guidance to assist employment consultants in their work with job seekers: An experimental study

Abstract

BACKGROUND:

The success of job seekers with disabilities in achieving their employment goals depends in large part on the quality of employment supports that they receive from employment consultants.

OBJECTIVE:

To test the effectiveness of data-enabled performance feedback to assist employment consultants in implementing standards of effective employment supports.

METHODS:

A total of 187 employment consultants in 30 states were randomly assigned to intervention or control groups. The intervention group received data-enabled performance feedback and guidance for 12 months, whereas the control group continued with business as usual. Both groups completed baseline and quarterly surveys throughout the intervention.

RESULTS:

One year after baseline, the intervention group reported a statistically significant improvement in job seekers’ work hours, compared to the control group. Earnings and time to hire improved as well, but the change was not statistically significant. There was no meaningful difference in the number of job seekers hired across the intervention and control groups.

CONCLUSION:

Challenges in the fidelity of implementation of the intervention make it premature to draw conclusions about the effectiveness of data-enabled performance feedback to employment consultants for improving job seekers’ employment outcomes.

Keywords

Employment specialist job development intellectual disabilities training evidence-based

1 Introduction

People with intellectual and developmental disabilities want to work, be independent, and contribute to society (Barrows et al., 2016; Butterworth et al., 2015; Gilson et al., 2018; Migliore et al., 2007). However, based on the 2016–2017 National Core Indicators Survey, only 15% of people with intellectual and developmental disabilities work in paid individual employment. Furthermore, those who work earn wages considered inadequate for achieving economic self-sufficiency (Hiersteiner et al., 2018; Wehman et al., 2018). Low rates of employment and low earnings place people with disabilities at further economic, social, and cultural disadvantages compared to their peers without disabilities (Nye-Lengerman & Nord, 2016).

Fortunately, federal and state policies have been enacted to promote employment of people with disabilities, including the Workforce Innovation and Opportunity Act of 2014 (WIOA), state-level Employment First policies, and the Home and Community-Based Services (HCBS) Settings Final Rule (CMS 2249-F/2296-F) in 2014. Moreover, over 5,400 employment programs and an estimated 32,000 employment consultants are available for providing employment support to job seekers with disabilities (Haines et al., 2013; Migliore et al., 2010). For the purposes of this study, employment consultants are employment support professionals who assist job seekers with disabilities in finding employment. Employment consultants may be referred to also as employment specialists, job developers, or career development specialists.

These employment consultants can access training, credentialing, technical assistance, and an extensive literature to learn about standards of effective employment supports (ACRE, 2013, 2018; APSE, 2019; Novak et al., 2014). Figure 1 summarizes these standards under an umbrella of five elements that lead to a job match and hire (Migliore, Nye-Lengerman et al., 2018).

Fig. 1

The flow chart shows on the left four elements arranged in a circle around “Job match”. The four elements include build trust, get to know job seekers, find tasks/jobs, and supports planning. This block connects on the right with the outcome “Hire” and a fifth element of the model: Support after hire.

Building trust with job seekers is consistent with the literature about self-determination and ensuring that people with disabilities make their own choices in all aspects of their lives, including their career paths (Barrows et al., 2016; Shogren et al., 2016; Smull & Sanderson, 2009; Wehmeyer, 2011). Engaging with family members can help with finding better job matches and sustaining retention (Blacher et al., 2010; Jones & Gallus, 2016; Migliore et al., 2007).

Getting to know job seekers’ strengths, goals, and skills is key for informing the job search criteria and achieving the best job match (Griffin et al., 2007; Hoff et al., 2000). The literature emphasizes asking questions, observing job seekers in community environments, and learning from other people in the job seekers’ social and familial circles (Callahan et al., 2009; Griffin et al., 2007; Petner-Arrey et al., 2016; Phillips et al., 2009; Wehman et al., 2016).

Supports planning refers to assisting job seekers in identifying and accessing any other supports that lead to a smooth job entry. This includes supporting them to develop their work and social skills, planning for transportation early in the process, and facilitating access to benefits planning (Friedman & Rizzolo, 2016; Harvey et al., 2013; Haveman et al., 2013; Luecking & Luecking, 2013; Riesen et al., 2015).

Finding tasks/jobs aims at getting a job offer. Finding jobs is most effective through leveraging personal connections with employers, including tapping into the job seekers’ and their families’ personal and professional networks (Bolles, 2018; Levinson & Perry, 2011; Petner-Arrey et al., 2016). Listening to employers and addressing their needs is key for gaining trust and building long-term relationships with the business community (Carpenter & Daly, 2019; Gilbride & Stensrud, 2008; Luecking, 2008). When existing job openings are not a good fit, the emphasis should be on looking for tasks, rather than jobs. This approach aims at expanding the employment opportunities by negotiating new job descriptions that better align with job seekers’ skills and preferences while addressing unmet needs of employers (Griffin et al., 2007).

Supports after hire is about assisting the businesses to make sure that the new hire becomes a full contributing member of the social and professional fabric of the workplace. It includes facilitating natural supports on the job and promoting the new hire’s professional advancement (Barrows et al., 2016; Carpenter & Daly, 2019; Griffin et al., 2007; Mank et al., 1999; Wehman et al., 2012).

As shown above, an extensive body of knowledge exists about how to support job seekers in their pursuit of employment. Unfortunately, these standards of effective employment supports are not necessarily implemented in the field. In fact, fidelity of implementation is a widespread challenge in human services and beyond (Baker et al., 2001; Fixsen et al., 2005; Grol & Wensing, 2004; IHI, 2003; Shojania & Grimshaw, 2005). As Bhattacharyya, Reeves, and Zwarenstein (2009) put it in reference to human services:

“ ... there is a large gap between what is known and what is consistently done ... ” (p. 491)

To bridge the gap between theory and practice, the literature emphasizes tracking behavior and providing performance feedback (Amabile & Kramer, 2011; Bajpai et al., 2015; Bond et al., 2012; Seifert et al., 2017). In the case of employment consultants, the need for tracking their employment support practices and providing them with guidance for improvement is well documented (Drake et al., 2009, Graham et al., 2013; Inge et al., 2016; NIDILRR, 2017; Sudsawad, 2007; Wehman et al., 2018). When tracking behavior, theories from clinical psychology and adult learning emphasize shifting the emphasis from retrospective self-report of past behavior to repeated measures of current behavior (Shiffman et al., 2008; Walz et al., 2015). Based on these considerations, we developed and tested an intervention package—described in the Method section—that included data-enabled performance feedback and guidance designed to assist employment consultants in the implementation of standards of effective employment supports when assisting job seekers with intellectual and developmental disabilities. The experimental study aimed to answer one core research question:

Do employment consultants who are exposed to a package that includes data-enabled performance feedback, peer supports, online training, micro-learning, and distant mentoring report better employment outcomes for the job seekers whom they support?

Our hypothesis was that employment consultants who received the package of intervention activities would report better employment outcomes, including more hires, longer work hours, higher earnings, and shorter time from job search to hire, compared to their peers who did not receive the same intervention activities.

2 Method

The research design of this study was experimental, with random assignment of employment programs and their employment consultants to either an intervention or a control group. In this paper, “employment” refers to work that includes being paid at least minimum or prevailing wage; being paid by the host employer, not the service provider; and working in settings where the majority of co-workers have characteristics that reflect the surrounding communities. To minimize repetition, we use the generic term “job seeker” when referring to a job seeker with intellectual and developmental disabilities unless otherwise specified. The following sections describe the participants, instruments, procedure, and data analysis.

2.1 Participants

A total of 187 employment consultants enrolled, and their organizations were randomly assigned to either an intervention or control group as described later in this section. Table 1 provides baseline characteristics of the 120 employment consultants who participated for the entire one-year duration of the intervention study. Overall, there were only small differences in gender, proportion of job seekers with intellectual and developmental disabilities on the caseload, percentage of time invested in providing supports before hire, holding a Certified Employment Support Professional (CESP) certification, geographic location, and number of hires during the year before baseline.

Table 1
Characteristics of Employment Consultants at Baseline

Intervention Control Effect Size^*

Gender n (%)

Male 11(18%) 16(27%) 0.22 (Small)

Female 48 (80%) 43(72%) 0.19 (Negligible)

Race n (%)

White 53(88%) 48(81%) 0.19 (Negligible)

Black or African American 4(7%) 6(10%) 0.11 “ “

Asian 1(2%) 1(2%) 0.00 “ “

Other 2(3%) 4(7%) 0.19 “ “

Ethnicity n (%)

Hispanic or Latino 1(2%) 3(5%) 0.17 “ “

Not Hispanic, Not Latino 57(98%) 56(95%) 0.17 “ “

Education n (%)

High school or some college 19(32%) 18(31%) 0.02 “ “

Undergraduate or graduate 41(68%) 41(69%) 0.02 “ “

Professional Characteristics Mean (Min, Max, n)

Years of tenure 6 (1, 34, 60) 6 (1,28, 60) 0.00 “ “

Weekly work hours 42(35, 60, 60) 41(30, 60, 60) 0.15 “ “

Total caseload Mean 20(1, 165, 60) 23(0, 80, 60) 0.13 “ “

Caseload % intellectual/developmental dis. 83% (0, 100, 60) 75% (0, 100, 60) 0.20 (Small)

Other Professional Characteristics n (%)

Supervisory role 27(45%) 28(47%) 0.04 (Negligible)

Over 50% time invested in supports before hire 34(57%) 45(75%) 0.38 (Small)

CESP certification 20(33%) 11(18%) 0.35 “ “

ACRE certification 13(22%) 15(25%) 0.07 (Negligible)

Geographic Location of EC n (%)

Northeast 18(30%) 26(43%) 0.27 (Small)

Midwest 23(38%) 10(17%) 0.48 “ “

West 11(18%) 15(25%) 0.17 (Negligible)

South 8(13%) 9(15%) 0.06 “ “

Employment Outcomes Mean (Min, Max, n)

Total hires (IDD+Other disabilities) 8(0, 28, 60) 11(0, 46, 60) 0.31 (Small)

% hires with IDD of the total hires 86(0, 100, 57) 82(0,100,55) 0.17 (Negligible)

Months from job search to hire 6(0, 66, 52) 6(0, 48, 54) 0.04 “ “

Hires’ weekly work hours 16(2, 35, 53) 19(2, 48, 53) 0.34 (Small)

Hires’ hourly earnings $10 (7, 13, 54) $10 (7, 15, 56) 0.13 (Negligible)

	Intervention	Control	Effect Size^*
Gender n (%)
Male	11(18%)	16(27%)	0.22 (Small)
Female	48 (80%)	43(72%)	0.19 (Negligible)
Race n (%)
White	53(88%)	48(81%)	0.19 (Negligible)
Black or African American	4(7%)	6(10%)	0.11 “ “
Asian	1(2%)	1(2%)	0.00 “ “
Other	2(3%)	4(7%)	0.19 “ “
Ethnicity n (%)
Hispanic or Latino	1(2%)	3(5%)	0.17 “ “
Not Hispanic, Not Latino	57(98%)	56(95%)	0.17 “ “
Education n (%)
High school or some college	19(32%)	18(31%)	0.02 “ “
Undergraduate or graduate	41(68%)	41(69%)	0.02 “ “
Professional Characteristics Mean (Min, Max, n)
Years of tenure	6 (1, 34, 60)	6 (1,28, 60)	0.00 “ “
Weekly work hours	42(35, 60, 60)	41(30, 60, 60)	0.15 “ “
Total caseload Mean	20(1, 165, 60)	23(0, 80, 60)	0.13 “ “
Caseload % intellectual/developmental dis.	83% (0, 100, 60)	75% (0, 100, 60)	0.20 (Small)
Other Professional Characteristics n (%)
Supervisory role	27(45%)	28(47%)	0.04 (Negligible)
Over 50% time invested in supports before hire	34(57%)	45(75%)	0.38 (Small)
CESP certification	20(33%)	11(18%)	0.35 “ “
ACRE certification	13(22%)	15(25%)	0.07 (Negligible)
Geographic Location of EC n (%)
Northeast	18(30%)	26(43%)	0.27 (Small)
Midwest	23(38%)	10(17%)	0.48 “ “
West	11(18%)	15(25%)	0.17 (Negligible)
South	8(13%)	9(15%)	0.06 “ “
Employment Outcomes Mean (Min, Max, n)
Total hires (IDD+Other disabilities)	8(0, 28, 60)	11(0, 46, 60)	0.31 (Small)
% hires with IDD of the total hires	86(0, 100, 57)	82(0,100,55)	0.17 (Negligible)
Months from job search to hire	6(0, 66, 52)	6(0, 48, 54)	0.04 “ “
Hires’ weekly work hours	16(2, 35, 53)	19(2, 48, 53)	0.34 (Small)
Hires’ hourly earnings	$10 (7, 13, 54)	$10 (7, 15, 56)	0.13 (Negligible)

^*Effect Size d (means) and h (proportions) > 0.20 = Small; > 0.50 = Medium; > 0.80 = Large (Cohen, 1988).

Several employment consultants were the only participants from their organizations (30% and 45% for intervention and control groups respectively), some had one or two colleagues also involved in the study (64% and 52% in the intervention and control group), whereas a very small proportion of participants had more than three colleagues involved in the study (6% and 3% in the intervention and control group).

As part of the study, each employment consultant selected a job seeker who most recently—before the baseline survey—achieved employment with the employment consultant’s primary support. As the Table 2 shows, there were some small differences across the intervention and control groups in regard to the characteristics of the job seekers most recently hired, including gender, race, ethnicity, age, education, on-the-job support after hire, community-based non-work services after hire, and residential settings. There was a medium difference in regard to having a college or higher education background.

Table 2

Characteristics of Most Recently Hired Job Seekers, at Baseline

	Intervention	Control	Effect Size^*
Gender n (%)
Male	33(58)	37(65)	0.14 (Negligible)
Female	24(42)	18(32)	0.21 (Small)
Race n (%)
White	44(77)	35(63)	0.31 “ “
Black or African American	7(12)	14(25)	0.34 “ “
Asian	1(2)	2(3)	0.06 (Negligible)
Other	5(9)	5(9)	0.00 “ “
Ethnicity n (%)
Hispanic or Latino	5(9)	2(4)	0.21 (Small)
Age Mean (Min, Max, n)	32(17, 75, 56)	29(19, 62, 55)	0.30 “ “
Education n (%)
Some high school	8(13)	9(16)	0.09 (Negligible)
High school diploma	36(63)	41(72)	0.19 “ “
Some college or higher education	11(20)	3(6)	0.43 (Small)
Don’t know	2(4)	4(7)	0.13 (Negligible)
Hours Requiring On-the-Job Support n (%)
None	8(13)	16(27)	0.36 (Small)
Up to 50%	28(47)	19(32)	0.31 “ “
Over 50%	20(34)	23(38)	0.08 (Negligible)
Day Services Received after Hire n (%)
None	27(47)	32(55)	0.16 “ “
Community-based non-work	17(30)	9(16)	0.34 (Small)
Facility-based work or non-work	6(10)	9(16)	0.18 (Negligible)
Other or don’t know	7(13)	8(14)	0.03 “ “
Residential Settings n (%)
Living on their own	10(18)	12(21)	0.08 “ “
With parents/extended family	32(56)	30(53)	0.06 “ “
Semi-independent living, no staff	5(9)	6(10)	0.04 “ “
Group home with overnight staff	6(10)	6(10)	0.00 “ “
Shared living or foster care	3(5)	0(0)	0.45 (small)
Other or don’t know	1(2)	3(6)	0.21 “ “

^*Effect Size d (means) and h (proportions) > 0.20 = Small; > 0.50 = Medium; > 0.80 = Large (Cohen, 1988).

2.2 Instruments

A baseline survey, four quarterly surveys, and a daily survey were used in this study. The baseline survey and four quarterly surveys were administered to both intervention and control groups, whereas the daily survey was administered to the intervention group only. The baseline survey—distributed at the onset of the study, in the spring of 2017—asked about the characteristics of the employment consultants and of the most recent job seeker hired. The four quarterly surveys—administered between June 1, 2017 and May 31, 2018—tracked any major changes in the participants’ professional characteristics, as well as job seekers’ employment outcomes. The employment consultants received a $25 gift card for completing each quarterly survey.

The daily survey was distributed to the participants in the intervention group each weekday at a different, random time between 9:00 am and 4:00 pm, for the one-year duration of the intervention between June 1, 2017 and May 31, 2018. The daily survey asked three multiple-choice questions about the 30 minutes right before the employment consultants received a text with the link to the survey: 1) what was the employment consultants’ primary activity, 2) who was the interaction with, and 3) where did the activity take place. The response to each of the three questions led to second-level questions asking for more detail. For example, if a participant responded that the primary activity was finding jobs, a follow-up question asked what type of job-finding activity was performed (e.g., browsing ads, networking, cold calling). If a participant responded that the primary interaction was with the job seeker’s “Family/Social Circle,” a follow-up question inquired who that person was (e.g., a parent, a sibling, a friend). Moreover, once per month the daily survey asked whether employment consultants had been able to implement the goal set during the most recent community of practice (explained below).

Participants could suspend the daily survey when on vacation, on a leave, or out sick. The daily response rate of employment consultants at work was 90% (Min = 58%; Max = 99%). More details about the daily survey are available in Migliore, Butterworth, et al., 2018.

2.3 Procedure

After receiving approval from the Institutional Review Board (IRB) at the two universities involved in carrying out the study, we proceeded with recruitment, random assignment, and the delivery of the intervention.

2.3.1 Recruitment

The employment consultants were recruited through a call for participation distributed through the mailing lists of the Association of People Supporting Employment-First (APSE), the National Association of State Directors of Developmental Disabilities Services (NASDDDS), and the State Employment Leadership Network (SELN). A total of 103 directors of employment programs responded to the call and completed a short online form providing some basic information about their programs. After verifying these programs’ focus on providing employment services to job seekers with intellectual and developmental disabilities, we sent an invitation email to be forwarded to their individual employment consultants. The invitation included information about the study and eligibility criteria, an IRB-approved consent form, and a link to the baseline online survey. The eligibility criteria for employment consultants included 1) providing employment support services to job seekers with disabilities, 2) having at least one year of experience in this role, 3) working full time, and 4) including at least 50% of job seekers with intellectual or developmental disabilities on their caseload.

As illustrated in Fig. 2, a total of 219 employment consultants completed the baseline survey, and 187 were eligible and enrolled in the study. A total of 107 employment consultants were assigned to the intervention group, and 80 employment consultants were assigned to the control group through random assignment as described later. During the one-year duration of the study, 47 participants dropped out of the intervention group, either because they were no longer providing employment services (n = 11) or for personal reasons (n = 36). A further 20 employment consultants dropped out of the control group, 10 because they no longer provided employment services and 10 for personal reasons. One year after baseline, the retention rates were 63% for the intervention group and 86% for the control group, after excluding participants who interrupted participation because they no longer provided employment support services.

Fig. 2

The flowchart summarizes the flow of participants from 219 employment consultants at baseline to 60 employment consultants in each group at the end of the study.

As shown in Table 3, the employment consultants in the intervention group who dropped out tended to report the same number of hires, but fewer months to hire and more weekly work hours, compared to their peers who remained active throughout the duration of the study. The employment consultants who dropped out from the control group differed from their peers in the control group as they reported fewer hires, fewer months from job search to hire, and fewer weekly work hours compared to their peers who remained active throughout the duration of the study.

Table 3

Differences in Baseline Outcomes Across Active and Drop-out Participants

	Active	Dropped Out	Effect Size
	Mean (Min, Max, n)	Mean (Min, Max, n)
Intervention Group Participants
Hires (any disability)	8 (0, 28, 60)	8 (0, 25, 42)	0.01 (Negligible)
% hires with IDD of the total hires	86% (0%, 100%, 57)	79% (0%, 100%, 38)	0.29 (Small)
Months to hire	6 (0, 66, 52)	5 (0, 22, 34)	0.21 “ “
Hours/week	16 (2, 35, 53)	20 (4, 40, 36)	0.54 (Medium)
$/Hour	$10 ($7, $13, 54)	$10 ($7, $14, 35)	0.26 (Small)
Control Group Participants
Hires (any disability)	11 (0, 46, 60)	7 (0, 30, 18)	0.34 (Small)
% hires with IDD of the total hires	82% (0%, 100%, 55)	86% (50%, 100%, 16)	0.17 (Negligible)
Months to hire	6 (0, 48, 54)	4 (1, 18, 16)	0.26 (Small)
Hours/week	19 (2, 48, 53)	17 (1, 30, 15)	0.25 “ “
$/Hour	$10 ($7, $15, 56)	$10 ($7, $12, 17)	0.33 “ “

^*Effect Size d (means) and h (proportions) > 0.20 = Small; > 0.50 = Medium; > 0.80 = Large (Cohen, 1988).

2.3.2 Randomization

The organizations, not employment consultants, were randomly assigned to either the intervention or control groups to avoid spillover of intervention effects among employment consultants at the same employment program. More specifically, organizations were matched in pairs with another organization with similar employment outcomes at baseline. Each organization within the pair was then randomly assigned to either the intervention or control groups. This design—a special case of randomized block design—is recommended for increasing the likelihood that both the intervention and the control group include participants with similar key characteristics (Friedman et al., 2015). To exclude the risk that participants in the control group had exposure to the DirectSupport College of Employment Services (CES)—a component of the intervention potentially available to anyone on the Internet—all organizations whose employment consultants reported exposure to the CES lessons before baseline were assigned to the intervention group by default and matched with similar organizations in the control group. As a result of the random assignment described above, 107 employment consultants were assigned to the intervention group and 80 to the control group.

2.3.3 Intervention

The intervention activities lasted one year—from June 1, 2017 to May 30, 2018—and were organized around tracking the employment consultants’ implementation of employment supports through a daily survey, promoting reflection and change through data-enabled performance feedback, and delivering content about standards of effective employment supports. The performance feedback and content delivery were focused on the standards of effective employment supports described in the introduction, including 1) building trust with job seekers and their families, 2) getting to know job seekers, 3) supports planning, 4) finding tasks/jobs, and 5) providing supports after hire. Promoting reflection and behavior change was pursued through the following activities:

Community of practice

Online learning

Distance mentoring

Micro-learning

Project website

Community of practice (CoP): The monthly CoP (Lave & Wenger, 1991; Webber, 2016; Wenger et al., 2002) was organized around sharing data from the daily survey, checking successes and challenges in accomplishing the goal set the month before, setting a new goal for the upcoming month, reviewing standards of effective employment supports, and asking the employment consultants to share their stories. The CoP was held using an online platform—Adobe Connect—that allows participants to interact with each other and with the facilitators by speaking, typing comments in a chat pod, or responding to polls. Typically, 8–10 employment consultants attended each CoP. Overall, 36 employment consultants attended at least one CoP throughout the one-year intervention, whereas 5 employment consultants attended at least 50% of the events. The main reason reported for the low attendance was lack of time. As an incentive to join the community of practice, the attendees received a certificate of attendance valid for use toward professional certification. To reach participants who were unable to attend the event live, we distributed a written summary of the CoP and the slides after each event. About 68% of the employment consultants reported that they read these summaries.

Online learning: The participants in the intervention group had free access to 15 video lessons from the online DirectCourse College of Employment Service (CES; These lessons addressed standards of effective employment supports including, for example, getting to know job seekers, finding/customizing jobs, addressing work incentives, and facilitating natural supports. Each lesson lasted about 45 minutes, included a pretest and posttest, and could be watched at any time convenient for the participant. A total of six employment consultants completed all 15 lessons, 4 consultants completed some lessons, and the remaining did not complete any lesson.

Distance mentoring: During the fall of 2017 and early in 2018, two training specialists reached out to employment consultants who in the quarterly survey expressed an interest in discussing their employment support strategies. The training specialists contacted 47 interested participants over the phone, and were able to talk with 17 of them and schedule nine follow-up appointments. Topics discussed ranged from engaging employers in rural communities to facilitating participation of family members in the employment process.

Micro-learning: This activity consisted of tips, quotes, short videos and articles, and summary daily survey data (charts) shared in the closing screen of the daily survey. The content—replaced every three days—was aligned with the monthly topic and goal featured at the most recent CoP. The purpose of this intervention activity was to remind and nudge the employment consultants to implement the goal discussed at the most recent CoP as well as to provide easy access to know-how. An advantage of micro-learning is that it allows users to access knowledge in small chunks, when they have a few minutes in between meetings or while waiting for an appointment, without the need to block out a large amount of time (Buchem & Hamelmann, 2010; Coccoli et al., 2011; Omer, 2018). The first example in Figure 3 shows a closing screen with a tip about the importance of job negotiation in combination with a short video describing an example of job negotiation. The second example shows a pie chart based on data from the daily survey documenting how many employment consultants had been able to implement the goal of facilitating natural support set during a recent CoP. A total of 88% of participants in the intervention group reported they read the closing screens.

Fig. 3

The closing screen on the left provides this tip: “Ask employers: What tasks in your business are sometimes left unfinished”? It also share a link to a video about “Maggie’s negotiate job”. The pie chart in the closing screen on the right shows that 44% of the employment consultants accomplished the goal of increasing natural support after hire and 36% said that they will try.

Project website: Finally, a project website acted as a hub for participants to stay current with updates, a calendar of events, project activities, resources, slides and summaries of the community of practices, a dashboard with live data from the surveys collected, key literature about effective employment supports, and general information about the research project and the research team.

2.4 Data analysis

Descriptive analysis was carried out to explore the data and prepare the dataset for analysis. To test the null hypothesis of no effects of the intervention, we computed the change in employment outcomes reported by each individual employment consultant one year after baseline and then we compared the average change across the intervention and control groups. The following outcome variables were examined:

Change in the number of job seekers who gained paid individual employment

Change in hourly earnings

Change in weekly hours

Change in the length of time from the beginning of job search to hire

Earnings, work hours, and time to hire were based on data regarding the most recent hire reported by each employment consultant at baseline and one year after baseline. One-way ANOVA was run to test for statistical significance of the changes.

3 Results

The findings showed increased weekly work hours, hourly earnings, and reduced time to hire reported by the intervention group, compared to the control group. However, only the weekly work hours improvement was statistically significant. The change in the number of hires was negligible (Table 4). Note that the total number of hires included job seekers with intellectual and developmental disabilities as well as those with any other types of disabilities.

Table 4
Changes in Outcomes After Intervention

Baseline Average Change Min Change Max Change St. Dev Change N I vs. C Change Cohen’s Effect Size d

Hires (any disability)

Intervention 8 0.9 –19.0 36.0 8.4 59 0.1 0.01 (Negligible)

Control 11 0.8 –24.0 20.0 9.0 60

Hours

Intervention 16 3.5 –19.0 36.0 12.7 39 5.9^* 0.48 (Small)

Control 19 –2.4 –40.0 18.0 11.4 46

Earnings

Intervention $10 $1.09 –$2.00 $15.00 $2.71 39 $0.59 0.25 (Small)

Control $10 $0.50 –$4.50 $7.00 $2.04 48

Months to Hire

Intervention 6 –2.3 –42 11 9 38 –2.2 0.21 (Small)

Control 6 –0.1 –45 36 12 46

	Baseline	Average Change	Min Change	Max Change	St. Dev Change	N	I vs. C Change	Cohen’s Effect Size d
Hires (any disability)
Intervention	8	0.9	–19.0	36.0	8.4	59	0.1	0.01 (Negligible)
Control	11	0.8	–24.0	20.0	9.0	60
Hours
Intervention	16	3.5	–19.0	36.0	12.7	39	5.9^*	0.48 (Small)
Control	19	–2.4	–40.0	18.0	11.4	46
Earnings
Intervention	$10	$1.09	–$2.00	$15.00	$2.71	39	$0.59	0.25 (Small)
Control	$10	$0.50	–$4.50	$7.00	$2.04	48
Months to Hire
Intervention	6	–2.3	–42	11	9	38	–2.2	0.21 (Small)
Control	6	–0.1	–45	36	12	46

^*P value < 0.05 two tails.

3.1 Hires

On average, the employment consultants in the intervention group reported an increase of 0.9 hires from eight at baseline (95% CI = –1.3, 3), whereas the control group reported an increase of 0.8 hires from 11 at baseline (95% CI = –1.5, 3.0). Therefore, the participants in the intervention group outperformed their peers in the control group by 0.1 more hires on average. However, the effect size of this difference was negligible and not statistically significant.

The number of hires at baseline and one year after baseline and the change in the number of hires over the course of the year ranged widely in both the intervention and control groups: from a decrease of 24 hires as reported by a participant in the control group to an increase of 36 hires as reported by a participant in the intervention group (Fig. 4).

Fig. 4

The two bar charts in this figure show that employment consultants in both the intervention and in the control group reported a number of hires ranging widely from just a few hires to over 30–40 hires. The bar charts show also that the change in hires from baseline to one year after baseline varied widely across the two groups.

3.2 Work hours

The intervention group reported an average increase of 3.5 weekly work hours for job seekers from 16 hours at baseline (95% CI = –0.4, 7.5), whereas the corresponding figure for the control group was a decrease of 2.4 hours from 19 hours at baseline (95% CI = –5.7, 0.9). Therefore, the intervention group outperformed the control group by 5.9 more weekly work hours on average (Cohen’s Effect Size d 0.48; p = 0.027, two tails). Again here, the change in job seekers’ work hours over the course of the study ranged widely in both the intervention and control groups: from a decrease of 40 weekly hours as reported by a participant in the control group to an increase of 36 weekly hours as reported by a participant in the intervention group.

3.3 Earnings

The intervention group reported an increase of $1.09 per hour from $10 at baseline (95% CI = $0.24, $1.94), whereas the control group reported an increase of only $0.49 per hour from $10 at baseline (95% CI = –$0.08, $1.07). Therefore, the intervention group outperformed the control group by reporting $0.59 greater increase in hourly earnings on average (Cohen’s Effect Size d = 0.25). However, this difference was not statistically significant. The lack of statistical significance could be the result of a wide variation in the change in earnings—from a decrease of –$4.50 to an increase of $15.00—resulting in low statistical power for the achieved effect size (1–β err prob = 31%).

3.4 Time to hire

The intervention group reported 2.3 fewer months required from job search to hire, from six months as reported at baseline (95% CI = –5.1, 0.5), whereas the control group reported a decrease of only 0.1 months, also from six months at baseline (95% CI = –3.4, 3.3). Therefore, the intervention group outperformed the control group by reporting 2.2 fewer months from job search to hire on average (Cohen’s Effect Size d = 0.21). However, the difference was not statistically significant. Also in this case the lack of statistical significance could be the result of a wide variation in the change in time from job search to hire—from a decrease of 45 months to an increase of 36 months—resulting in low statistical power for the achieved effect size (1–β err prob = 24%).

3.5 Participants’ qualitative feedback

Several employment consultants reported that the intervention activities helped them to reflect on and be more intentional about how they invested their time. Below are examples of comments from the participants:

“ ... I like this survey, it keeps me thinking about how I’m spending each day and reminds me of my clients progress towards placement and a successful closure ... ”

“ ... I find that it causes me to pause for a moment and reflect on how I am spending my time, energy and resources ... ”

“ ... The graphic and analytical feedback works well to help hone employment strategies to improve results for individuals and for leveraging employer opportunity in various business environments ... ”

4 Discussion

This study tested the hypothesis that employment consultants who received a package of data-enabled performance feedback and guidance would report improved employment outcomes of job seekers compared to a control group who continued with business as usual. One year after the start of the intervention, weekly work hours, hourly earnings, and months to hire improved, although only the change in work hours was statistically significant. There was no meaningful change in the number of hires.

Despite the lack of robust statistically significant results, several employment consultants commented that the data-enabled performance feedback and guidance were useful for helping them to reflect on their support strategies and implement effective employment supports. Moreover, this study provides insights in further understanding employment support activities, outcomes, and ways for improving future research. Some of the challenges that may have contributed to the limited significant outcomes are discussed next.

4.1 Fidelity of implementation

While engagement in completing the daily survey was high, employment consultants’ attendance at the monthly community of practice and online learning—key elements of the intervention—was limited. Competing priorities in their work schedule were the major reason the employment consultants reported for not taking advantage of these elements of the intervention. Some employment consultants played multiple roles in their organizations, doing both job development and job coaching after hire, while others were also responsible for supervising. Also, findings from the daily survey suggested that employment consultants spent substantial time in administrative activities, limiting their availability for participation in the CoP and watching the College of Employment Services video lessons. This is consistent with the literature documenting that making time for learning is the number one challenge of the modern workforce (Bersin, 2019; Spar et al., 2018). Involving managers and supervisors in the study could have helped in supporting their employment consultants to make time for participating in the learning opportunities (Grol & Wensing, 2004; Salas et al., 2012).

4.2 Complexity of change

Employment consultants function in a complex system, and are influenced by multiple organizational and community factors (Grol & Wensing, 2004; Horvath, 2013; IHI, 2003). Delivering learning experiences may not be enough if the target participants are not ready for change or if the organization they function in does not support change (IHI, 2003; Kotter, 1995; Prochaska et al., 1992). External forces such as policy and funding mechanisms may prevail in determining the behavior and practices implemented by human services agencies and their staff (Deming, 1986; NCD, 2018; Winsor et al., 2018). For example, employment consultants may be less likely to promote natural support after hire—a standard of effective employment supports—if their programs are paid for the amount of time that employment consultants spend providing job coaching after hire (NCD, 2018).

4.3 Length of the study

Even if the intervention activities were effective, the effects on behavior and on outcomes might not have been fully captured because it may take time for participants to understand and respond to an intervention and for the effects of change to manifest (Ashbaugh, 2008; Baker et al., 2003; Heath & Heath, 2010; Kotter, 1995). For example, it is plausible that it took a few months before the employment consultants started to absorb the message of change and were able to act on it by modifying their support strategies (Rogers, 2003). On average, job seekers took six months from job search to hire. Therefore, it is possible that job seekers who applied for services in the second semester during our intervention generated outcomes that were not measurable until after our study ended (Bond et al., 2008; Drake et al., 2012; Friedman et al., 2015).

4.4 Economic environment

Finally, the intervention was carried out during a time of historically low unemployment rates of 3.8% by the end of the intervention (US Bureau of Labor Statistics, 2019). It is plausible that in this economic environment, finding jobs was fundamentally easy, even for employment consultants who had minimal job development skills and who, like the participants in the control group, were not exposed to any intervention activities. While a strong economy is always desirable, testing the effectiveness of interventions aimed at improving employment consultants’ effectiveness might yield more distinct outcomes during times of high unemployment rates, when finding jobs is harder.

4.5 Limitations and strengths

Low attendance in the community of practice and limited engagement with the online learning undermined the strength of the intervention. Moreover, the one-year duration of the intervention may have reduced the ability of this study to capture the effects of the intervention that manifested after that time frame. Also, some important quality outcomes including job satisfaction, inclusion in the workplace, and job retention could not be documented. Finally, despite randomly assigning the employment consultants’ organizations to intervention and control groups, the sample sizes and the characteristics of the employment consultants at baseline were not consistent across the intervention and control groups and may have played a role in the findings.

This study has also strengths. The retention rate and the daily survey response rates were relatively high, which increased the quality of the data collected. Moreover, despite findings that did not fully support the hypotheses of this study, feedback from participants was very positive about the usefulness of the daily survey as a reminder to be more intentional in implementing standards of effective employment supports. Finally, by using an experimental design—one of the most robust designs for evidence-based quantitative research—this study contributes to pushing forward the agenda for improving the effectiveness of employment supports available to job seekers.

4.6 Recommendations for policy, practice, and research

Based on positive feedback from several participants, we recommend that policy makers promote policies that supports employment programs to use modern technologies to track implementation of standards of effective employment supports and use the data to deliver guidance to the employment consultants. For example, electronic data management systems—often used by employment programs for billable and compliance purposes—should be leveraged for tracking the implementation of effective standards of employment supports described in the literature. Feedback could be provided in the form of dashboards displaying the implementation of standards of effective employment supports for employment consultants and their supervisors to use as guidance in their work with job seekers. Feedback could be paired with personalized micro-learning resources including tips, short videos, or short articles that address specific know-how needs of employment consultants at the moment when they need it.

We recommend that experimental research be implemented to further validate best strategies for supporting employment consultants through providing data-enabled performance feedback and micro-learning personalized resources. Moreover, research should also expand to tracking job seekers’ outcomes such as job satisfaction, career opportunities, and social connectedness in the workplace. Given the complex interplay between the organizational structures of employment programs, the services used by job seekers, and the employment outcomes experienced by job seekers, it will be important that research involves both the organizations’ management and the job seekers.

5 Conclusion

Whether or not the intervention activities delivered in this study were effective, the main conclusion remains that employment consultants may benefit from data-enabled performance feedback and guidance. Since employment consultants play an important role in determining the employment successes of job seekers, maximizing supports to employment consultants using modern technologies is central for increasing job seekers’ chances to achieve their employment goals.

Conflict of interest

The authors declare that they have no conflict of interest.

Footnotes

Acknowledgments

The development of this manuscript was supported in part by Grant #90RT5028 and grant # 90RT5019, National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR), Administration for Community Living (ACL), U.S. Department of Health and Human Services (HHS). The content of this presentation does not necessarily represent the policy of NIDILRR, ACL or HHS.

The authors would like to acknowledge the contribution of Paul Foos for developing the daily survey application, Mark Hutchinson for technical support, and Anya Weber for copyediting.

References

Akkerman,

, Kef,

, & Meininger,

H. P.

(2018). Job satisfaction of people with intellectual disabilities: The role of basic psychological need fulfillment and workplace participation. Disability and Rehabilitation40(10), 1192–1199. https://doi.org/10.1080/09638288.2017.1294205

Amabile,

, & Kramer,

(2011). The progress principle: Using small wins to ignite joy, engagement, and creativity at work. Harvard Business School Publishing.

APSE. (2019). Certified supported employment professional: 2019 certification handbook. Author.

Ashbaugh,

(2008). Managing system change in human service agencies. Intellectual and Developmental Disabilities46(6), 480–483.

Association of Community Rehabilitation Educators (ACRE). (2018) ACRE competencies with an emphasis on customized employment. Author.

Association of Community Rehabilitation Educators (ACRE). (2013). 2013 ACRE competencies: Employment services. Author.

Bajpai,

, Jilla,

, Tiwari,

V. N.

, Venkatesan,

S. M.

, & Narayanan,

(2015). Quantifiable fitness tracking using wearable devices. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Milan, Italy. https://doi.org/10.1109/EMBC.2015.7318688

Baker,

, Reddish,

, Robertson,

, Hearnshaw,

, & Jones,

(2001). Randomised controlled trial of tailored strategies to implement guidelines for the management of patients with depression in general practice. British Journal of General Practice51(470), 737–741.

Barrows,

, Billehus,

, Briton,

, Cohen-Hall,

, Huerena,

, LeBlanc,

, McVay,

, & Topper,

(2016). The truth comes from us: Supporting workers with developmental disabilities. University of Massachusetts Boston, Institute for Community Inclusion. Retrieved from http://www.thinkwork.org/sites/thinkwork.org/files/files/ThinkWork_sabe_14_F.pdf

10.

Bersin,

(2019). Employee engagement 3.0: From feedback to action. Retrieved from https://www.linkedin.com/pulse/employee-engagement-from-feedback-action-josh-bersin

11.

Bhattacharyya,

, Reeves,

, & Zwarenstein,

(2009). What is implementation research? Rationale, concepts, and practices. Research on Social Work Practice19(5), 491–502. https://doi.org/10.1177/1049731509335528

12.

Blacher,

, Kraemer,

, & Howell,

(2010). Family expectations and transition experiences for young adults with severe disabilities: Does syndrome matter?Advances in Mental Health and Learning Disabilities4(1), 3–16. https://doi.org/10.5042/amhld.2010.0052

13.

Bolles,

R. N.

(2018). What color is your parachute? A practical manual for job-hunters and career-changers paperback. Ten Speed Press.

14.

Bond,

, R., Campbell,

, & Drake,

R. E.

(2012). Standardizing measures in four domains of employment outcome for individual placement and support. Psychiatric Services63, 751–757. https://doi.org/10.1176/appi.ps.201100270

15.

Bond,

G. R.

, McHugo,

G. J.

, Becker,

D. R.

, Rapp,

C. A.

, & Whittey,

(2008). Fidelity of supported employment: Lessons learned from the National Evidence-Based Practice Project. Psychiatric Rehabilitation Journal31(4), 300–305. https://doi.org/10.2975/31.4.2008.300.305

16.

Buchem,

, & Hamelmann,

(2010). Microlearning: A strategy for ongoing professional development. eLearning Papers21, 1–15.

17.

Butterworth,

, Hiersteiner,

, Engler,

, Bershadsky,

, & Bradley,

(2015). National Core Indicators: Data on the current state of employment of adults with IDD and suggestions for policy development. Journal of Vocational Rehabilitation42(3), 209–220. https://doi.org/10.3233/JVR-150741

18.

Callahan,

, Shumpert,

, & Condon,

(2009). Discovery: Charting the course to employment. Translating the activities of everyday lives into possibilities for employment. Marc Gold & Associates.

19.

Carpenter,

, & Daly,

(2019). Creating an employment collaborative. Journal of Vocational Rehabilitation50(3), 331–338. https://doi.org/10.3233/JVR-191015

20.

Coccoli,

, Mercurio,

, Torre,

, & Torsani,

(2011). Design of a framework for personalized micro learning. In IADIS International Conference e-Learning 2011 (pp. 171-175). Retrieved from https://www.scopus.com/inward/record.uri?eid=2-s2.0-84865091284&partnerID=40&md5=9fb9ab5315afe13daa8d7aedd6b8b03b

21.

Cohen,

(1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates.

22.

Deming,

E. W.

(1986). Out of the crisis. MIT Press.

23.

Drake,

R. E.

, Bond,

G. R.

, & Becker,

D. R.

(2012). Individual placement and support: An evidence-based approach to supported employment. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199734016.001.0001

24.

Drake,

R. E.

, Bond,

G. R.

, & Essock,

S. M.

(2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin35(4), 704–713. https://doi.org/10.1093/schbul/sbp041

25.

Fixsen,

D. L.

, Naoom,

S. F.

, Blase,

K. A.

, Friedman,

R. M.

, & Wallace,

(2005). Implementation research: A synthesis of the literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network (FMHI Publication #231).

26.

Friedman,

L. M.

, Furberg,

C. D.

, Demets,

D. L.

, Reboussin,

D. M.

, & Granger,

C. B.

(2015). Fundamentals of clinical trials (5th ed.). Springer. https://doi.org/10.1007/978-1-4419-1586-3

27.

Friedman,

, & Rizzolo,

M. C.

(2016). The state of transportation for people with intellectual and developmental disabilities in Medicaid Home and Community-Based Services 1915(c) Waivers. Journal of Disability Policy Studies27(3), 168–177. https://doi.org/10.1177/1044207316644413

28.

Gilbride,

, & Stensrud,

(2008). Why won’t they just do it?Rehabilitation Education22(2), 125–132. https://doi.org/10.1891/088970108805059462

29.

Gilson,

C. B.

, Carter,

E. W.

, Bumble,

J. L.

, & McMillan,

E. D.

(2018). Family perspectives on integrated employment for adults with intellectual and developmental disabilities. Research and Practice for Persons with Severe Disabilities43(1), 20–37. https://doi.org/10.1177/1540796917751134

30.

Graham,

, Inge,

, Wehman,

, Murphy,

, Revell,

W. G.

, & West,

(2013). Moving employment research into practice: Knowledge and application of evidence-based practices by state vocational rehabilitation agency staff. Journal of Vocational Rehabilitation39, 75–81. https://doi.org/10.3233/JVR-130643

31.

Griffin,

, Hammis,

, & Geary,

(2007). The job developer’s handbook: Practical tactics for customized employment. Brookes Publishing.

32.

Grol,

, & Grimshaw,

(2003). From best evidence to best practice: Effective implementation of change in patients’ care. Lancet362, 1225–1230. https://doi.org/10.1016/S0140-6736(03)14546-1

33.

Grol,

, & Wensing,

(2004). What drives change? Barriers to and incentives for achieving evidence-based practice. Medical Journal of Australia180(6), S57–S60. https://doi.org/10.5694/j.1326-5377.2004.tb05948.x

34.

Haines,

, Domin,

, & Butterworth,

(2013). The 2010-2011 national survey of community rehabilitation providers: Estimating the number of community rehabilitation providers in the United States. Research to Practice Brief 53. University of Massachusetts Boston, Institute for Community Inclusion.

35.

Harvey,

, Szoc,

, Dela Rosa,

, Pohl,

, & Jenkins,

(2013). Understanding the competencies needed to customize jobs: A competency model for customized employment. Journal of Vocational Rehabilitation38(2), 77–89. http://doi.org/10.3233/JVR-130622

36.

Haveman,

, Tillmann,

, Stöppler,

, Kvas,

Š.

, & Monninger,

(2013). Mobility and public transport use abilities of children and young adults with intellectual disabilities: Results from the 3-year Nordhorn Public Transportation Intervention Study. Journal of Policy and Practice in Intellectual Disabilities10, 289–299. https://doi.org/10.1111/jppi.12059

37.

Heath,

, & Heath,

(2010). Switch: How to change things when change is hard. Broadway Books.

38.

Hiersteiner,

, Butterworth, , Bershadsky,

, & Bonardi,

(2018). Working in the community: The status and outcomes of people with intellectual and developmental disabilities in integrated employment—Update 3. NCI Data Brief, April 2016. Human Services Research Institute.

39.

Hoff,

, Gandolfo,

, Gold,

, & Jordan,

(2000). Demystifying job development: Field-based approaches to job development for people with disabilities. Training Resource Network, Inc.

40.

Horvath,

A. R.

(2013). From evidence to best practice in laboratory medicine. Clinical Biochemist Reviews34(2), 47–60.

41.

Institute for Healthcare Improvement (IHI). (2003). The breakthrough series: IHI’s collaborative model for achieving breakthrough improvement. IHI Innovation Serieswhite paper. Author.

42.

Inge,

K. J.

, Graham,

C. W.

, Erickson,

, Sima,

, West,

, & Cimera,

R. E.

(2016). Improving the employment outcomes of individuals with traumatic brain injury: The effectiveness of knowledge translation strategies to impact the use of evidence-based practices by vocational rehabilitation counselors. Journal of Vocational Rehabilitation45(1), 107–115. https://doi.org/10.3233/JVR-160815

43.

Jones,

J. L.

, & Gallus,

K. L.

(2016). Understanding deinstitutionalization: What families value and desire in the transition to community living. Research and Practice for Persons with Severe Disabilities41(2), 116–131. https://doi.org/10.1177/1540796916637050

44.

Lave,

, & Wenger,

(1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.

45.

Levinson,

J. C.

, & Perry,

(2011). Guerrilla marketing for job hunters: 400 unconventional tips, tricks, and tactics for landing your dream job. John Wiley & Sons Ltd.

46.

Luecking,

D. M.

, & Luecking,

R. G.

(2015). Translating research into a seamless transition model. Career Development and Transition for Exceptional Individuals38(1), 4–13. https://doi.org/10.1177/2165143413508978

47.

Luecking,

R. G.

(2008). Emerging employer views of people with disabilities and the future of job development. Journal of Vocational Rehabilitation29(1), 3–13.

48.

Kotter,

(1995). Leading change: Why transformation efforts fail. Harvard Business Review. Retrieved from https://hbr.org/1995/05/leading-change-why-transformation-efforts-fail-2

49.

Mank,

, Cioffi,

, & Yovanoff,

(1999). Impact of coworker involvement with supported employees on wage and integration outcomes. Mental Retardation37(5), 383–394. https://doi.org/10.1352/0047-6765(1999)037<0383:IOCIWS>2.0.CO;2

50.

Migliore,

, Butterworth,

, Lyons,

, Nye-Lengerman,

, & Foos,

(2018). Supporting employment consultants in their work with job seekers: A longitudinal study. Journal of Vocational Rehabilitation49(3), 273–286. https://doi.org/10.3233/JVR-180973

51.

Migliore,

, Hall,

A. C.

, Butterworth,

, & Winsor,

(2010). What do employment specialists really do? A study on job placement practices. Research & Practice for Persons with Severe Disabilities35(1-2), 15–23. https://doi.org/10.2511/rpsd.35.1-2.15

52.

Migliore,

, Mank,

, Grossi,

, & Rogan,

(2007). Integrated employment or sheltered workshops: Preferences of adults with intellectual disabilities, their families, and staff. Journal of Vocational Rehabilitation26(1), 5–19.

53.

Migliore,

, Nye-Lengerman,

, Lyons,

, Bose,

, & Butterworth,

(2018). A model of employment supports for job seekers with intellectual disabilities. Journal of Rehabilitation84(2), 3–13.

54.

National Council on Disability (NCD). (2018). National disability employment policy, from the New Deal to the real deal: Joining the industries of the future. Retrieved from https://ncd.gov/sites/default/files/New%20Deal%20to%20Real%20Deal%20FINAL_508.PDF

55.

National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR). (2017). 2017-2022: Long-range plan for the National Institute on Disability, Independent Living, and Rehabilitation Research. Retrieved from https://www.acl.gov/sites/default/files/news%202017-05/NIDILRR-Long-Range-Plan-DRAFT.pdf

56.

Novak,

, Parent-Johnson,

, Owens,

, & Keul,

(2014). National certification initiative for employment support professionals: Promoting quality integrated employment services. Journal of Vocational Rehabilitation40(2), 99–107. https://doi.org/10.3233/JVR-140679

57.

Nye-Lengerman,

, & Nord,

(2016). Changing the message: Employment as a means out of poverty. Journal of Vocational Rehabilitation44(3), 243–247. https://doi.org/10.3233/JVR-160794

58.

Omer,

(2018). 4 reasons why you should convert long eLearning courses to microlearning. Retrieved from https://elearningindustry.com/convert-long-elearning-courses-to-microlearning-4-reasons

59.

Petner-Arrey,

, Howell-Moneta,

, & Lysaght,

(2016). Facilitating employment opportunities for adults with intellectual and developmental disability through parents and social networks. Disability Rehabilitation38(8), 789–795.

60.

Phillips,

W. L.

, Callahan,

, Shumpert,

, Puckett,

, Petrey,

, Summers,

, & Phillips,

(2009). Customized transitions: Discovering the best in us. Journal of Vocational Rehabilitation30(1), 49–55. https://doi.org/10.3233/JVR-2009-0452

61.

Prochaska,

J. O.

, DiClemente,

C. C.

, & Norcross,

J. C.

(1992). In search of how people change: Applications to addictive behaviors. American Psychologist9, 1102–1114. https://doi.org/10.1037//0003-066x.47.9.1102

62.

Riesen,

, Morgan,

R. L.

, & Griffin,

(2015). Customized employment: A review of the literature. Journal of Vocational Rehabilitation43(3), 183–193. http://doi.org/10.3233/JVR-150768

63.

Rogers,

M. E.

(2003). Diffusion of innovations (5th ed.). Free Press.

64.

Salas,

, Tannenbaum,

S. I.

, Kraiger,

, & Smith-Jentsch,

K. A.

(2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest13(2), 74–101. https://doi.org/10.1177/1529100612436661

65.

Seifert,

, Schlomann,

, Rietz,

, & Schelling,

H. R.

(2017). The use of mobile devices for physical activity tracking in older adults’ everyday life. Digital Health, 3. https://doi.org/10.1177/2055207617740088

66.

Shiffman,

, Stone,

A. A.

, & Hufford,

M. R.

(2008). Ecological momentary assessment. Annual Review Clinical Psychology4, 1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415

67.

Shogren,

K. A.

, Gotto

IV, G. S.

, Wehmeyer,

M. L.

, Shaw,

, Seo,

, Palmer,

, Snyder,

M. J.

, & Barton,

K. N.

(2016). The impact of the self-determined career development model on self-determination. Journal of Vocational Rehabilitation45(3), 337–350. http://doi.org/10.3233/JVR-160834

68.

Shojania,

K. G.

, & Grimshaw,

J. M.

(2005). Evidence-based quality improvement: The state of the science. Health Affairs24(1), 138–150. https://doi.org/10.1377/hlthaff.24.1.138

69.

Smull,

M. W.

, & Sanderson,

(2009). Essential lifestyle planning for everyone. The Learning Community.

70.

Spar,

, Lefkowitz,

, Dye,

, & Pate,

(2018). Workplace learning report. Learning Solutions. Retrieved from https://www.cornerstoneondemand.com/sites/default/files/partner/asset/files/linkedin-learning-workplace-learning-report-2018.pdf

71.

Sudsawad,

(2007). Knowledge translation: Introduction to models, strategies, and measures. Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research. Retrieved from http://ktdrr.org/ktlibrary/articles_pubs/ktmodels/

72.

U.S. Bureau of Labor Statistics. (2019). Labor force statistics from the current population survey. Retrieved from https://data.bls.gov/timeseries/LNS14000000

73.

Walz,

L. C.

, Nauta,

M. H.

, & Aan Het Rot,

(2014). Experience sampling and ecological momentary assessment for studying the daily lives of patients with anxiety disorders: A systematic review. Journal of Anxiety Disorders28(8), 925–937. https://doi.org/10.1016/j.janxdis.2014.09.022

74.

Webber,

(2016). Building successful communities of practice. Self-published on amazon.com.

75.

Wehman,

, Brooke,

A. M.

, Ham,

, Schall,

, McDonough,

, Lau,

S. J.

, Seward,

, & Avellone,

(2016). Employment for adults with autism spectrum disorders: A retrospective review of a customized employment approach, 61–72. http://doi.org/10.1016/j.ridd.2016.01.015

76.

Wehman,

, Lau,

, Molinelli,

, Brooke,

, Thompson,

, Moore,

, & West,

M. D.

(2012). Supported employment for young adults with autism spectrum disorder: Preliminary data. Research and Practice for Persons with Severe Disabilities37, 160–169.

77.

Wehman,

, Taylor,

, Brooke,

, Avellone,

, Whittenburg,

, Ham,

, Brooke,

A. M.

, & Carr,

(2018). Toward competitive employment for persons with intellectual and developmental disabilities: What progress have we made and where do we need to go. Research and Practice for Persons with Severe Disabilities43(3), 131–144. https://doi.org/10.1177/1540796918777730

78.

Wehman,

P. H.

, Schall,

C. M.

, McDonough,

, Kregel,

, Brooke,

, Molinelli,

, Ham,

, Graham,

C. W.

, Riehle,

J. E.

, Collins,

H. T.

, & Thiss,

(2014). Competitive employment for youth with autism spectrum disorders: Early results from a randomized clinical trial. Journal of Autism and Developmental Disorders44(3), 487–500. https://doi.org/10.1007/s10803-013-1892-x

79.

Wenger,

, McDermott,

, & Snyder,

(2002). Cultivating communities of practice. Harvard Business Press.

80.

Wehmeyer,

M. L.

(2011). What is next for the transition and employment supports movement?Journal of Vocational Rehabilitation35(3), 153–156. https://doi.org/10.3233/JVR-2011-0563

81.

Winsor,

, Landa,

, Hall,

, & Narby,

(2018). Pushing the integrated employment agenda: A case study of Maryland’s high-performing employment system. Issue 16. University of Massachusetts Boston, Institute for Community Inclusion.