Abstract
The discovery fidelity scale (DFS) is a 15-item instrument used to determine fidelity to systems and services level discovery best practices. Fidelity scale development is typically guided by an iterative, three-part process that includes identifying and specifying the fidelity structure and content, measuring and confirming the content, and assessing the internal consistency and reliability of the scale. This study is the initiation of the third step of the fidelity process to determine if items on the DFS accurately and reliably measure the discovery construct (internal consistency). The study also examined outcomes for individuals engaged in the discovery process. The results of the study suggest that both the systems and services components appear to measure their respective constructs and the overall discovery construct appears to have acceptable internal consistency.
The Rehabilitation Act, as amended in the Workforce Innovation and Opportunity Act (WIOA, 2014), contained specific provisions to mitigate the disproportionately poor employment opportunities and outcomes for people with significant disabilities. One such provision modified the definition of supported employment (SE) to include customized employment (CE) and now state vocational rehabilitation agencies can offer CE as an employment service. The statute outlines specific strategies for implementing CE, including (a) exploring jobs with the individual; (b) working with employers to facilitate placement, including customizing a job description based on current employer needs or on previously unidentified and unmet employer needs; (c) developing a set of job duties, a work schedule, and job arrangement, along with specifics of supervision (including a performance evaluation review), and determining a job location; (d) representing by a professional chosen by the individual, or self-representation of the individual in working with an employer to facilitate placement; and (e) providing services and supports at the job placement (29 U.S.C § 705 et seq.).
While CE has shown promise as a strategy that leads to improved employment outcomes for people with significant disabilities, the extant research on CE practices has not met requisite standards to be considered an evidence-based practice (Leahy et al., 2018). The paucity of research establishing the operationalized evidence for effective and consistent implementation of CE was highlighted in a literature review by Riesen et al. (2015). The results of this review revealed the majority of research on CE is descriptive in nature and at the time of the review, there were no group experimental/quasi experimental studies. Recent calls for rehabilitation practices to be evidence-based (Huber et al., 2017; Wehman et al., 2018) require a more concerted effort to systematically measure the degree to which interrelated CE processes, such as discovery, align with operational, reliable, and valid practices.
Customized employment discovery is psychosocial rehabilitation process used to determine an individual’s strengths, interests, skills and support needs to obtain and maintain customized employment. The discovery process includes interviews, observations, documentation review, and interactions with the employment seeker (Workforce Innovation Technical Assistance Center [WINTAC], 2017). Discovery also uses observations of the employment seeker engaged in familiar and less familiar activities and requires interviews with family members and other influential persons in the employment seeker’s life. This information is used to develop well-coordinated customized job development activities. Unfortunately, there is relatively little research outlining the specific components of the discovery process. One way to gain a better understanding of this process is to develop an objective fidelity measure that operationalizes and ensures adherence to validated systems and services discovery practices.
Fidelity scales include information about the structure and process of an intervention or model, such as the length and intensity of the program, the procedures and activities during the process, and the roles and qualifications of the program providers (Mowbray et al., 2003). Fidelity scales are generally designed using a multi-step process which includes (a) identifying and specifying fidelity structure and content, (b) measuring and confirming the content developed in step one, and (c) assessing whether the content has internal consistency and reliability (Bond & Drake, 2019; McGrew et al., 1994; Mowbray et al., 2003). Research suggests that programs with high scoring fidelity have better outcomes than their counterparts (Bond et al., 2011). Fidelity scales have been developed for other employment support models such as the individual placement and support model (IPS) for people with severe mental illness (Bond et al., 1997). The use of an IPS fidelity scale promoted standardized service implementation and provided guidelines for quality improvement for supported employment programs (Bond et al., 2011; Knaeps et al., 2012). In addition, fidelity scales promote sustainability of programs and guide the planning process for new programs (Kim et al., 2015).
The development and validation of a CE discovery fidelity scale is necessary to capture and delineate the essential practices and procedures of discovery. Moreover, a validated discovery fidelity scale can be used as an objective measure of performance to assist funding agencies and rehabilitation counselors to determine which community rehabilitation providers implement discovery according to best practices. The Discovery Fidelity Scale (DFS) was developed to measure fidelity to CE discovery best practices (Hall et al., 2018) and it was revised and updated based on the iterative process outlined in the literature (McGrew et al., 1994; Mowbray et al., 2003). First, Hall and colleagues conducted a number of focus groups and interviews with professionals to identify and specify the fidelity structure and refine the content. Second, to confirm the content, Riesen et al. (2019) used a three round, modified Delphi process to generate consensus about what experts believe are acceptable and not acceptable tenets of the DFS. The Delphi panel reviewed and rated the fidelity descriptors for discovery systems and services. The information obtained from the Delphi study was used to further refine the scale. The purpose of this study was to initiate the third step of the fidelity scale development process by examining if items on the DFS accurately and reliably measure the discovery construct. The study also examined outcomes for individuals engaged in the discovery process.
Method
Participants
We used nonrandom sampling to recruit and select participants to be DFS administrators for this study. The DFS administrator is responsible for examining discovery records and scoring these records using the DFS. The DFS administrator trainings were offered by Griffin-Hammis Associates, a consulting agency specializing in CE. The training consisted of a three-phase process designed to increase administrators’ scoring reliability of the DFS. Phase one consisted of small group trainings covering each of the scale’s fidelity tenets and scoring criteria. Phase two occurred about four months later when each of the administrators were likely to have at least some beginning experience using the scales on completed discovery. Phase two was an open discussion, question-and-answer session. The session included the same participants as phase one, however additional participants, such as the employment specialists or job developers that were directly responsible for the products being scored by the fidelity scale administrators also attended. Phase three consisted of 1:1 technical support with each DFS administrator after they had used the DFS to score a discovery work product. The first 1:1 technical support session typically resulted in 4 to 6 scoring changes to the DFS, with two or three subsequent 1:1 technical support meetings. By the fourth technical support session, with rare exceptions, all DFS administrators were authorized to use the fidelity scales without further 1:1 technical assistance.
Recruitment letters were distributed to vocational rehabilitation professionals (i.e., counselors and supervisors), intellectual and developmental disability agency case managers and support coordinators, and community rehabilitation providers, including employment service supervisors and managers who were trained to be DFS administrator in their respective state. When a DFS administrator contacted the first author to participate in the study, a letter of informed consent was sent to the DFS administrator to review and sign. Once the first author received a signed consent letter, the participant was provided with information about how to proceed with the study. Forty-one trained DFS administrators from four states (CA, IA, SD, and UT) returned a consent letter and agreed to participate in the study.
Procedures
When the DFS administrator returned the informed consent, the administrator was emailed information and directions about how to participate in the study. A secure, online, web platform, REDcap, was used to create the online data collection database. REDcap is a HIPPA-compliant data collection system that allows researchers to develop, link, and track responses from surveys across all study phases. Each DFS administrator was asked to complete data responses for demographics, DFS scoring, and customized employment seeker outcomes. The DFS administrator was automatically assigned an identification number that was used to track responses as the administrator entered data responses at different times during a 4-month timeframe. Responses were linked to each DFS administrator’s identification number. The first section was the DFS administrator demographic section which collected information about type of employer, position, years employed in the current position, education level, and types of training certification. After completing the first section, the DFS administrator received a link to the demographic section for the individual engaged in discovery. This section collected information about the employment seeker’s primary disability classification, age, and type of adult services.
After both demographic sections were completed, the DFS administrator received an online link to complete the DFS for the individual engaged in discovery. At that point, the DFS administrator reviewed and examined discovery documentation for the employment seeker. The discovery documentation is used to stage, structure, and detail components of the discovery process and includes narrative description of the discovery process. During each review, the DFS administrator entered corresponding DFS fidelity scores into the online, REDcap database. We sent the DFS administrator an outcome survey 90-days after completing the online DFS. The outcome survey asked a number of questions related to the process and outcomes of discovery, such as: (a) hours to complete discovery, (b) employment and wage status, (c) hours worked per week, (d) essential functions of the job, (e) elapsed time from discovery to employment, (f) type of business, (g) types of task, and (f) size of business.
DFS Instrumentation
Hall et al. (2018) developed the discovery fidelity scale (DFS) to measure adherence to CE discovery best practices. Hall and colleagues developed the DFS by reviewing literature on CE, conducting interviews and focus groups with practitioners, and conducting case studies and surveys with experts with demonstrated knowledge and experience in discovery implementation. Riesen et al. (2019) used the Delphi process to build consensus among CE experts about which fidelity descriptors on the DFS were considered acceptable and unacceptable. Adjustments to the DFS were made based on the results of the Delphi study. The final DFS consists of two subsections: discovery systems fidelity and discovery services fidelity. The systems section consists of five discovery system tenets and corresponding scaled fidelity descriptors. The services section consists of 10 discovery services tenets and corresponding scaled fidelity descriptors. The scaled fidelity descriptors for each of the systems and services tenets represent levels of fidelity to the discovery process for each respective tenet and are scored from: -1 unacceptable, +1 acceptable, +2 good, and +3 exemplary. For example, systems tenet 1.2, “Discovery is part of CE or SE,” has four anchored descriptors. The unacceptable descriptor (-1) states, “Discovery is used as an assessment to determine whether someone can work.” The acceptable descriptor (+1) states, “Discovery is authorized as part of a CE or SE referral for service.” The good descriptor (+2) states, “Discovery referrals are issued to the specific agency who will also deliver the person’s CE or SE services.” Finally, the exemplary descriptor (+3) states, “The employment specialist who provides discovery also provides the CE or SE employment service.” Both systems and services tenets are linked to specific components or factors on the scale. The discovery systems section has three components/factors: (a) authorization and access (two tenets), (b) financing (one tenet), and (c) discovery providers (two tenets). The discovery services section has five components/factors: (a) home and neighborhood visits (four tenets), (b) discovery activities (two tenets), (c) informational interviews (one tenet), (d) vocational profile narrative review (two tenets), and (e) employment plan (one tenet). The DFS scores are calculated for the systems and services sections as well as the combined overall score.
Reliability
Individual scoring reliability checks were conducted for 11 (64.7%) of the completed DFSs. During these reliability checks, DFS administrators reviewed the fidelity scores for each tenet and discussed the rationale for providing the score with the second author. Based on the discussion and review, the DFS administrator was provided the opportunity to complete a second DFS and change the initial fidelity score. The percentage of agreement between the first and second DFS was calculated at 76.6%.
Data Analysis
Descriptive statistics were calculated and summarized for DFS administrator demographics, job seeker demographics, and the 90-day outcome survey. Cronbach’s alpha for the five tenets in the systems section, the 10 tenets in the services section, and all 15 items of the DFS were calculated using SPSS software to determine internal consistency of the systems, services, and overall discovery constructs. The corrected item correlation was used to determine how well individual items correlate to the overall score of the systems, services sections, and the overall scale. Items with correlations less than .30 were flagged and considered for revision or deletion from the scale.
Results
A total of 14 DFS administrators from four states (CA, IA, SD, and UT) participated in the study. Six (42.85%) worked for state vocational rehabilitation, four (28.57%) worked for community rehabilitation providers, three (21.42%) worked for intellectual and developmental agencies, and one (7.14%) indicated “other.” Six DFS administrators (42.85%) indicated they were a vice president, director, or program manager, four (28.77%) were employment specialists, two (14.88%) were vocational rehabilitation counselors, one (7.14%) was a community resource specialist, and one (7.14%) was a team manager. In regard to years in current position, seven (52.09%) reported five or more years, two (14.28%) reported three to five years, four (28.57%) reported one to two years, and one (7.14%) reported less than a year at current position. Eight administrators (57.14%) had a master’s degree, four (28.57%) had a bachelor’s degree, and two (14.28%) had an associate’s degree. Finally, 10 administrators (71.42%) had a basic Association for Community Rehabilitation Educator (ACRE) certification with an emphasis in CE, 2 (14.28%) were a certified rehabilitation counselor (CRC), and 2 (14.28%) did not list any certifications.
The DFS administrators reviewed and rated 17 individual discovery records of employment seekers with disabilities engaged in the discovery process. The average age of the employment seeker was 24.4 years-old with a high of 45 and low of 22 years-old. Eight (47.05%) of the employment seekers had an intellectual/developmental disability, five (29.41%) had cerebral palsy, and four (23.52%) had autism. Nine (52.94%) of the employment seekers had an individualized plan for employment (IPE) as outlined by a state rehabilitation agency. Three (23.52%) had a Medicaid Home and Community-Based Services (HCBS) Waiver.
Table 1 reports Cronbach’s alpha for the five discovery systems tenets. Cronbach’s alpha indicated moderately acceptable internal consistency for the systems construct (α =.753). The preliminary results suggest that all discovery systems items should be retained. Table 2 reports Cronbach’s alpha for the ten discovery services tenets and the results indicate moderately acceptable internal consistency for most of service tenets (α =.798). Seven of the services items should be retained as the overall α would decrease if deleted. Three items, 2.2, 2.3, and 2.4, appear to be not sufficiently correlated to the services construct and the overall internal consistency of the services tenets would improve if these items were removed or revised. Table 3 reports Cronbach’s alpha for all of the 15 discovery systems and services tenets combined. The results indicate moderately acceptable internal consistency for the combined items (α =.765). When the two sections were combined, items, 1.1, 1.2, 2.2, 2.3, 2.4, and 2.5 reduced the overall internal consistency for the DFS construct. The overall inconsistency result was not unexcepted since items 1.1 through 1.5 measure the systems related construct and items 2.1 through 2.10 measure the services related construct.
Cronbach’s Alpha Item Analysis for Systems Fidelity Tenets (n = 17).
Note.
Cronbach’s Alpha Item Analysis for Services Fidelity Tenets (n = 17).
Note.
indicates correlation is less than .30 and item should be revised or deleted.
Cronbach’s Alpha Item Analysis all DFS Tenets (n = 17).
Note.
indicates correlation is less than .30 and item should be revised or deleted.
A number of 90-day outcome variables for the employment seeker with a disability were collected. First, the number of hours to complete the discovery process (intake to completed discovery) ranged from a low of 21.15 hrs, to a high of 140 hrs with a mean of 58.31 hrs. A total of six (35.3%) employment seekers who participated in the discovery process obtained a paid job. The mean wage for the these individuals was $11.71 per hour and the mean hours worked per week was 14.33 hr. Two of the employment seekers who obtained customized employment worked in food preparation and service occupations, one worked in business and financial occupations, one worked in architecture and engineering occupations, one worked in building and grounds cleaning and maintenance occupations, and one worked in sales or related occupations. Four individuals worked in businesses with 1-10 employees, and two worked in businesses with more than 50 employees.
Discussion
This preliminary study was a promising step in validating a fidelity scale that reliably outlines discovery systems and services practices. The results suggest that both the systems and services constructs have acceptable internal consistency. It appears that all of the systems level tenets had moderately high internal consistency. The systems level tenets are designed to determine fidelity for three primary components/factors related to improved discovery including authorization and access, financing, and discovery providers. Authorization and access fidelity ensures customized employment is considered for anyone eligible for customized employment. Financing fidelity ensures both the payment rate and financing methodology do not interfere with or diminish the thoroughness of the customized employment process. Discovery provider fidelity ensures customized employment service providers are trained, knowledgeable, and experienced with customized employment, beginning with the initial process of discovery.
The overall services construct had moderately acceptable internal consistency, however, there are several tenets that should be either removed or revised to improve consistency of this construct. Specifically, three items related to the home and neighborhood visits (tenets 2.2., 2.3., and 2.4) appear to be not sufficiently measuring fidelity to home and neighborhood visits. Tenet 2.2, “Uses a conversational style during interviews with the employment seeker, family, members, and others for information and understanding, not answers,” is designed to determine whether employment specialists gather information during the home visit through organic, evolving conversations with the employment seeker and family members versus the unacceptable practice of asking a standardized list of discrete questions. Because ratings of “acceptable” to “exemplary” on the other tenets in the home and neighborhood visit section are inherently incompatible with the type and nature of information gathered during a pro forma question-and-answer session conducted in the home, this tenet will be removed from the scale. Removing it also prevents ongoing validity issues arising from DFS administrators misinterpreting the term “conversational style” as meaning to literally speak/talk during the home visit.
Tenet 2.3, “Observe and learn about the employment seeker’s personal spaces during interviews and visits to the employment seeker’s home,” is designed to learn the history, background, and others involved with personal items or belongings of importance to the employment seeker. Based on the results, tenet 2.3 will be amended in future versions of the DFS to address the use and history of personal items of importance to the employment seeker that were found or revealed by the employment seeker during the home visit. Finally, tenet 2.4, “The employment specialist becomes familiar with the employment seeker’s neighborhood and surrounding area,” will also be amended by changing “neighborhood” to “community” as the interpretation of “neighborhood” varied from the few houses easily seen from the employment seeker’s home to a several mile area within a large city. The purpose of tenet 2.4 was to ensure the employment specialist was aware of the employment seeker’s larger community, in particular the kinds of business, commercial interests, and potential employers that may be located in reasonable proximity to the employment seeker’s home.
Combining the two subscales for services and systems provided some insights to internal consistency and the psychometric properties of the DFS. That is, when Cronbach’s alpha was calculated on the combined systems and services scores, the internal consistency of the scale remained in the moderately acceptable range. However, additional tenets with correlations less than .30 emerged (i.e., tenets 1.1,1.2, and 2.5). This finding indicates that the systems and services tenets are indeed measuring their respective constructs. As expected, combining the two subscales caused fluctuation with tenets that previously had acceptable internal consistency.
Moving beyond the internal consistency of the DFS, the study also provided preliminary outcome data on individuals engaged in the discovery process. On average it took 58.31 hr to complete the discovery process which is on the high end of current recommendations (Hall et al., 2018; WINTAC, 2017). This suggests that the employment specialist might need more training and guidance to effectively facilitate discovery and develop more experience to build fluency and efficiency while delivering the discovery service with fidelity to best practices. It may also suggest that the consistent delivery of discovery with fidelity to best practice typically takes longer than current recommendations indicated. It is worth noting that the goal of delivering discovery with fidelity to best practice, which includes evaluation of the efficiency of services, is not to decrease the amount of time it takes to deliver discovery services. Rather, the goal is to ensure the consistent use of best practice discovery. Gathering data related to the amount of time it takes novice and experienced employment specialists to deliver best practice discovery services is critical to establishing policies, processes, and funding mechanisms that fully and appropriately support fidelity implementation. In addition, critiques of the timelines associated with discovery often encompass not the raw number of hours (58.31 per this study) but the duration or number of months it takes to complete it. Of note, the extensive duration may reflect systemic issues within the employment agency, such as the employment specialist having multiple job duties in addition to customized employment implementation, or leaving limited hours during the work week or month to devote to discovery. The number of hours it takes to complete discovery services should remain the same whether these hours are spread over four weeks or nine months. Because the DFS also includes “Timeliness Scores” to rate the duration of each phase of discovery, the timeline data generated by high-fidelity discovery implementation should validate and substantiate the average duration of discovery in addition to the average number of hours.
Only six of the 17 individuals engaged in the discovery process obtained a customized job. These individuals worked an average of 14.3 hr per week with mean hourly wages exceeding the federal minimum wage. These outcomes should be interpreted with caution as this study did not examine whether or how information obtained during the discovery process was linked to distinct customized employment job development practices. It is important to highlight discovery is not a standalone service but rather, it is the first phase of the customized employment process. The efficacy of discovery is contingent upon its immediate, compulsory transition to the second phase, customized job development. A perfect fidelity score on discovery implementation means little if the employment seeker is simply placed in a traditional job or worse, receives no job development services at all. Similarly, the possibilities associated with a customized job are greatly diminished absent the use of CE consultation and supports. The effectiveness of CE may be weakened when phases are viewed as discrete parts versus ensuring that they are each an integral part of one larger, interconnected process.
Limitations and Future Research
There are several major limitations to this study that should be noted. First, the COVID-19 pandemic interrupted the customized employment activities and data collection. Some individuals who completed the discovery process may not have obtained a customized job because of business closings. Second, some employment specialists themselves lost employment when integrated employment services were suspended by provider agencies. Third, because we were required to terminate data collection activities, the study had a small sample size which limits the findings of this study. The small sample size constrained our ability to conduct a more robust and parametric analysis of the data.
Additional research is needed with a larger sample to more clearly establish the psychometric properties of the DFS. Factor analysis should be conducted to determine how the underlying subdimensions of the systems and service level are measuring their respective constructs. Future research should also examine the predictive validity of the DFS in relation to customized job development, consultative employment training and supports, and employment outcomes. Future research should examine the proportional and disproportional delivery of customized employment discovery, CE job development, and CE consultative training and supports considering gender, age, race, and whether the employment seeker is or isn’t currently receiving group day or residential services.
In conclusion, this preliminary study is an important step in establishing the DFS as a measure of customized employment discovery delivered with fidelity to best practices. Use of a discovery fidelity scale may help researchers and practitioners understand the critical dimensions of discovery systems and services practices. Once the critical elements of discovery are identified, rehabilitation counselors and other employment professionals can use the DFS to make decisions about return on investment for CE services. In addition, state funding agencies and rehabilitation counselors can ensure a CE activities align with reliable and validated best employment outcome practices.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
