Abstract
This article highlights evidence demonstrating the validity and precision of the revised American Occupational Therapy Association’s Fieldwork Performance Evaluation items and supports academic and fieldwork settings for occupational therapy students and occupational therapy assistant students.
The vast majority of occupational therapy education programs use the American Occupational Therapy Association (AOTA) Fieldwork Performance Evaluations (FWPEs) to meet the educational standards set by the Accreditation Council for Occupational Therapy Education (ACOTE®). Because of changes in the health care environment, education, and society (e.g., changes in reimbursement, productivity expectations, and emerging areas of practice), but also because of changes within the profession and the discipline of occupational therapy, AOTA appointed a series of task forces to revise the 2002 versions of the FWPE for the Occupational Therapy Student (OTS) and the FWPE for the Occupational Therapy Assistant Student (OTAS). In 2015, the final task force, which drew from the work of previous task forces, submitted recommended new and revised items for both versions of the FWPE. The item revisions were suggested to better match the larger diversity in work environments for the professions and to align better with the profession and discipline of occupational therapy. A team of researchers at University of Illinois at Chicago, commissioned by the AOTA, then developed and followed a rigorous two-stage process to determine whether the revised FWPEs would demonstrate evidence of validity and reliability/precision when applied in various educational and fieldwork settings.
The first phase of the validation process, which used the method of cognitive interviewing (Collins, 2015), aimed to establish initial validity evidence for test content. This first phase is presented in Part 1 (Preissner et al., 2020) of this two-part article series. The findings from the qualitative approach unearthed some areas of the tool that were in need of refinement, including, but not limited to, the relevance of items to a variety of settings, distinctions between items, and alignment of items with current AOTA documents. The findings also highlighted that the new proposed criterion-referenced rating scale was well received (Preissner et al., 2020).
In line with the process by which the 2002 versions of the FWPEs were evaluated (Atler, 2003), this second phase of the study (Part 2) used an application of a Rasch model (Bond & Fox, 2015) to further evaluate evidence of internal structure, response processes, and precision in testing of the revised FWPEs (Preissner et al., 2020). The Rasch model is suitable for ordinal scales and has, for decades, been the major mathematical approach in occupational therapy in the development and validation of test and scale construction. The Rasch model can also be recommended when items in a test may demonstrate various challenges, and also to monitor whether such item hierarchy reflects an underlying theoretical construct. The use of Rasch models may, therefore, serve several purposes when monitoring empirical data in relation to evidence of validity and reliability/precision as well as theory (American Educational Research Association, American Psychological Association, and National Council of Measurement in Education, 2015).
The specific research questions for this study are as follows: How well are the rating scales functioning in the revised FWPE for the OTS and FWPE for the OTAS? Is there satisfactory validity evidence in relation to internal structure and respondents’ response processes in the revised FWPE for the OTS and FWPE for the OTAS? Is it possible to separate distinct groups among the respondents? In other words, do the revised FWPE for the OTS and FWPE for the OTAS have the ability to separate students into different levels of fieldwork competence? What is the relationship between the current (2002) FWPE scales and the revised FWPE scales?
Method
Epistemological Approach
The epistemological–ontological approach for the overall validation project is based on postpositivism and critical realism (Cruickshank, 2012). The underlying belief is that a systematic objective system can be developed to evaluate the student’s quality of fieldwork performance, with arguments that both quantitative and qualitative methods are relevant approaches within such validation processes.
Recruitment and Data-Gathering Procedures
Before initiation of the data collection, the University of Illinois at Chicago Institutional Review Board approved the study. Before beginning recruitment, the research team made presentations and announcements regarding the study at both the AOTA Education Summit and the annual AOTA conference. Doing so provided the occupational therapy fieldwork education community with information about the study and advanced notice of the recruitment process. Participants were later recruited by dissemination of emails and flyers regarding the study to academic fieldwork coordinators in OT and OTA programs, clinical site supervisors, and state occupational therapy associations; postings to academic fieldwork coordinator electronic distribution lists; and an announcement in OT Practice magazine. Two online surveys were developed—one FWPE for OTSs and one FWPE for OTASs—and fieldwork educators who were interested in participating in the study e-mailed the research team to verify their eligibility. To be eligible for the study, a participant had to be a fieldwork educator who had supervised an OTS or OTAS within the previous 4 weeks. If the fieldwork educator met the eligibility criteria, they were enrolled in the study and provided a link to an anonymous survey containing the new items, matching the education (OT/OTA) of their student. They were asked to rate the revised FWPE items for a student they had recently evaluated using the 2002 version of the FWPE. They were also asked to provide the student’s final sum score from the 2002 FWPE, as well as demographic information about themselves and the student (on the basis of the fieldwork educator’s perceptions). Fieldwork educators who supervised multiple students were asked to complete a separate survey for each student. Demographic data on fieldwork educators and students were gathered and coded according to principles used within the University of Illinois at Chicago Institutional Review Board, in relation to ethnicity, to minimize risks of unequitable presentation of data.
Participant Sample
Two hundred sixty-seven fieldwork educators from 39 states and a wide variety of practice settings responded to the survey (see Table A.1 in the Supplemental Appendix, available online with this article at https://research.aota.org/ajot). Of these respondents, 228 were occupational therapists (OTs) who each supervised one OTS, and 39 were occupational therapy assistants (OTAs) or OTs who each supervised one OTAS. None of the fieldwork educators reported multiple student evaluations into the online survey systems. The top four practice settings reported were acute care, inpatient rehabilitation, outpatient pediatrics, and schools. Fifty-nine educators (22.1%) supervised their student in multiple settings. Respondents had a wide range of experience, with some being first-time fieldwork educators and others having educated fieldwork students for more than 30 yr. More than one third of the respondents (38.6%) reported completing the AOTA Fieldwork Educator Certificate Workshop. This 2-day workshop is designed specifically for fieldwork educators and academic fieldwork coordinators by AOTA and is organized throughout the country in regional workshops. The data echoed the shifting demographics of the occupational therapy field and of the U.S. workforce, with students coming from more diverse ethnic backgrounds than educators (occupational therapy fieldwork educators vs. students: 92.5% vs. 80.3% White/Caucasian; OTA fieldwork educators vs. students: 94.9% vs. 71.8% White/Caucasian). Educators’ ages ranged from the 20s to more than 60 yr (38.2% were ages 30–39). Although the vast majority of OTSs were ages 20 to 29, OTASs had a wider distribution of ages (see also Supplemental Table A.2).
FWPE Instruments
The FWPE for OTS and FWPE for OTAS online surveys included 37 and 31 items, respectively. All items were scored using a similar 4-point, ordinal, criterion-referenced rating scale ranging from unsatisfactory (1) to exemplary (4). Twenty-one of the items were the same for both FWPE scales. The items in the FWPEs are divided into different domains. In the FWPE for OTS, the following domains are labeled: Fundamentals of Practice (Items 1–3), Basic Tenets (Items 4–6), Screening and Evaluation (Items 7–15), Intervention (Items 16–24), Management of Occupational Therapy Services (Items 25–28), and Communication and Professional Behavior (Items 29–37). In the FWPE for OTAS, the domains are labeled as follows: Fundamentals of Practice (Items 1–3), Basic Tenets (Items 4–6), Screening and Evaluation (Items 7–11), Intervention (Items 12–18), Management of Occupational Therapy Services (Items 19–22), and Communication and Professional Behaviors (Items 23–31). The definitions of the items included in the FWPEs for OTS and OTAS are presented in Appendixes A.1 and A.2 in the Supplemental Appendix (see also Preissner et al., 2020).
Statistical Analysis
The analysis of the FWPE survey data was guided by a Rasch rating scale model (Bond & Fox, 2015). The four-category raw scores for the 37 items of the FWPE for the OTS and the 31 items of the FWPE for the OTAS were initially analyzed using the WINSTEPS Rasch computer software program, Version 3.91.0.0 (Linacre, 2015). We performed the analyses following a systematic stepwise approach as described in previous studies (Lerdal & Kottorp, 2011; Lerdal et al., 2016; Rustøen et al., 2018). In this study, the various sources of validity and reliability/precision evidence (American Educational Research Association, American Psychological Association, and National Council of Measurement in Education, 2015) were addressed with outcomes as follows: response processes (rating scale functioning and person goodness-of-fit), internal structure (local independence, item goodness-of-fit, unidimensionality, and correlations between current and revised FWPEs), and reliability/precision (separation indices).
All item analyses were performed in parallel processes between the FWPE for the OTS and the FWPE for the OTAS. A decision was taken to start the analysis by evaluating each of the FWPE domains/subscales but then to also analyze the FWPEs as total scales, because of earlier research and the current practices of the FWPEs (Atler, 2003).
Response Processes: Rating Scale Functioning
First, the rating scale functioning of the four-category rating scales was investigated to determine whether the average measures on each item for each category advanced monotonically, whether these measures are associated with outfit mean square values less than 2.0 for each of the step calibrations, and whether the Andrich rating scale thresholds advance by at least 1.4 logits but less than 5.0 logits (Linacre, 2004). Because the rating scale was similar across all items in the FWPEs and the scores on item level did not meet the criterion of at least 10 scores by category as suggested by Linacre (2004), we analyzed the rating scale functioning across all FWPE items included.
Internal Structure: Local Independence
In the second step, we evaluated Rasch model’s assumption of local independence among the FWPE items by monitoring the correlations between the item score residuals (Yen, 1984, 1993). A criterion of a shared variance between the standardized item score residuals that is not more than 50% (corresponding to a correlation coefficient similar to or more than 0.7 between them) was set to support local independence among items (Linacre, 2021).
Internal Structure: Item Goodness-of-Fit and Unidimensionality
A continued evaluation of the fit of the item responses (Bond & Fox, 2015) was then performed. An FWPE item that did not show acceptable goodness-of-fit to the model (evidenced by a response pattern across individual item scores that was more unexpected than expected) was removed, and the psychometric properties of the remaining items were reanalyzed until all remaining items demonstrated acceptable goodness-of-fit to the Rasch model. As sample size has an impact on item fit statistics, an adjusted criterion for acceptable item goodness-of-fit was set for infit mean square values between 0.7 and 1.3 logits (Smith et al., 2008) for the FWPE for the OTS, because the sample was considered larger (n = 228). Because the sample for the FWPE for the OTAS was smaller (n = 39), with an increased risk that individual unexpected scores have a higher impact on item fit statistics, we accepted a wider range for item infit mean square values between 0.6 and 1.4 logits (Wright & Linacre, 1994), which also has been applied in earlier studies (Hällgren et al., 2011; Kottorp et al., 2003; Patomella et al., 2006).
Furthermore, we evaluated the level of unidimensionality in the generated FWPE measures using a principal-components analysis of the residuals, with the criterion that the first latent dimension should explain at least 50% of total variance, in line with earlier studies (Lerdal & Kottorp, 2011; Lerdal et al., 2016; Rustøen et al., 2018). The eigenvalue of the secondary dimension (reported here as first contrast) was also monitored, with a cutoff of ≥2.0 to signal a lack of convergence in the data (Linacre, 2021; Raiche, 2005).
Response Processes: Person Goodness-of-Fit
In the following step, person response validity for both FWPE domains/subscales, as well as for total scales, was evaluated. A criterion for evaluating person goodness-of-fit was to reject infit mean square values of ≥1.4 logits that were associated with a z value of ≥2, accepting that 5% of the sample may, by chance, fail to demonstrate acceptable goodness-of-fit without threatening evidence of person response validity (Hällgren et al., 2011; Kottorp et al., 2003; Patomella et al., 2006). In contrast to the item goodness-of-fit analysis, people who did not demonstrate acceptable goodness-of-fit were not excluded, because all respondents were considered sufficiently relevant to be included in the study design (being OTSs or OTASs).
Precision: Separation Index
We also addressed the precision in the FWPE measures by estimating the ability of the FWPE domains/subscales, as well as of the total scales, to separate students into distinct groups. To determine whether the FWPE scales could distinguish students who demonstrated different levels of fieldwork practice competence, we assessed the person-separation reliability index for each scale. For clinical purposes, a criterion that the FWPE scale should be able to distinguish at least three groups (indicating high, medium, and low levels of competence) was chosen, which requires a person separation index of at least 2.0 (Fisher, 1992).
Internal Structure: Correlations Between Current and Revised FWPEs
Finally, we used Pearson’s correlation coefficients to evaluate the relationships between the total sum scores from the current FWPEs (Atler, 2003) and the Rasch-generated measures of the item-reduced total revised FWPE scales from the analyses performed in this study.
Results
Response Processes: Rating Scale Functioning
The rating scales for the FWPEs for the OTS and the OTAS demonstrated acceptable outcomes in relation to established criteria. Only 18 missing values in item scores were detected in the FWPE for the OTS dataset (0.2%), and there were no missing values in the FWPE for OTAS dataset. The average measures for each category advanced monotonically and were associated with outfit mean square values less than 2.0. The Andrich thresholds also met the set criteria for the FWPE rating scales, except for the threshold distance between Categories 2 and 3 and Categories 3 and 4 on the FWPE for the OTS, which exceeded the set difference (5.61 vs. 5.0). See Table 1 for a summary of the findings of the FWPE rating scales.
Rating Scale Functioning in the Revised Fieldwork Performance Evaluations for the OTS and the OTAS
Note. FWPE = Fieldwork Performance Evaluation; n/a = not applicable; OTAS = occupational therapy assistant student; OTS = occupational therapy student.
Internal Structure: Local Independence
The assumption of local independence was confirmed overall in the FWPE for the OTS scale, where none of the standardized item-residual correlation coefficients exceeded 0.7. The assumption of local independence was also confirmed overall in the FWPE for the OTAS scale.
Internal Structure: Item Goodness-of-Fit and Unidimensionality
When analyzing the fit statistics of the FWPE for the OTS subscales, we found that two out of five subscales had items that did not demonstrate acceptable goodness-of-fit: Item 28 did not meet the criterion for fit in the Management of Occupational Therapy Services subscale, and Item 35 did not meet the criterion for fit in the Communication and Professional Behaviors subscale. These items were then excluded, and the analysis was repeated for the remaining items in those subscales. In the second iteration, all subscales met the criterion for item goodness-of-fit. In the analysis of the fit statistics of the FWPE for the OTAS subscales, all subscales met the criteria for item fit. See Table 2 for more information.
Statistical Approach, Criteria, and Results of the Rasch Analysis of the Domains in the Revised Fieldwork Performance Evaluations for the OTS and the OTAS
Note. FWPE = Fieldwork Performance Evaluation; MnSq = mean square; OTAS = occupational therapy assistant student; OTS = occupational therapy student.
When analyzing the infit mean square statistics for the 37 items in the FWPE for the OTS total scale, we found that all but three items (Items 28, 32, and 35) demonstrated acceptable goodness-of-fit to the model. In a second iteration, all but two items (Items 18 and 34) demonstrated fit. After these two items were excluded, the next analysis demonstrated that all remaining 32 items fell within the acceptable range (see Table 3).
Statistical Approach, Criteria, and Results of the Rasch Analysis of the Revised Fieldwork Performance Evaluations for the OTS and the OTAS Total and Reduced Scales
Note. FWPE = Fieldwork Performance Evaluation; MnSq = mean square; OTAS = occupational therapy assistant student; OTS = occupational therapy student.
The unidimensionality of the FWPE for the OTS subscales were mixed, with only one subscale demonstrating a variance explained clearly above the set criterion (Basic Tenets, 64.4%), with an acceptable eigenvalue of 1.55. The four other subscales were below or just on the set criterion for the total variance explained but met the criterion for eigenvalue in the first contrast (See Table 2). The original 37-item FWPE for the OTS scale also demonstrated a variance lower than the set criterion (37.5%), with an eigenvalue of 3.03 in the first contrast. The unidimensionality of the 32-item FWPE for the OTS scale was higher, but it was still lower than the set criterion (39.0%), with an eigenvalue of 2.85 (see Table 3).
When analyzing the infit mean square statistics of the 31 items in the FWPE for the OTAS total scale, we found that all but two items (Items 26 and 29) demonstrated fit. These two items were excluded, and the analysis was repeated with the remaining 29 items. In the second iteration, all items but one (Item 22) demonstrated acceptable goodness-of-fit to the model. In the third iteration, yet another item demonstrated misfit (Item 9). After exclusion of this item, the next analysis demonstrated that all remaining 27 items fell within the acceptable range (see Table 3).
The unidimensionality of the FWPE for the OTAS subscales was also mixed, with four subscales demonstrating a variance explained clearly above the set criterion associated with acceptable eigenvalues less than 2.0. The subscale Communication and Professional Behaviors only explained 46.8% of the total variance, with an eigenvalue in the first contrast of 2.03 (see Table 2). The unidimensionality of the original 31-item FWPE for the OTAS scale was lower than our set criterion (47.6%), with an eigenvalue of 4.30 in the first contrast. The unidimensionality of the 27-item FWPE for the OTAS scale was also lower than our set criterion (49.2%), with an eigenvalue of 4.03 (see Table 3).
The hierarchies of the FWPE items for the OTS and OTAS total scales are presented in Appendixes A.1 and A.2 in the Supplemental Appendix. A higher measure (in logits) indicates a more challenging item; that is, an item with a lower probability of receiving higher scores by any student as compared with less challenging items. Three items demonstrating misfit on both total scales were similar: (1) Takes responsibility for attaining professional competence by seeking out learning opportunities and interactions with fieldwork educator(s) and others; (2) Demonstrates effective time management; and (3) Meets productivity standards or volume of work expected of occupational therapy (assistant) students.
Response Processes: Person Goodness-of-Fit
All subscales of the FWPE for the OTS demonstrated person misfit that was lower than the set criterion (5.0%; see Table 2). On the reduced FWPE for the OTS total scale, 18 out of 228 students (7.9%) demonstrated misfit to the Rasch model, which was higher than our set criterion (see Table 3). There were no systematic differences found between students demonstrating fit versus misfit in relation to the gender, ethnicity, and AOTA certification status of the fieldwork educator and the gender and ethnicity of the student.
All subscales except one demonstrated person misfit that was lower than the set criterion (5.0%) of the FWPE for the OTAS (see Table 2). Three persons (7.7%) demonstrated misfit on the Intervention subscale. Two students out of 39 (5.1%) demonstrated misfit to the Rasch model on the reduced FWPE for the OTAS scale, which was also higher than the set criterion (see Table 3).
Precision: Separation Index
The separation indices of the FWPE for OTS subscales were relatively low overall: Two subscales (Basic Tenets and Management of Occupational Therapy Services) were unable to differentiate the sample into any subgroups, two subscales (Screening and Evaluation and Communication and Professional Behaviors) were able to differentiate the sample into two groups, and one subscale (Intervention) was able to differentiate the sample into three groups (see Table 2). The separation index of the FWPE for the OTS total scale decreased when the five items demonstrating misfit were excluded (from 3.48 to 3.24), but the scale was still capable of detecting at least four distinct groups among the students (see Table 3). A person-item map of the FWPE for the OTS total scale is presented in Figure 1A. The mismatch between the FWPE for the OTS respondents’ ability measures and the FWPE item calibration challenges indicates that the FWPE items are relatively easier than the overall abilities among the OTSs. The FWPE items are better targeted toward students with overall lower fieldwork performance abilities.
Wright person-item map for the Fieldwork Performance Evaluation (FWPE) for the Occupational Therapy Student (OTS) and for the Occupational Therapy Assistant Student (OTAS): (A) FWPE for the OTS (n = 228). Each pound sign represents two people, and each period represents one person. (B) FWPE for the OTAS (n = 39). At left, each X represents one person.
The separation indices of the FWPE for OTAS subscales were also low overall: Two subscales (Basic Tenets and Management of Occupational Therapy Services) were not able to differentiate the sample into any subgroups, one subscale (Screening and Evaluation) was able to differentiate the sample into two groups, and two subscales (Intervention and Communication and Professional Behaviors) were able to differentiate the sample into three groups (see Table 2). The separation index of the FWPE for the OTAS total scale also decreased when the four items demonstrating misfit were excluded (from 3.89 to 3.75), but the scale was still sensitive enough to detect at least four distinct groups (see Table 3). A person-item map of the FWPE for the OTAS total scale is presented in Figure 1B. A similar mismatch between the OTAS respondents’ ability measures and the FWPE item challenges indicates that, here, the FWPE items are also better targeted toward OTASs with overall lower fieldwork performance abilities.
Internal Structure: Correlations Between Current and Revised FWPEs
The correlation coefficients between the current FWPEs for the OTS and OTAS total sum scores (2002) and the Rasch-generated measures of the item-reduced total FWPE for the OTS and OTAS scales were r = .72 for the OTS (p < .001) and r = .82 for the OTAS (p < .01), indicating that the revised FWPE scales evaluated in this study measure constructs that are different but related to those measured by the current (2002) FWPE scales. The correlation coefficients between the total scale FWPE measures and the reduced FWPE scale measures were both r = .99 for the OTS (p < .01) and the OTAS (p < .01).
Discussion
The findings of this second phase of the validation process of the new FWPEs for the OTS and the OTAS scales demonstrate mixed evidence of the new tools’ validity and reliability/precision. The study findings, therefore, provide important topics for discussion to further support evidence-based practice among fieldwork placements.
The new proposed criterion-referenced rating scale, which was positively regarded in the cognitive interviews from a test content perspective (Preissner et al., 2020), also provided supporting evidence of response processes for both FWPE scales in this study. It is, however, important to be aware of the specific differences within the definitions of scale steps between the 2002 version (Atler, 2003) and this revised version, because the scale step definitions are different, and to be aware that the new version does not use any norm-based references but only criterion-based competence criteria in its scoring system. Therefore, it will be important that the new FWPE forms clearly orient the fieldwork educator to the new rating scale step definitions. This was easily achieved in the online version of the tools used in the study.
Although a larger than expected number of items demonstrated misfit in both the total FWPE for the OTS (5/37 = 13.5%) and the FWPE for the OTAS (4 out of 31 = 12.9%), the findings are relatively in line with the current version of the FWPE with 8.7% item misfit (2/23), which also exceeds the 5% criterion (Atler, 2003). Three items demonstrating misfit were similar across the OTS and OTAS scales, which can suggest more generic measurement problems of fieldwork performance than program-specific problem with these items. The exclusion of items from a scale is always a complex decision. Because the OTS and OTAS items were developed by task forces and derived from processes involving various stakeholders and perspectives (supporting the generalizability of the item construction and definitions), and the findings from the qualitative study did not specifically indicate that the items demonstrating misfit were especially problematic or complex to score (Preissner et al., 2020), the recommendation is that these items currently remain in the FWPEs for the OTS and for the OTAS scales. Evidence of validity that is based on test content, regarding the relevance of the FWPE items (Preissner et al., 2020), may not always be in line with evidence of validity that is based on internal structure, as indicated in this study. Similar findings have been reported (Malinowsky et al., 2015; Schulze et al., 2016), which indicates that items that can be viewed as highly relevant in a new context/country/time do not necessarily fit the same underlying construct when the unidimensionality or internal structure of a scale is evaluated.
Still, validity evidence of the internal construct of fieldwork performance competence is not fully supported by this study, as indicated by the lower than acceptable unidimensionality in both the FWPEs for the OTS and OTAS subscales and total scales, as well as indications of multidimensionality (as evidenced by the eigenvalues in the first contrast). At the same time, fieldwork performance addresses such a wide variety of competencies and skills that, from a pragmatic perspective, the findings may still be viewed as positive, especially given that the overall pattern of scores does not seem to be biased by specific items demonstrating misfit, as indicated by the strong relationships between the total and reduced FWPE scales. Although the unidimensionality was larger in the subscales, consideration must also be given to the lack of precision in those subscales. The development of “good enough” unidimensional measures that also are sensitive enough to detect differences between individuals or groups is a balancing act. Similar patterns between subscales and total scales regarding item misfit, unidimensionality, and separation have been found in other assessments within health care education (Hasnain et al., 2017; Kottorp et al., 2019). Still, given also the mixed evidence of unidimensionality among the FWPE subscales, future studies of the tools with larger samples may explore whether different subdimensions within the scales can be identified and distinguished from each other to provide better unidimensional measures of the construct(s) being targeted. Such studies should also strive for a larger variance in the response patterns, because this could also influence the estimations of unidimensionality across subscales as well as total scales. This could be achieved with the FWPEs by also including responses from midterm evaluations in the sample.
The 7.9% (OTS) and 5.1% (OTAS) of students who did not demonstrate acceptable goodness-of-fit were also higher than the set criterion but still lower than the percentages from the current (2002) version of the FWPE, in which 10% of OTSs did not meet the criteria in the pilot study (Atler, 2003). Therefore, the findings from the new version of the FWPE are in line with the current FWPE for the OTS. In addition, this study provides new knowledge in relation to the FWPE for the OTAS, demonstrating acceptable evidence of validity in response processes also for this scale. Although no clear response pattern was found in relation to the students who demonstrated misfit, larger samples of student performance may reveal whether such systematic differences can be found. Such analyses can also reveal whether there are certain items for which certain groups demonstrate different response patterns as compared with other students, explored by differential item functioning (DIF) analyses. Such relationships can then facilitate better identification of certain student groups in need of specific supporting strategies to develop certain fieldwork performance skills and provide further evidence of fairness in testing. Given that AOTA’s Vision 2025 highlights the intention to be inclusive and equitable, and to embrace diversity, it is crucial for any evaluation assessing clinical competence among OTSs and OTASs not to be biased in relation to demographic or other variables. As the subsamples in this dataset were too small to promote stable analyses of DIF, future studies with larger subsample sizes are, therefore, urgently needed to explore these issues.
The person-item maps illustrate that both FWPEs demonstrate some mismatch between respondents’ ability measures and item calibration challenges, where items overall are easier to manage for the majority of students included in this study. This may limit the use of the scales to distinguish between high-performing students in fieldwork practice. On the other hand, the new tools and items are well matched to identify low-performing students, which may be more clinically relevant for fieldwork evaluations. The item hierarchies presented (see Figure 1; see also Appendixes A.1 and A.2 in the Supplemental Appendix) also pose the following questions for future discussions in relation to both pedagogical theories as well as the development of learning strategies in fieldwork settings: Which skills are easier to achieve for the students, and what skills are more challenging to master during fieldwork? How is this reflected in the educational program, as well as taken into consideration in various fieldwork settings?
Another finding in this study is the relationships between the current and the revised FWPEs for OTS and for OTAS. This can be viewed as an indication that the competencies required for occupational therapy practice today are related, but not exactly similar, to the ones 2 decades ago. This is also evidenced by the changes in item definitions; for more details, see Preissner et al. (2020). This finding can be interpreted also as measurable evidence for the changing profession/discipline and changing responsibilities in various practice settings today. Still, the major part of variance is shared between the two scales, indicating that the occupational therapy core competencies in general also remain stable over time.
Limitations
Several methodological considerations need to be addressed in relation to the findings. Although the number of participants (especially for the FWPE for the OTAS) is low, the sample description supports some level of generalizability of the findings; the geographical and fieldwork site representation reflect the diversity among OTs and OTAs in practice within the United States. The small samples in this study are also relatively similar to the sizes used in the study to develop the 2002 FWPE (Atler, 2003). The previous study relied on paper forms in data gathering. In this project, however, online surveys were used. Although the online format of the FWPE was highlighted as positive among the participants (Preissner et al., 2020), the overall response rate was still low.
Another limitation to consider may be the distribution of the sample, especially in relation to fieldwork performance skills. Because most data came from the final fieldwork evaluation, the distribution of data across the rating scales was less than optimal for a more in-depth analysis of the qualities of the tools. By also including more midterm evaluations (with a potential wider range of scores across the rating scale and items), a better estimation of the psychometric properties of the tools could be achieved. Furthermore, the inclusion of both midterm and final FWPEs in a systematic manner could also generate important knowledge about skill development among OTSs and OTASs over time. Such data could also complement the computation of separation indices with a potentially wider range of FWPE scores, as well as detect whether predefined known groups score in line with specified hypotheses (e.g., students at midterm evaluations score overall lower than at final FWPEs). This information could also serve as a basis for targeted interventions during fieldwork education.
Finally, it was not possible to address rater severity and rater consistency in this study. Although rater severity usually explains a large amount of the variance in outcomes when included as a confounder (Dewberry et al., 2013; Mallinson et al., 2014), it is usually relatively complicated to adjust for in relation to outcomes. To partly address the issue of rater severity and consistency, the development of an online training workshop for fieldwork educators that focuses on both the new items and the new rating scale used in the revised FWPE tools is, therefore, suggested. Such an initiative should promote overall improved quality of the scoring procedure as well as better alignment between raters. To address some of the aforementioned methodological limitations, we suggest that AOTA develop an online training, as well as an online survey format, for the FWPE for OTS and the FWPE for the OTAS and, if possible, that they collect the data in large databases systematically and anonymously, fully according to ethical guidelines. This would allow for further evaluation of the FWPE tools from a validity and fairness perspective, with larger sample sizes, as well as for the generation of new knowledge about how fieldwork skill development occurs and can be supported by academic programs and fieldwork practices. By doing so, an even greater evidence base can be established, implemented, and evaluated.
Implications for Occupational Therapy Practice and Education
According to AOTA’s Vision 2025, the occupational therapy profession aims for maximizing health, well-being, and quality of life for all people, populations, and communities through effective solutions that facilitate participation in everyday living (AOTA, 2017). Within this directive, it is also crucial that occupational therapy professionals support and evaluate students in relation to well-defined, up-to-date target areas and skills that reflect the uniqueness of the profession and discipline. An evidence-based evaluation of fieldwork performance competencies among OTSs and OTASs is, therefore, a good fit with this mission. The FWPE item challenge calibrations presented in Appendixes A.1 and A.2 in the Supplemental Appendix can be used by fieldwork educators to monitor specific strengths and limitations, by transferring the raw scores into the presented rank order of the item calibration challenges. In using the Rasch model, higher scores are expected on easier items, and lower scores are expected on more challenging items (Bond & Fox, 2015). If a score is unexpectedly higher in relation to the hierarchical order, it can be viewed as a specific strength in the student’s performance. If a score is unexpectedly lower in relation to the hierarchical order, it can be viewed as a specific target area for improvement in the student’s performance.
A recommendation is also to align all educational tools (including fieldwork performance evaluations) with ACOTE (2018) standards, because these standards represent critical knowledge and skills in occupational therapy education, to ensure consistency in all OTS and OTAS competencies in the United States. It is also important to highlight the need to constantly monitor the items included in FWPEs so they are aligned with current and future ACOTE standards. The benefits of using Rasch analysis in the development of the FWPE is a relatively easy validation process when including new items and/or revisions of existing items, so that validity in relation to internal structure is not compromised when ACOTE standards are updated and revised.
Finally, a transparent process for developing cutoff-criteria for the revised FWPEs for the OTS and the OTAS is also crucial for any clinical use. This has been further described in the Supplemental Technical Report of this article (available online with this article at https://research.aota.org/ajot).
Conclusion
This validation project has resulted in revised FWPEs for the OTS and the OTAS, with new evidence of validity that is based on test content, internal structure, response processes, and precision. Tools that measure the quality of competencies among students will also support the development of programs that create the best future OTAs and OTs. The revised FWPEs for the OTS and OTAS will, ultimately, then help OTs to provide clients with optimal services to support them in engaging in the occupations that they need and want to perform to participate in the home, community, and society to their full potential.
Supplemental Materials
Supplementary material for The Revised American Occupational Therapy Association Fieldwork Performance Evaluations: Evaluation of Internal Structure, Response Processes, and Precision—Part 2
Supplementary material, sj-pdf-1-aot-10.5014_ajot.2023.050207.pdf for The Revised American Occupational Therapy Association Fieldwork Performance Evaluations: Evaluation of Internal Structure, Response Processes, and Precision—Part 2 by Anders Kottorp, Catherine Killian, Kathryn Duke, Caniece Leggett, Ruxandra Drasga and Katharine Preissner in The American Journal of Occupational Therapy
Supplementary material for The Revised American Occupational Therapy Association Fieldwork Performance Evaluations: Evaluation of Internal Structure, Response Processes, and Precision—Part 2
Supplementary material, sj-pdf-2-aot-10.5014_ajot.2023.050207.pdf for The Revised American Occupational Therapy Association Fieldwork Performance Evaluations: Evaluation of Internal Structure, Response Processes, and Precision—Part 2 by Anders Kottorp, Catherine Killian, Kathryn Duke, Caniece Leggett, Ruxandra Drasga and Katharine Preissner in The American Journal of Occupational Therapy
Footnotes
Acknowledgments
The research for this article was supported by the American Occupational Therapy Association. All work was conducted at the University of Illinois at Chicago.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
