Abstract
This article describes a program evaluation of student learning during the Level I fieldwork experience of a master of science in occupational therapy program. We sought to determine the utility of Goal Attainment Scaling (GAS) as a measure of student learning. Over four semesters, from fall 2013 through fall 2014, 240 students completed goal attainment scales for an individualized learning goal for the semester. Using a quantitative pretest–posttest design, we found significant increases in levels of student learning goal attainment during Level I fieldwork. Qualitative data from exit surveys revealed high levels of satisfaction with the program, particularly related to the opportunity to learn a new way of envisioning and writing individualized goals. These findings support the use of GAS as an outcome measure of student learning during fieldwork.
Program evaluation, including measurement of student learning, has become a critical component of the academic setting (Lovett & Johnson, 2012). Fieldwork is an important element in the educational process that must address the development of both entry-level clinical competencies and professional behaviors (Allison & Turpin, 2004; American Occupational Therapy Association, 2009; Cohn & Crist, 1995; Kasar & Muscari, 2000). The Level I fieldwork experience provides the first opportunity for students to apply the techniques and theoretical models they have learned in the classroom to real-life situations. Much literature supports the need to assess professional behaviors of students in preparation for entry to the profession (Knightbridge, 2014; Ledet, Esparza, & Peloquin, 2005; Smith Randolph, 2003). Professional behaviors include self-assessment, critical thinking, and reflective practice (Giles, Carson, Breland, Coker-Bolt, & Bowman, 2014). These skills are universal for health care professionals but can be difficult to concretize and measure (Cavanaugh & Konrad, 2012).
Learning is an individualized and highly subjective process, influenced by personality traits, learning style, and preferences (Mitcham, 2014). Bandura’s (1986) social cognitive model emphasized that learners direct their own learning processes and level of goal attainment by establishing challenging goals for themselves. Zimmerman (1989) applied Bandura’s model to the academic setting in developing the social cognitive view of self-regulated academic learning. Zimmerman posited that student learning and achievement are guided by self-observation, self-judgment, and self-reactions and that all are needed to develop academic self-efficacy.
Student-centered learning, often referred to as self-directed learning, is linked to early work by Rogers, who developed client-centered therapy and later applied these concepts to student-centered education (Rogers, Lyon, & Tausch, 2014). A major concept of student-centered education is that intrinsic learning motivation is necessary for lifelong learning and ability to adapt to change. A review of the literature reveals that student-centered learning emphasizes student choice in what and how to study as well as principles of active learning and student responsibility and accountability (O’Neill & McMahon, 2005). Moreover, research has indicated that self-selected, individualized goals are attained at a higher rate than goals selected by another person (Maher, 1982; Sheldon & Elliot, 1998; Webb & Glueckauf, 1994) and that students’ involvement in defining their process of professional development enhances learning (Kozloff, 1987).
Shaping students’ professional behaviors and improving their clinical skills are important goals of the Level I fieldwork experience. However, the expectations and structure of each Level I fieldwork can vary greatly, from observational to intensive, hands-on engagement; from individual to group treatment; and from distant supervision by a non–occupational therapy fieldwork educator to close supervision by an occupational therapist or occupational therapy assistant (Johnson, Koenig, Piersol, Santalucia, & Wachter-Schutz, 2006). Duration of Level I fieldwork experiences can also vary (e.g., from 1–2 days a week for a semester, to 1 wk full time, to 1 mo full time).
In addition to lack of uniformity in supervision, setting, and length of time, no universal Level I fieldwork assessment tool has been developed to measure professional behaviors and clinical skills, resulting in a variety of evaluation methods. Accreditation Council for Occupational Therapy Education (ACOTE®) Standard C.1.10 for Level I fieldwork (ACOTE, 2012) states only that all academic programs must document a mechanism for formal evaluation of student performance, but the standard does not mandate the use of any particular performance outcome measure. Recently, a task force was appointed to create a new universal Level I fieldwork competency evaluation tool by 2016 for use across all practice settings for both occupational therapist and occupational therapy assistant students (Geraci, 2015).
Literature Review
A growing number of studies have explored educational strategies that encourage students to reflect on their learning experiences. Mackenzie (2002) used both briefing and debriefing processes with students before and after fieldwork experiences to better integrate fieldwork into the curriculum. Scheerer (2003) conducted focus groups and interviews with students, in both classroom and fieldwork settings, about professional behavior feedback. Some studies have incorporated self-reflective writing to identify changes in personal growth over time (Knightbridge, 2014; Zimmerman, Hanson, Stube, Jedlicka, & Fox, 2007). None of these studies, however, used a quantitative measurement of change over time in student-selected learning goals.
Because fieldwork offers varied experiences and types of supervision, measurement of student professional behaviors and clinical skills, even using standardized rating forms, can be subjective, limited, and different from evaluator to evaluator. To address these limitations, Goal Attainment Scaling (GAS; Kiresuk & Sherman, 1968) was selected as an outcome measure for this study to provide a student-centered, individualized assessment of student learning over time. GAS is a method of goal scaling using a 5-point Likert scale from +2 to –2 to capture all possible outcomes for a given goal (see Figure 1 for a completed example and GAS level definitions). GAS, originally developed for evaluating effectiveness of community mental health services, has been successfully used to measure clinical outcomes in adult rehabilitation settings (Bouwens, van Heugten, & Verhey, 2009; Hurn, Kneebone, & Cropley, 2006) and pediatrics (Mailloux et al., 2007; Ruble, McGrew, & Toland, 2012) and is increasingly used in program evaluation (Kolip & Schaefer, 2013; Ottenbacher & Cusick, 1990). GAS has also been used in conjunction with the Canadian Occupational Performance Measure (Law et al., 2014) to study the combined clinical utility of both measures (Doig, Fleming, Kuipers, & Cornwell, 2010) and to measure clinical and housing changes in a homeless population (Chapleau, Seroczynski, Meyers, Lamb, & Buchino, 2012). It has also been used in K–12 education as an academic and behavioral assessment (Clark & Chandler, 2013).
GAS levels and corresponding examples.
Despite its successful application in such varied populations and settings, little is known about the clinical utility of GAS as a measure of professional development in higher education. In one higher education study (Sheldon & Elliot, 1998), GAS was used to compare autonomous and controlled goal attainment in college students and resulted in greater goal attainment for self-selected goals. However, this study focused on short-term project goals of undergraduate students enrolled in a psychology course.
The overall purpose of this program evaluation was to measure changes in student learning during Level I fieldwork. Our secondary aim was to use GAS in a novel way to enable students to take an active role in selecting and monitoring their own professional development while learning more about setting goals and predicting outcomes, skills needed for clinical practice.
Method
Program Evaluation Design
The project was reviewed and deemed exempt by the institutional review board for the protection of human participants. A one-group pretest–posttest quasi-experimental design was used for the evaluations during each semester. Thus, for one measure, a single observation was taken on a group of occupational therapy students at the beginning of each semester, at the semester midpoint, and at the end of the semester.
Instruments
GAS was used as a quantitative outcome measure of student learning in addition to the traditional competency-based assessment, adapted from the Ability-Based Assessment in Physical Therapy (May, Morgan, Lemke, Karst, & Stone, 1995), already in use at the end of each semester. The latter, a competency-based assessment with a total possible score of 91, provides a gross measure of students’ clinical competencies across 11 constructs of professional behaviors such as communication skills, effective use of time and resources, and use of constructive feedback.
GAS is an individualized non–norm-referenced measure that is used to determine within-participant change over time. GAS scores can be calculated using a t score formula that measures change in performance for a goal, with a mean of 0 and a standard deviation of 1 on the +2 to –2 scale. GAS has been shown to have good content validity, responsiveness, and interrater reliability when goal scales represent measurable behaviors that are established at equidistant levels (Marson, Wei, & Wasserman, 2009; Palisano, 1993). The need for adequate training for goal setters to maximize validity and reliability has been extensively discussed in the literature (Krasny-Pacini, Hiebel, Pauly, Godon, & Chevignard, 2013; Palisano, 1993; Ruble et al., 2012). Numerous studies have found GAS to be highly sensitive to change, making it a preferred outcome measure by parents, teachers, and clinicians (Mailloux et al., 2007; Miller, Coll, & Schoen, 2007).
In addition to GAS, a brief, written exit survey, developed by the authors, was added at the end of the study for both students and faculty supervisors to provide anonymous feedback for program evaluation purposes. The survey consisted of four items:
How did the initial GAS orientation prepare you to participate in the project?
What were the benefits of participating in the GAS project this semester?
Describe any challenges you may have encountered.
Provide any additional information/comments that may be helpful to us as we assess the effectiveness of this program evaluation project.
These items were designed to elicit responses that would address the secondary aims of the study: to capture students’ perspective of the process, its effectiveness in measuring professional development, and students’ perception of the usefulness of the process in developing skills in setting goals and predicting outcomes.
Participants
Participants consisted of all Level I fieldwork students completing either their first or second practicum of their undergraduate occupational therapy program and engaging in approximately 8 hr of client contact per week throughout the semester under the direct supervision of faculty. Students were placed at 1 of 14 sites during one of the four semesters of the evaluation.
Procedures
This Level I fieldwork program is distinct because all students are directly supervised by occupational therapy faculty. Eighteen occupational therapy faculty members were trained in GAS through several methods. Several faculty members were familiar with GAS before the project. Several attended a 1-hr presentation provided by the first author (Chapleau) at a regional conference in spring 2013, before the fall project start date. A 30-min training session was provided at a faculty retreat in August 2013 for all full-time faculty members, and one-to-one training sessions were provided to part-time faculty members before the start of the semester. These training sessions varied from 30 to 60 min and included use of example templates and opportunities to practice goal scaling.
All students completed their learning goal scales during their orientation period, which varied slightly from clinic to clinic but generally occurred during the first 2 wk of the semester. During this orientation period, students were trained in GAS by the first author, their faculty supervisor, or both. Students were asked to prioritize one learning goal for the semester and were assisted in identifying their baseline level of performance and all possible outcomes of the goal using the 5-point scale. The training process was similar to that used for the faculty supervisors. Goal scale forms were submitted by faculty supervisors to the authors within 48 hr after they were approved and signed by the faculty supervisor. During the first semester, all forms were distributed and returned in hard copy. Feedback from faculty supervisors after the first semester led to the use of electronic forms and data submission by means of fax or email during the second and subsequent semesters. Additionally, GAS templates were posted on the e-learning site for each fieldwork course. Students were asked to submit midterm and final assessments of progress on their goals at midterm and 2 wk before semester’s end, respectively. Reminder emails were sent and telephone calls made to students and faculty supervisors for any missing data.
After receipt of an internal funding grant to enhance student assessment, two additional components were added to the program in fall 2014, the last semester of the program evaluation. First, a 10-min instructional video, produced by the first author using the university’s media services, was made available to all students and faculty supervisors to demonstrate the use of GAS in establishing and scaling student learning goals. The video outlines step-by-step directions and includes simulated role-play scenarios to demonstrate the process of goal development and quantification of each point on the GAS scale.
Second was administration of an anonymous, written exit survey. A sample of convenience was used in asking students from two of the fieldwork clinics to provide additional information to better inform future expansion of the program. The survey was administered by the authors during a group meeting.
Variables
Predictor variables of interest included first or second rotation of Level I fieldwork, fieldwork site, faculty supervisor, and semester. Outcome variables were baseline, midterm, and final GAS scores.
Data Analysis
Outcome measures included GAS assessment at three time points—baseline, midterm, and final—on a 5-point Likert scale, an ordinal variable. Paired-sample t tests were used to evaluate differences between mean baseline and midterm scores, midterm and final scores, and baseline and final scores. Point-biserial tests and analysis of variance (ANOVA) were completed to evaluate the relationship between semesters (15-wk fall or spring vs. 8-wk summer), first and second fieldwork experiences, fieldwork sites, and GAS scores.
Results
A total of 240 unduplicated students participated between fall 2013 and fall 2014, 110 (45.8%) in their first Level I fieldwork, 129 (53.8%) in their second Level I fieldwork, and 1 who did not report her GAS results. The number of students participating each semester was as follows: fall 2013: 46 students, spring 2014: 63 students, summer 2014: 48 students, and fall 2014: 83 students.
Students were assigned to a total of 14 Level I fieldwork sites, with a total of 18 faculty supervisors, each hosting between 2 and 10 students each semester. No fieldwork site hosted more than 20% of the students. Level I fieldwork sites on campus included an after-school psychosocial program, adult rehabilitation and vocational training programs, sensory clinic and feeding programs, and a pediatric trauma assessment center. Off-campus sites included a psychiatric hospital, a pediatric developmental clinic, an elementary school, a homeless shelter, a hand therapy clinic, and a family health center.
Of the 240 unique reports scaling baseline for the GAS learning goal, the mean was –1.90, very close to the bottom of self-reported competency on GAS. A smaller reporting sample, 204, for the midterm scaling had mean scores near the center of the scale. Final fieldwork GAS scores for the 233 students reporting had a mean of 0.61, between the middle and top end of the GAS scale. See Table 1 for results of mean baseline, midterm, and final GAS scaling.
Goal Attainment Scaling Self-Rating Among Fieldwork Students
Note. M = mean; SD = standard deviation.
To measure changes in student learning during Level I fieldwork, a paired sample t test was used (Table 2). The paired sample t test indicated significantly higher midterm GAS scores compared with baseline GAS scores (p < .0001), final GAS scores compared with midterm GAS scores (p < .0001), and final GAS scores compared with baseline GAS scores (p < .0001). Students had significantly higher scores at midterm and end of semester than they did at the beginning of the semester.
Baseline, Midterm, and Final GAS Scores for Fieldwork Students (N = 203)
Note. GAS = Goal Attainment Scaling; MD = mean difference; SD = standard deviation.
Several variables are important to test to determine the effect of the change over time in GAS scores: first or second Level I fieldwork experience, semester (15-wk fall or spring vs. 8-wk summer), and site of fieldwork. In evaluating whether the fieldwork level was associated with the outcome of GAS baseline, midterm, or final scores, a point-biserial test was performed. Whether a student was in his or her first or second Level I fieldwork was significantly associated with baseline GAS scores (t score = –2.62, p < .01) and midterm GAS scores (t score = –2.45, p < .05), with students who completed their second Level I fieldwork reporting higher scores. Fieldwork level was not significantly associated with final GAS scores (t score = –1.64, p = .10).
Because training varied by semester, the relationship between semester and GAS scores was evaluated. The semester during which students completed GAS was not significantly associated with GAS baseline score (mean [M] = –1.90, standard deviation [SD] = 0.35, F [3, 236] = 1.03, p = .39), GAS midterm score (M = –0.50, SD = 1.02, F [3, 200] = 1.36, p =.26), or GAS final score (M = 0.57, SD = 0.98, F [3, 229] = 0.52, p = .67).
To determine whether the fieldwork site was associated with GAS baseline, midterm, or final scores, an ANOVA was completed. Fieldwork site was significantly associated with GAS baseline score (M = –1.90, SD = 0.35, F [13, 226] = 2.81, p = .001) and GAS final score (M = 0.57, SD = 0.98, F [13, 219] = 3.77 p < .001). Bonferroni post hoc tests demonstrated that 3 sites, the pediatric developmental clinic, the elementary school, and the adult vocational training program, had significantly different scores from the other 11 sites.
At the end of the fall 2014 semester, 9 students completed the exit survey. Using a sample of convenience, 12 students, representing two clinics, had been invited to attend the end-of-semester lunch meeting. Three students reported schedule conflicts and did not attend. When asked how the initial GAS orientation prepared them to participate in the project, all students responded positively but with vague comments such as “It worked well” or “It was helpful.” When asked to identify the benefits of participating in the GAS project, students provided more specific examples such as
“GAS is a much easier way of identifying progress and writing goals. I learned that I definitely need to make sure I only have one identifying/measurable factor.”
“I was able to analyze what area(s) needed to develop further (professionally).”
“The GAS project allowed me to see the progress I was making toward my goals at both midterm and final.”
“The process of making multiple goals (levels) under one major goal.”
None of the 9 students identified any challenges in using GAS for scaling learning goals.
Discussion
Our findings support the use of GAS as a method of student assessment of learning during Level I fieldwork and provide useful academic program evaluation data. Qualitative data also support the use of GAS in enabling students to monitor their own professional development while learning more about goal setting and predicting outcomes. In an analysis of GAS scores at the beginning and end of a Level I fieldwork experience, scores significantly improved over time. Students’ goal attainment related to professional growth and competency increased significantly across different sites and for both the first and second Level I fieldwork experience. The higher level of goal attainment among students completing their second Level I experience confirms Kasar and Muscari’s (2000) conceptual model of professional development stages, supporting the belief that students progress developmentally in acquiring professional behaviors. GAS also allowed students to establish their own learning goals in collaboration with their faculty supervisor and gave them experience in using a quantitative measure to capture learning over time, which students reported as valuable. Even though training resources varied during different semesters, there was no correlation between semester and GAS scores.
Qualitative data supported quantitative findings and illustrate the utility of GAS in supporting developmental learning. Students reported appreciating the training video and the exposure to GAS. As the study progressed, students were increasingly exposed to GAS in the classroom before they began their fieldwork. Students also reported that GAS was a helpful visual tool that enabled them to see progress on a continuum, rather than an “all or nothing” outcome. One student commented on GAS helping her accept a less than “perfect score” (+2), demonstrating growth in awareness of learning as a developmental process.
The variation from the mean in 3 of the 14 sites may be attributable to differences in faculty supervisor training as a result of the GAS instructional video being available only during the last semester of the project. Survey findings support the value of the video and are consistent with the literature, which emphasizes the need for thorough training in consistent goal-scaling methods (Krasny-Pacini et al., 2013; Palisano, 1993; Ruble et al., 2012). Although goal attainment varied among these 3 sites, no significant differences were found between outcomes of students trained by the first author, who had GAS experience, and outcomes of those trained by faculty supervisors who were new to GAS. This finding is reassuring in relation to the validity of the tool in measuring growth in student learning.
Limitations
This program evaluation was conducted in an academic program in which both Level I fieldwork experiences were supervised by faculty, allowing an ideal opportunity to improve consistency in the supervisory process and student expectations. This unique model, although useful for our purposes, may limit the generalizability of the results to other Level I fieldwork structures. A confounding variable was change over time: Later cohorts of students were increasingly exposed to GAS instruction in the classroom as faculty became more familiar with the outcome measure and introduced it before fieldwork as a standard educational tool. In addition, the qualitative data from the exit survey were limited to two fieldwork groups and, thus, may not be generalizable to the other participants’ experiences.
Future Research
On the basis of the findings from this program evaluation, we have implemented a 1-yr, interprofessional study of student learning using GAS to gather additional information about the perceived value of the GAS process to both students and faculty supervisors. We have collaborated with faculty from speech pathology and audiology, nursing, and music therapy departments. Early findings show promise for use of GAS as an interprofessional outcome measure of student learning.
Conclusion and Implications for Occupational Therapy Education
These findings support the utility of GAS as an assessment of student learning during Level I fieldwork. Facilitating the setting of individualized learning goals and personal reflection on goal progress throughout fieldwork may be beneficial in preparing students for entry-level practice. The following recommendations are suggested for occupational therapy education:
Ensure thorough training and support as well as ongoing access to training materials throughout the Level I fieldwork.
Provide multiple methods and time points for training of faculty supervisors, including in person, video, and telephonic.
Develop a curricular thread to build student competency in using GAS as a tool for self-reflective learning both before and after Level I fieldwork.
Use a developmental approach to Level I fieldwork supervision, providing additional supports for students completing their first clinical rotation and promoting increased student-directed learning in later rotations.
Footnotes
Acknowledgments
We thank Ben Atchison, Western Michigan University (WMU) occupational therapy department chair, for his continual support; Sara Clark, WMU occupational therapy clinic coordinator, for her editorial feedback; and all faculty supervisors and students for their interest in learning and adopting GAS. This work was supported by funds from the Faculty Assessment Award Program, Institutional Effectiveness Office, WMU.