Abstract
Educators can build on current practices to design robust assessments that require students to demonstrate knowledge of occupation in practice contexts and everyday life.
Human occupation and its relation to health constitute the most central topic that students in occupational therapy must master (Hooper et al., 2015; World Federation of Occupational Therapists, 2016). Graduating students with demonstrated mastery of this topic as it relates both to practice and to life in general is critical if future practitioners are to realize occupational therapy’s distinct value, a value deeply informed by the concept of occupation (Lamb, 2017). To date, however, empirical studies assessing student mastery of knowledge and skills related to occupation are lacking.
Assessment involves collecting data about student knowledge, skills, and attitudes related to specified learning outcomes; it is undertaken to modify teaching, courses, and programs for improved learning (Suskie, 2018). By conveying educator priorities for learning, practices related to assessment shape what students believe is imperative to learn and how they spend time and identify as students (Brown, 1997). Assessment is so critical to learning that scholars urge educators to consider assessment practices before academic content and instructional strategies (Burke, 2010; Fink, 2013). As Brown (1997) posed, “If you want to change student learning then change the methods of assessment” (p. 7).
Strong assessment plans include and value both direct and indirect assessments of student learning (Banta & Palomba, 2015; Burke, 2010; Suskie, 2018). Direct assessments, those that generate direct evidence through systematic standard-driven evaluation, are considered to provide more convincing evidence that students “have learned what they need to persist, graduate, transfer, obtain jobs, and otherwise succeed” than indirect assessments, those that generate indirect evidence on the basis of students’ or stakeholders’ reflections or reports (Suskie, 2018, p. 26). Examples of direct assessments in occupational therapy include the Fieldwork Performance Evaluation (AOTA, 2020); national certification examination; and systematically graded simulations, written work, oral presentations, or capstone experiences. Indirect assessments include student self-assessments, employer surveys, and job placement or retention reports.
Whether direct or indirect, assessments of assignments and other forms of student performance vary in quality and require evaluation using criteria of robust assessments. These criteria include clear measurable objectives of expected learning, clear standards for and quality indicators of performance, alignment with course objectives, and a data collection tool. All criteria must also align (Suskie, 2018). Although learning experiences that lack these criteria could constitute an assignment, they would fall short of being a robust assessment. Leaders in assessment scholarship have thus argued for more robust assessments (Banta & Palomba, 2015; Burke, 2010; Suskie, 2018). Such assessments convey the most important things that educators want students to learn, assess their reasoning and applications of knowledge more than memorized knowledge, and provide data for both individualized student feedback and improved teaching.
As indicated earlier, the field of assessment is evolving. Yet, assessment has not been widely addressed in education scholarship in occupational therapy. A 2013 systematic mapping review of 129 articles on education found 1 study of assessment practices; however, this study did not address assessment related to occupation (Hooper et al., 2013). To our knowledge, no empirical studies related to assessment and occupation have since been published. The present article was derived from a national study of how educational programs in the United States broadly address occupation (Krishnagiri et al., 2017). We asked the following research question: How do educators in occupational therapy assess student knowledge and mastery of the concept of occupation?
Method
Design
A generic or basic qualitative research design was used (Percy et al., 2015). Specific methods and analytic strategies used in the aforementioned national study have been extensively detailed elsewhere (Krishnagiri et al., 2017).
Participants
Twenty-nine educators from 25 academic programs participated and are herein collectively referred to as participants. Programs in the eastern (17), middle (4), and western (4) United States were selected through stratified random sampling. Programs ranged across Carnegie institution types and included associate-level programs.
Data Collection
First, individual interviews were conducted to explore how occupation was generally addressed in educators’ programs and how educators knew whether students were acquiring desired knowledge of occupation. Second, each program was asked to submit classroom videos and artifacts. A total of 234 artifacts were submitted. Artifacts, video data, and transcribed interviews were uploaded into the qualitative analysis software Atlas.ti (Version 6.2; Scientific Software Development, Berlin, Germany) for analysis.
Data Analysis
All interviews were analyzed with the constant comparative method (Creswell, 2013). Data saturation was reached after analyzing interviews of educators from 15 occupational therapy and 10 occupational therapy assistant programs. Video and artifact data were next analyzed with a codebook generated through prior analyses of the interviews. All data coded as “desired learning outcome” and “learning assessment” were queried in Atlas.ti and underwent additional analyses for this article. Two researchers, working independently, inductively analyzed artifacts and then compared interview and artifact findings to identify which artifacts described learning activities and assignments that included assessments of student knowledge. The researchers then reached consensus that 82 artifacts encompassing student observations, self-analyses, interviews, exams, papers, presentations, discussion boards, and lab practicals constituted such assessments.
Using the constant comparative method (Creswell, 2013), the researchers then sorted these 82 artifacts into categories that distinguished whether collected evaluation data provided direct versus indirect evidence of student learning; categories of instruction-level versus program-level approaches were also made to assess student learning. Finally, a team of five researchers conducted matrix analyses to compare identified learning activities and assignments against criteria of robust assessment, namely, inclusion of clear objectives, performance criteria, quality indicators, and tools for measuring performance (Suskie, 2018).
Trustworthiness
To ensure credibility of findings, forms of triangulation included multiple data sources, analysts working in cycles of independent and team analyses, and analytic rounds and tools (inductive, deductive, and matrix analyses). Triangulations of findings from interview and artifact data were reviewed against published best practices. The multiple iterations of analyses were further guided by critique and consensus building by all team members.
Results
It was often unclear how the academic programs in this study determined that students achieved learning outcomes related to knowledge of occupation. On the one hand, all participants reported valuing learning outcomes related to occupation such as “understanding the importance and power of occupation”; “defining and analyzing occupation and the factors that influence occupational development, choice, and health”; or “applying their understanding to provide occupation-based interventions.” At both instructional and programmatic levels, student knowledge of occupation was assessed mostly in relationship to practice; yet, it was also assessed at instructional and programmatic levels apart from practice. On the other hand, learning activities and assignments varied considerably when compared with criteria of robust assessments. As next detailed, robust assessments that generated direct evidence of student knowledge of occupation were found to be developed at varying degrees of completeness or not undertaken at all.
Assessments of Student Knowledge of Occupation as a Tool for Practice
At the instructional level of assessment, most assessments of knowledge of occupation as a tool of practice were indirect and lacked criteria of robust assessment. For example, one assignment required students to develop an “occupational profile and wellness plan” for a person they had interviewed whose background differed from their own. During interviews, students gathered information related to the profile, challenges to occupational function, and wellness. Students had to use occupation to achieve wellness goals and explain how selected interventions enabled occupation. Possible interventions included “remediation of client factors and performance skills, adaptations or modifications of contextual or occupational demands, and [establishment of] habits and routines.” Student performance was measured by a rubric, a criterion of robust assessment. Yet, the rubric omitted criteria for knowledge of occupation and its alignment with learning objectives; it instead stressed technical aspects of applying the Occupational Therapy Practice Framework: Domain and Process (3rd ed.; American Occupational Therapy Association, 2014), formatting goals, and writing.
Another assignment, an online discussion board, provided indirect evidence of student knowledge of occupation. Level II fieldwork students posted weekly answers to questions about occupation-based interventions, including how to problem solve, create opportunities, or interact effectively with clinical instructors. The fieldwork coordinator used posted discussions to assess learning. However, the posts were not associated with clear descriptions of learning objectives, performance criteria, quality indicators, or measurement tools. Because these data provided indirect evidence of student knowledge and performance of occupation-based practice, they also could not demonstrate students’ capacities for implementing such practices.
Interviewees sometimes bemoaned the lack of assessments to measure student knowledge of occupation as a tool for practice; for instance, “I don’t think we have an overarching assessment method,” or “there’s not a test per se.” One interviewee believed “the biggest thing that shows us whether they’re getting this stuff or not are the lab practicals.” “Lab practicals” were frequently mentioned and can generate direct evidence. Yet, submitted lab practicals included neither occupation-focused learning objectives nor quality indicators of performance. Interviewees also frequently mentioned “debriefing.” One interviewee who observed that “students get it” through the curriculum noted that faculty do much debriefing after students encounter emergent practices, have other field experiences, or engage in group work. Although debriefing can offer instructive indirect evidence of student knowledge of occupation, it was unclear from our interview and artifact data whether this assessment method included reflections on occupation-focused learning objectives or tools for evaluating reflections.
Finally, one assignment, a case study, was found to include nearly all criteria of a robust assessment of knowledge of occupation. Compared against criteria of robust assessment, this assignment (1) included clear performance criteria, namely, students had to identify appropriate assessment approaches, performance needs, and rationales for at least two occupation-based interventions; (2) was tied clearly and explicitly to a course objective, which was to “identify and describe occupational therapy assessments used to measure areas of occupation, performance skills, performance patterns, contexts, activity demands, and client factors in psychosocial disorders”; and (3) included a rubric to measure performance that aligned this objective with specifics of the assignment, for example, the “occupational history is clear and complete,” and “two interventions are occupation based with good rationales.” Data were thus collected that provided direct assessment of student knowledge of occupation. However, although most criteria of robust assessment were present and aligned, quality indicators for a 4-point scale used to measure performance on the rubric were missing.
At the program level, indirect assessments of student knowledge of occupation that lacked criteria or tools for measuring student learning and knowledge of occupation were also most common. Other programs and interviewees reported using focus groups or group discussions for assessment purposes. Faculty were reported to hold focus groups with students “to see how the academic portion went” and to get more information “about the whole program” after their fieldwork experiences. During a discussion with onsite evaluators from the Accreditation Council for Occupational Therapy Education, students were reported to have been “able to articulate that occupation is the center of occupational therapy.” One interviewee said that “we don’t measure” knowledge of occupation; yet, students provided anecdotal evidence “especially during Level II fieldwork meetings” when they are “saying it.” Although believing that student knowledge of occupation “could be stronger,” an interviewee offered that students “recognize the meaning [occupation] has for clients and . . . the power of doing something” meaningful. This view was evident, according to the interviewee, through students’ abilities to look “at this occupation, at this particular piece even of this occupation, and recognize . . . what skills, materials, whatever, [it takes to do] and analyze.” These examples are all indirect assessments at programmatic levels that lack criteria or tools for measuring student knowledge of occupation but yet share a strong focus on occupation.
One direct assessment at the program level was found, although it also did not meet robust assessment criteria. An interviewee reported establishing an oral examination that students took before their second Level II fieldwork placement and again before exiting the program. During this examination, students were asked to demonstrate their knowledge of occupation by answering the following questions: “What is occupation-based practice?” “What are the important aspects of providing it?” “How do you provide it in practice?”
Assessment of Knowledge of Occupation Apart From Therapy
At the instructional level, most assessments of knowledge of occupation apart from practice were similarly indirect and lacked criteria of robust assessment. The purpose of one assignment was to assess knowledge of occupation apart from its use in therapy. Students (1) interviewed other students, family, or friends about their past, current, and future occupations, and associated meanings, habits, and routines, and (2) compared interview data with ideas about occupation professed by occupational therapy’s founders. In another assignment, students (1) kept time diaries in which they tracked their occupations, habits, and routines; (2) considered impacts of personal, sociocultural, and historical contexts on their occupational development, values, interests, and choices; and (3) reflected on how their health and well-being were influenced. Both of these assignments creatively engaged students in learning about occupation apart from practice. Another approach reported by interviewees was to gauge knowledge of occupation on the basis of students’ contributions to classroom discussions. Although focused on occupation, all three examples lacked course or assignment learning objectives and criteria for measuring student knowledge of occupation. Hence, it was unclear whether or how learning occurred.
One direct assessment of knowledge of occupation apart from practice was found that incorporated criteria of robust assessment. Students identified and presented their occupations and roles, plus relationships with their values and identity. Performance was assessed with three criteria: (1) “clearly expresses own occupational identity,” (2) “identifies valued occupations and [their centrality] to occupational identity,” and (3) “discusses value and role of occupation in own life.” However, quality indicators were missing (e.g., 1 = superficial; did not illustrate connections among occupation, identity, role, or values; 5 = clearly illustrated connections among occupation, identity, role, and values).
At the program level, indirect assessment also predominated. One program surveyed students annually about their views of occupation. Others collected similar data through focus groups. Employer surveys were also reported, although their relationships to student knowledge of occupation were unclear. Many interviewees reported using such data formatively to evaluate and improve curricula. Curriculum materials also often included outcomes that encompassed the breadth and complexity of occupation, for example, “students will be able to communicate and demonstrate their commitment to the value of occupation to support health, participation, and wellness.” How programs assessed such outcomes was unclear. Only one program described a direct assessment, an examination related to occupation given at the program’s end.
Discussion
The 25 academic programs and 29 faculty members who participated in this study intended that students become knowledgeable about occupation, its role as a therapeutic agent in practice, and its role in human health and well-being broadly. With some exceptions, however, submitted learning activities and assignments suggested substantive barriers to robust assessment of student knowledge of occupation. We next discuss these barriers and strategies for addressing them.
In this study, learning activities and assignments were found to reflect a continuum of missing versus present, and aligned versus misaligned, features of robust assessment of student knowledge of occupation. Few fell on the optimal far right end of the continuum characterized by strong alignment of all features of robust assessment identified by Suskie (2018): clear objectives for what students were expected to learn about occupation, relationships of assignment objectives to course objectives, performance standards with quality indicators, and tools for collecting performance data. When present and aligned, these assessment features together “define what students are supposed to be able to do with the content they have learned” (Biggs, 2014, p. 8; see also Grajo & Gutman, 2019).
Yet, most learning activities and assignments fell on the continuum’s far left end or near its middle, suggesting many barriers to robust assessment. Those on the continuum’s far left end omitted essential assessment features. For example, a “we know it when we see or hear it” reliance on educator intuition was essentially assessment-less because it omitted all features of robust assessment. Learning activities and assignments clustered around the continuum’s middle included key assessment features but misaligned them. For instance, included course learning objectives were not specific to the assignment, or procedural performance indicators (e.g., presentation skills, grammar) were substituted for occupation-related learning objectives. Without alignment, students may not know where to concentrate, leading to a surface approach to learning (Rust, 2002). The paucity of learning activities and assignments aimed at assessing knowledge of occupation apart from therapy also raised a considerable barrier to robust assessment of student knowledge of occupation.
How, then, can educators robustly assess student knowledge of occupation as applied to practice and, just as important, to everyday life? First, we propose that educators use backward course design (Fink, 2013) to help position assessments on the most educationally profitable far right end of the assessment continuum. Using this tool, educators determine learning outcomes and their measurement before they design (or redesign) courses, select academic content, and create learning activities and assignments.
Second, we propose that educators establish clear learning objectives that explicitly target knowledge of occupation for all learning activities and assignments that aim to address occupation. For example, the following four objectives could guide what students focus on and demonstrate while completing an interview assignment on meanings of occupation: (1) synthesize conceptual understandings of meaning and occupation from three authors, (2) identify three ways meaning is discernable in people’s occupational performance or selections of occupations, (3) craft three questions to help the interviewee talk about meanings of their occupations, and (4) analyze the interviewee’s responses informed by cited works.
Finally, we also propose clear performance criteria such as exemplary, proficient, satisfactory, and unsatisfactory that operationalize students’ respective achievements. For instance, a quality indicator of exemplary performance for Objective 1 in the previous paragraph could be “identifies themes across three authors’ discussions of meaning, discusses authors’ similarities and nuanced differences related to meaning, and generates own examples of authors’ key messages.” Data collection tools that measure quality indicators, rubrics being a classic example, are also essential.
For indirect assessments, clear objectives are likewise recommended to shape what assessment data are gathered and how. For example, an employer survey that aims to measure competence in occupation-based practice can ask, “How often does this graduate engage clients in occupations they want and need to do or document their occupational goals?” A focus group that aims to identify students’ perceptions of competence in occupation-based practice can ask, “In what situations did you demonstrate such competence, or what specific experiences and content in the curriculum helped you develop it?” Systematic processes or tools for compiling assessment data relating to aims of indirect assessments are also essential.
With respect to student knowledge of occupation broadly, educators need to consider that targeting occupation-related learning strategies nearly exclusively in relationship to practice can hinder student mastery of occupation’s significance to human development, identity, adaptation, and healthful and meaningful living broadly. Moreover, if knowledge of occupation in everyday life is “jumped over,” then students may never grasp rationales underscoring its indispensability to practice (Hooper et al., 2015). Hence, it is of utmost importance that educators integrate assessment of student knowledge of occupation apart from practice throughout a curriculum. Educators may also consider using immersive, occupation-based educational processes to teach students about occupation’s significance in everyday life simultaneously with practice-based applications.
According to Vroman et al. (2010), students come to deeply understand how occupation works in their own and others’ lives when direct experiences render theoretical notions about occupation visible to them. Vroman et al. thus used service learning, leading occupation-based groups in the community, as a pedagogical approach for teaching occupation. In their assessment, student knowledge of “the gestalt of occupation” as a concept apart from practice was deepened by real-life, occupation-based practice (p. 261). Likewise, Bazyk et al. (2010) viewed community-based service learning as helping students gradually grasp the power of occupation. It is important to note, however, that robust instructional design and assessment practices must still be incorporated that generate and assess student knowledge of occupation as both a concept unto itself and practice tool.
Lastly, an overarching barrier to robust assessment is the limited availability of faculty development related to assessment (Suskie, 2018). Findings from this study suggest needs for greater faculty development on assessment of knowledge in occupational therapy related to the concept of occupation both apart from and applied to practice. In certain cases, certain faculty may benefit from more extensive development of a deep knowledge of occupation.
Limitations and Future Research
Participants may have submitted assignments without the assessment criteria used in association with the assignment, making it appear that assessment was limited. The timing of data collection interfered with the details obtained around assessments: The interviews occurred first and included questions about how knowledge of occupation was assessed; the artifacts were collected afterward, leaving no opportunity for clarifying questions or member checking after the artifacts were received. Future research should collect assessments and measurement tools of student learning and mastery of occupation and include interviews about the processes that surround those assessments.
Implications for Occupational Therapy Educators
The results of this study have the following implications for occupational therapy educators:
Educators should sponsor professional development on robust assessment and knowledge of occupation.
Educators, as a faculty group, should specify knowledge of occupation that students must acquire by graduation.
Educators should create learning activities and assignments that include features of direct robust assessment of student knowledge of occupation both apart from practice and as a tool of practice.
Educators should evaluate assignments for inclusion and alignment of all features of robust assessment.
Conclusion
To our knowledge, this is the first study to examine how a large representative sample of academic programs in occupational therapy at associate and graduate levels across the United States assessed student knowledge of occupation. Participants were pervasively committed to teaching the concept of occupation and reported many highly creative experiential learning activities and assignments aimed at helping students grasp the concept. Findings revealed prevalent shortfalls in assessment practices. Yet, they also pointed toward strategies that educators can use to assess student knowledge of occupation robustly and directly as applied in practice and, just as important, related to its dynamic presence and influence in everyday life.
Footnotes
Acknowledgments
We acknowledge the support of the Colorado State University Center for Occupational Education, the American Occupational Therapy Foundation, and the Society for the Study of Occupation: USA.