Effects of Feedback and Reflection on Communication Skills Training for Occupational Therapy Students

Participants

The study included three groups of participants: students, instructors, and patients who were recruited from a medical center in Taiwan. The students were fourth-year occupational therapy students completing fieldwork rotation in psychiatry, and the instructors were experienced occupational therapists with a minimum of 3 yr of teaching experience. We recruited the patients from the psychiatric ward using convenience sampling, with the following inclusion criteria:

a diagnosis of mental disorder (e.g., schizophrenia, bipolar disorder, or depressive disorder) according to the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association, 2013)

The study was registered at ClinicalTrials.gov (NCT 05457231). The study was approved by the ethics committee of the local hospital. All participants provided written informed consent.

Study Design

In this study, we used a pretest and posttest design. Two experts used the Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF; Calhoun et al., 2010) to assess the students’ CS. The experts were psychiatric occupational therapists, who were blinded to the study purpose and unaware of which interview session (e.g., first, second, or third) the student was conducting, to minimize the Hawthorne effect and observer bias. All experts completed a 50-hr training program on scoring and feedback, which included in-depth analyses of three transcripts of the student– patient interviews from the research data collected during the FR-CT developmental stage and participation in discussions with development researchers to ensure consensus on rating standards and feedback quality.

Intervention

The FR-CT was structured with five key components:

an initial skill development session

Specifics of the FR-CT are detailed in Appendix A in the Supplemental Material (available online with this article at https://research.aota.org/ajot), and further supporting materials can be found in Supplemental Appendixes B and C.

We used the Communication Skills Measure for Therapists (COT) to provide comprehensive feedback. An in-depth assessment tool for evaluating therapist– patient CS (Huang et al., 2020), the COT comprises eight CS domains: Initiating Conversation, Building Rapport, Communicating Information, Receiving Information, Expressing Empathy, Guiding the Discussion, Reaching a Consensus, and Concluding the Meeting. Each CS domain contains five to 19 verbal and behavioral indicators, totaling 87 indicators. The detailed verbal and behavioral indicators allow for a nuanced understanding of specific CS performance and related problems. The COT demonstrates acceptable content validity and face validity (Huang et al., 2020), making it suitable for identifying students’ difficulties in CS and useful for instructors to provide feedback for students (Huang et al., 2021).

Procedure

Intervention Procedure

During a 12-wk fieldwork period, each student alternated every 4 wk among the acute psychiatric wards, chronic psychiatric wards, and day-care ward. During the first week, students attended an initial skill development session. After this, students were scheduled to conduct three separate interviews with different patients once a month. Each interview lasted 30 to 40 min and was supervised and audiotaped by the instructor, who sent the recordings to a transcription company. The transcripts were then randomly assigned to one expert. After each interview, the trained instructor provided timely feedback using the GKCSAF, a concise tool that is well suited for quick evaluations. The students engaged in self-reflection using the GKCSAF. Within 2 wk of each interview, the two experts provided in-depth feedback using the COT, a tool with more comprehensive CS items that was used only for feedback rather than as an outcome measure. Because of the completion of the fieldwork after the third interview, in-depth feedback for that session was not provided (Figure 1).

OPEN IN VIEWER

Figure 1.

Flow chart of the FR-CT and outcome assessments

Measure: GKCSAF

The GKCSAF is widely recognized in medical education for evaluating CS during patient interviews and providing immediate feedback (Calhoun et al., 2009). The GKCSAF comprises nine CS domains: Builds a Relationship, Opens the Discussion, Gathers Information, Understands the Patient’s and Family’s Perspective, Shares Information, Reaches Agreement, Provides Closure, Demonstrates Empathy, and Communicates Accurate Information (Calhoun et al., 2009). Each domain contains two to five verbal and behavioral indicators and is rated on a 5-point Likert scale ranging from 1 (poor) to 5 (excellent). The sum score is calculated by summing the scores of the nine CS domains to present the overall CS performance. The GKCSAF includes two forced-choice and free-text items for specific comments on strengths and weaknesses. Three versions of the GKCSAF were used in this study: the clinician–faculty version for instructors and experts, the self-assessment version for students, and the patient–family version for patients. The GKCSAF has good interrater reliability (intraclass correlation = .83; Peterson et al., 2014).

Data Analysis

We used the Shapiro-Wilks test to assess data normality. For nonnormal distributions, we used the Wilcoxon signed-rank test to analyze the differences between the pre- and posttest GKCSAF scores. Effect size (r) values of .10, .30, and .50 indicate small, medium, and large effect sizes, respectively (Rosenthal et al., 1994). Missing data were imputed using the “last observation carried forward” approach (Overall et al., 2009). Statistical significance was denoted by p < .05.

Characteristics of the Raters

This study included nine instructors, 36 students, 95 patients, and four experts, as shown in Table 1. A total of 105 interviews were completed. Among them, 34 students each completed three interviews, 1 student completed two interviews, and 1 student completed one interview. Ten patients were interviewed twice, each time by a different student.

Primary Outcome: Scores on the GKCSAF Rated by the Blinded Experts

The pre- and posttest domain and sum scores of the GKCSAF as rated by the experts are shown in Table 2. Significant improvements (p < .05) were observed in four CS domains: Builds a Relationship, Gathers Information, Shares Information, and Provides Closure, with effect sizes ranging from small to medium (rs = .29–.37). The changes in scores for the other five CS domains showed nonsignificant improvements with small effect sizes (rs = .10–.22). Regarding overall CS performance, the sum score revealed a significant improvement (p = .006), with a medium effect size (r = .32).

Secondary Outcome: Scores on the GKCSAF Rated by the Instructors, Students, and Patients

The GKCSAF scores rated by the instructors, students, and patients are shown in Appendix D in the Supplemental Material. Sum scores on the GKCSAF showed that patients’ scores were the highest (pretest = 35.5; posttest = 36.0), followed by instructors’ scores (pretest = 23.0; posttest = 24.0) and students’ scores (pretest = 20.0; posttest = 22.0), with a maximum possible score of 45. Both the instructors’ and students’ ratings showed significant improvements (p < .05) on the Gathers Information domain, with a small effect size (r = .24 for both instructors and students), and on the Provides Closure domain, with medium effect sizes (r = .31 for instructors; and r = .32 for students). Students’ ratings on the Understands the Patient’s and Family’s Perspective, Reaches Agreement, and Communicates Accurate Information domains significantly improved (p < .05), with small to large effect sizes (rs = .27, .32, and .56, respectively). No significant differences were found among the patients’ ratings (ps = .089–.846).

For overall CS performance, the sum score as rated by the students revealed a significant improvement (p < .01), with a medium effect size (r = .33). Instructors’ and patients’ ratings did not show significant improvements (p < .05), although instructors’ ratings indicated a trend toward improvement (r = .19).

Outcomes Based on Blinded Expert Ratings

The students showed significant improvements in four CS domains—Builds a Relationship, Gathers Information, Shares Information, and Provides Closure—as well as in overall CS performance, with small to medium effect sizes on the basis of expert ratings. Our positive findings are similar to those of previous studies. One study noted enhanced capabilities in gathering information, building relationships, closing the session, and overall CS performance as evaluated by trained raters after a CS course, patient interviews, and the provision of in-depth feedback using a 24-item checklist (Labaf et al., 2014). Similarly, other studies have reported improvements in gathering information and summarizing comments after a CS lecture, role-play exercises, and in-depth feedback (Brown, 2017; Brown & Woods, 2011). Additionally, significant progress has been highlighted in gathering information, providing closure, and overall CS performance after conducting standardized patient interviews, self-reflection sessions, and feedback based on observations from multiple (expert, peer, and standardized patient) perspectives (Engerer et al., 2016, 2019). These studies partially support our results, which demonstrate the benefits of integrating feedback and reflection in CS training.

Findings with the FR-CT surpass those of previous studies by offering more comprehensive, detailed, and multifaceted CS training strategies. First, the FR-CT uses an in-depth assessment tool (i.e., the COT), enabling detailed feedback that encompasses various verbal and behavioral indicators in each CS domain, both orally and in written form. This contrasts with previous studies, which predominantly relied on concise checklists or concise CS measures that cover only a few CS domains, often providing feedback solely in verbal or written formats (Berney et al., 2017; Downar et al., 2017; Servotte et al., 2019). Second, the FR-CT incorporates multiple feedback perspectives, including input from experts, instructors, and patients, thus enriching the spectrum of insights. By contrast, numerous earlier studies may have solely depended on a solitary feedback source (Berney et al., 2017; Brown, 2017; Labaf et al., 2014; Servotte et al., 2019). Third, the FR-CT is applied in real patient interview scenarios, which reflects students’ real-world CS challenges more accurately and assists them in transferring skills to authentic clinical settings. This differs from previous studies that primarily relied on standardized patient interviews or role-playing exercises (Brown, 2017; Brown & Woods, 2011; Engerer et al., 2016, 2019). Fourth, the FR-CT seamlessly incorporates both FIT and RPT principles to create a robust training framework. This approach ensures that feedback is not merely delivered but is also followed by deliberate self-reflection and purposeful practice. This robust integration distinguishes our analysis of the FR-CT from studies that may have used only a simplistic set of feedback or self-reflection principles and methods (Berney et al., 2017; Labaf et al., 2014). To summarize, the FR-CT introduces more comprehensive training strategies, which may enhance students’ CS.

Despite significant improvements in several CS domains, the students’ performance in five CS domains—namely, Opens the Discussion, Understands the Patient’s and Family’s Perspective, Reaches Agreement, Demonstrates Empathy, and Communicates Accurate Information—exhibited only slight improvements, with no significant differences. Possible reasons include an insufficient 60-min initial skill development session, insufficient clinical practice sessions, patient variability, differences in instructional quality, and the dynamic nature of clinical situations. These reasons may have made it more challenging for the students to significantly improve these skills within the relatively short duration of their fieldwork training.

Instructor Ratings

On the basis of the ratings from instructors, the Gathers Information and Provides Closure domains demonstrated significant improvements after the FR-CT, with small to medium effect sizes. However, the students’ performance in the other seven CS domains and overall CS performance maintained or showed trends of improvement, although with no significant differences. One study that used feedback and self-reflection to improve students’ overall CS performance reported small and moderate effect sizes (Schüler et al., 2018), partially supporting our findings. The improvement in the Gathers Information domain may be attributed to students’ primary interview tasks, such as identifying the chief complaint and collecting occupational history, which are practiced more frequently and are thus more likely to show enhancement. The improvement in the Provides Closure domain may be due to students’ lower pretest scores in this area, indicating greater potential for enhancement. However, the limited improvement in the other seven domains highlights the need for further research to clarify the underlying reasons.

Student Ratings

According to students’ self-ratings, the overall CS performance and the five CS domains—Gathers Information, Understands the Patient’s and Family’s Perspective, Reaches Agreement, Provides Closure, and Communicates Accurate Information—showed significant improvements, with small to large effect sizes. One study that used simulated patient interviews, self-reflection, and multiperspective feedback found significant self-assessed improvements in gathering information, providing closure, and overall CS performance (Engerer et al., 2016). These results can be attributed to the integration of self-reflection through self-assessment, along with the use of multirater feedback to assist students in the self-reflection process, enabling them to gain diverse perspectives and information. Additionally, the students may have found it easier to adjust their skills after receiving timely feedback on their performance. As such, the FR-CT may help students improve their self-efficacy in these five CS domains as well as in overall CS performance.

Patient Ratings

The patient ratings showed no significant improvements in the students’ performance across all CS domains. A possible explanation is that the pretest scores of the students’ CS performance from the patients were already high. Therefore, although the posttest scores improved, a significant difference was not attainable. It is interesting that some previous studies have reported improved CS (Bourget et al., 2018; Little et al., 2015), whereas others have not (Marsh et al., 2021; Patel et al., 2019). Currently, limited research presents inconsistent findings regarding patient assessment outcomes, which might be affected by patients’ diagnoses and cognitive abilities. Thus, further investigation is needed to determine the effect of CS training on patient assessments.

Different Perspectives Among Experts, Instructors, Students, and Patients

This study revealed varying results depending on the perspectives of the three different groups of raters. Specifically, the experts reported significant improvements in overall CS performance and four CS domains, whereas instructors found significant improvements in only two of those domains. This variation may be attributed to differences in evaluation methods. Instructors assessed performance through onsite observation, which may be limited by short-term memory and preconceived impressions of the students, potentially underestimating or overestimating the students’ CS. Conversely, the experts assessed students’ performance solely on transcripts, so certain aspects, such as relevant nonverbal skills, were not considered. Therefore, the different evaluation methods may have contributed to the inconsistent results. Moreover, the students’ self-assessments showed significant improvements in both the overall CS performance and five CS domains. Students’ pretest scores were initially low, falling below half of the maximum score. As students perceived improvements in their CS, these gains were likely reflected in their higher posttest self-reported scores. In contrast, patients’ ratings showed no significant improvement, possibly because their pretest scores exceeded three-quarters of the maximum score, limiting potential gains. Additionally, patients might not have been as sensitive to incremental changes in students' CS, making it harder for improvements to be reflected in the posttest scores. These discrepancies highlight the importance of considering different raters’ perspectives and the potential impact of their preconceptions and expectations. Each group of evaluators contributes a unique perspective to CS evaluation.