Inter- and Intrarater Reliability of the Gap–Kalamazoo Communication Skills Assessment Form Among Occupational Therapy Interns

Abstract

Importance: Effective communication skills (CS) are essential for occupational therapists. The Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF) is a standard tool for assessing the CS of medical residents. However, the interrater reliability for the nine CS domain scores ranges from poor to good. The intrarater reliability remains unclear.

Objective: To examine the inter- and intrarater reliability of the GKCSAF’s nine domain scores and total score among occupational therapy interns.

Design: Repeated assessments with the GKCSAF.

Setting: Medical center psychiatry department.

Participants: Twenty-five interns and 49 clients with mental illness, recruited from August 2020 to December 2021.

Outcomes and Measures: The transcripts of 50 evaluation interviews between clients and interns were used. Three independent raters assessed each transcript twice, at least 3 mo apart.

Results: The GKCSAF demonstrated poor interrater reliability for the nine domain scores (weighted κ = .08–.30) and the total score (intraclass correlation coefficient [ICC] = .22, 95% confidence interval [CI] [.10, .35]). The GKCSAF showed poor to intermediate intrarater reliability for the nine domain scores (weighted κ = .27–.73) and fair reliability for the total score (ICC = .69, 95% CI [.60, .77]).

Conclusions and Relevance: The GKCSAF demonstrates poor interrater reliability and poor to intermediate intrarater reliability for the nine domain scores. However, it demonstrates fair intrarater reliability in assessing the overall CS performance of occupational therapy interns. Significant variations were observed when different raters assessed the same interns’ CS, indicating inconsistencies in ratings. Consequently, it is advisable to conservatively interpret the CS ratings obtained with the GKCSAF.

Plain-Language Summary: It is essential for occupational therapists to effectively communicate with clients. The Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF) is a standard tool that is used to assess the communication skills of medical residents. The study authors used the GKCSAF with occupational therapy interns in a medical center psychiatry department to assess how effectively they interviewed clients with mental illness. This study aids occupational therapy personnel in the interpretation of GKCSAF results. The study findings also highlight the importance of developing reliable and standardized measures to assess communications skills in the field of occupational therapy.

The authors used the Gap–Kalamazoo Communication Skills Assessment Form with occupational therapy interns in a medical center psychiatry department to assess how effectively they interviewed clients with mental illness.

Effective communication skills (CS) are crucial in psychiatric settings because they influence the efficacy of therapy (Shea, 2017). In psychiatry, the success of therapy is intimately linked to adopting a person-centered approach. This approach involves clients with mental illness in their care, respecting their perspectives and including them in decision-making processes (O’Donohue et al., 2023). By using strong CS, occupational therapy personnel are better equipped to meet these person-centered needs. These skills are essential for building solid therapeutic relationships, accurately understanding client information, and facilitating mutual understanding (Ha & Longnecker, 2010). Therefore, CS are indispensable for all occupational therapy personnel, particularly in psychiatric care.

Enhancing CS is essential in the formative education of occupational therapy personnel. Effective communication training, paired with thorough assessment, facilitates the identification of specific areas in which interns need improvement (Kluger & DeNisi, 1996). Through assessment, instructors can gain insights into interns’ specific CS strengths and weaknesses, enabling them to create learning goals and plans (Lai et al., 2020). Consequently, the cycle of assessment and feedback cultivates a learning environment conducive to developing adept CS among interns, thereby facilitating the improvement of communication efficacy.

A good CS assessment tool is crucial to assessing interns’ level of CS ability and identifying areas for improvement. The Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF) is a widely used measure in this regard (Calhoun et al., 2010). It consists of nine CS domains and offers a comprehensive assessment of CS capabilities in occupational therapy. Therefore, the GKCSAF has potential to provide a thorough evaluation of interns’ CS, aiding occupational therapy personnel and clinical instructors in enhancing interns’ CS.

Furthermore, the GKCSAF can be effectively used in various settings, such as psychiatric occupational therapy, in which CS are critical to clinical tasks. It guides therapists in applying relevant CS to cater to individual client needs, thereby enhancing the quality of psychiatric occupational therapy. This approach not only improves CS but also contributes to more tailored client care, demonstrating the GKCSAF’s potential to enhance psychiatric occupational therapy practice.

The reliability of CS measures, reflected in reproducibility by the same or different raters (intrarater and interrater reliability, respectively; Sharrack et al., 1999), is key to interpreting CS scores in teaching and clinical settings. The GKCSAF has excellent intrarater reliability (intraclass correlation [ICC] = .94) and good interrater reliability for the total score (ICCs = .80–.83; Amaral et al., 2016; Peterson et al., 2014). It also has good internal consistency (Cronbach’s α = .818; Amaral et al., 2016). However, poor to good interrater reliability was observed for the nine CS domain scores in simulated medical interviews (ICCs = .23–.82 in Amaral et al., 2016; ICCs = .53–.80 in Peterson et al., 2014).

In the context of occupational therapy, the GKCSAF has four reliability challenges. First, intrarater reliability validation across its nine CS domains is absent. Second, the interrater reliability outcomes for these domains show inconsistency. Third, the measure has not undergone reliability testing specifically for health professionals other than medical residents, including for occupational therapy personnel. Last, there is a lack of evidence supporting its reliability when conducting client interviews in actual clinical settings. These unresolved issues with reliability potentially affect the GKCSAF’s generalizability and interpretability for occupational therapy personnel.

This study examined the intra- and interrater reliability of the nine CS domain scores and the GKCSAF total score when assessing occupational therapy interns’ interviews with clients with mental illness. The findings of this study will be instrumental for occupational therapy instructors in determining the GKCSAF’s effectiveness in assessing the CS of interns in real-world clinical contexts.

Method

Participants

Participants, including occupational therapy interns and clients with mental illness, were recruited from August 2020 to December 2021 via convenience sampling at the psychiatry department of Kaohsiung Chang Gung Memorial Hospital. This medical center hosts approximately 40 occupational therapy interns annually in its psychiatry department, which includes four wards with 333 beds dedicated to mental health treatment. The inclusion criteria for occupational therapy interns were as follows: (1) a minimum age of 20 yr, (2) completion of at least 3 yr university or junior college education in occupational therapy, and (3) expressed willingness to participate in the study. The inclusion criteria for clients with mental illness were as follows: (1) a diagnosis of a mental disorder by attending physicians, as per the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; American Psychiatric Association, 2013); (2) a minimum age of 20 yr; and (3) stable physical and mental condition, along with basic oral CS, as assessed by their occupational therapist. Clients were excluded if they exhibited severe cognitive impairments, specifically an inability to follow instructions involving three or more steps as assessed by their occupational therapist.

This study was reviewed and approved by the medical center’s Institutional Review Board (202001194B0C501). All participants provided written consent.

Procedure

Each intern conducted three audio-recorded evaluation interviews with different clients, spaced at 3-wk intervals to align with their clinical rotations. These interviews occurred in the 2nd to 4th wk, 5th to 8th wk, and 9th to 12th wk of their 12-wk internship, allowing interns to adapt to new wards and thoroughly understand their clients’ profiles before each interview session. The interviews were conducted in a quiet, undisturbed, closed counseling room; each lasted approximately 30 min. Each intern was supervised by the occupational therapy instructor during the interviews. During the interviews, nine instructors (two men, seven women) with an average age of 38.7 yr and more than 3 yr of teaching and clinical experience observed and audio-recorded the interns’ CS performance without intervening. After the interviews, they provided feedback on the interns’ performance. Clients were selected for interview by the instructors, and these clients also received occupational therapy from the same interns and instructors.

Before the audio-recorded evaluation interviews (i.e., at the start of the internship in the first week), every intern underwent general clinical training, including occupational therapy assessment and treatment. All interns attended a 60-min orientation session, led by the researcher (Tzu Ting Chen). This orientation session introduced the CS domains of the GKCSAF and familiarized interns with the goals of the evaluation interview. The interview primarily aimed to gather clients’ main concern, personal information (e.g., medical history, occupational performance history, academic and work status), and subjective functions (e.g., physiological, psychological, social, and cognitive). Secondary goals included educating clients about occupational therapy, setting treatment goals and plans, and facilitating discussion of their acceptance of the treatment goals and plans.

After the interns completed three interview sessions, the audio recordings were converted to 84 interview transcripts. Transcription was performed by a professional transcription company to ensure a precise and comprehensive representation of the interviews. Subsequently, the transcripts were reviewed and verified by the principal investigator (Tzu Ting Chen) for accuracy and completeness in representing the interview process.

Fifty of the 84 interview transcripts were randomly assigned to three trained raters, who had an average age of 29.0 yr and an average 7.7 yr of experience as therapists. These raters worked in psychiatry departments at two medical centers in Taiwan and had 1 to 2 yr of prior GKCSAF rating experience. The raters were not acquainted with the interns or the clients who participated in our study. Using the GKCSAF, each rater twice independently evaluated interns’ CS on the basis of the 50 interview transcripts, at least 3 mo apart. This 3-mo interval between evaluations was implemented to ensure the reliability of the rater assessments and minimize potential bias. It provided sufficient time to diminish recall of the first evaluation, thereby reducing recall bias and enhancing the independence of the second evaluation.

Before they performed a formal evaluation, these raters were introduced to the GKCSAF’s content and scoring by Hsieh in a 30-min session. To enhance scoring consistency, they independently assessed three transcripts, dedicating 30 min to an hour to each. Raters then discussed each transcript for 1.5 hr with Hsieh, focusing on aligning scoring and achieving consensus. This process was designed for unbiased and accurate evaluations.

After the training, raters immediately commenced assessment of the 50 interview transcripts. A sample size of 50 was used, based on COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) guidelines (Mokkink et al., 2010), which suggest that this number is sufficient to provide the study with adequate power.

Measure

The GKCSAF includes nine CS domains: Builds a Relationship, Opens the Discussion, Gathers Information, Understands the Patient’s and Family’s Perspective, Shares Information, Reaches Agreement, Provides Closure, Demonstrates Empathy, and Communicates Accurate Information (Makoul, 2001). Each domain has two to five CS descriptions (items) for raters’ scoring reference and is rated on a 5-point Likert-type scale (ranging from 1 to 5), with the total score of the GKCSAF ranging from 9 to 45. A higher score indicates higher overall CS (Calhoun et al., 2010). This study used the clinician/faculty version of the GKCSAF (Makoul, 2001). In addition to the original English version, we collaborated with the original developers to translate the GKCSAF into Chinese and culturally adapt it for the Taiwanese population. Moreover, we have examined the face validity of the Chinese version of the GKCSAF. The Chinese version demonstrated preliminary face validity, indicating that it is perceived as a relevant and appropriate tool for assessing CS in the Taiwanese health care context.

Data Analysis

Interrater Reliability

Interrater reliability of the nine GKCSAF CS domain scores and the total score was examined. Weighted κ and percentage of agreement were used to examine agreement among the three raters on the nine CS domain scores. For each pair of raters (i.e., 1–2, 2–3, and 1–3), the weighted κ was calculated, and the three pairwise weighted κ values were averaged (Conger, 1980). Weighted κ values less than .40 were considered poor agreement; .40 to .75, intermediate agreement; and greater than .75, excellent agreement (Armitage et al., 2008). A high percentage of agreement indicated better inter- and intrarater reliability (Hayes & Hatch, 1999).

ICCs and 95% confidence intervals (CIs) were used to examine agreement on total scores among the three raters. ICC values of .70 to .79, .80 to .89, and .90 to .99 were considered fair agreement, good agreement, and high agreement, respectively (Koo & Li, 2016).

The differences in the nine CS domain scores and total scores were also compared among the three raters. The Kruskal–Wallis test and post hoc analysis of Dunn’s test were used to analyze the differences in the nine CS domain scores among the three raters and between each pair of raters. One-way analysis of variance, random effects modeling, and post hoc analysis (Tukey’s honestly significant difference test) were conducted to assess the differences in the total scores among the three raters and between each pair of raters. A p < .05 was indicative of significance. For data analysis, IBM SPSS Statistics (Version 26.0) was used.

Intrarater Reliability

Intrarater reliability of the nine GKCSAF CS domain scores and total score was examined, focusing on two assessment sessions rated by the same raters. The domain-level and total score–level analyses were the same as those for interrater reliability. The differences in the domain scores and total score between the two assessment sessions were also examined. The Wilcoxon signed-rank test and paired t tests were performed to assess the differences in the nine CS domains and total scores between the two assessment sessions. The statistical standards for weighted κ values, ICC values, and p values were identical to those used for interrater reliability.

Other Psychometric Properties

Internal consistency was investigated using Cronbach’s α. Minimal detectable change (MDC) was used to examine the variations in scores caused by random measurement error on the two assessment sessions. The MDC was calculated from the standard error of measurement (SEM) using the following formulas: $M D C = 1.96 \times \sqrt{2} \times S E M$

and $S E M = S D_{all assessment scores} \times \sqrt{(1 - ICC)}$

Results

Sample Characteristics

The 50 randomly selected interview sessions were completed by 25 interns and 49 patients (one client was interviewed by two different interns). The average duration of each interview session was approximately 33 min. The detailed demographic and clinical characteristics of the participants are presented in Table 1.

Table 1.

Demographics and Clinical Characteristics of the Interns and Patients

Characteristic	n (%)
Interns (n = 25)
Age, yr, M (SD)	21.3 (1.1)
Sex
Male	5 (20.0)
Female	20 (80.0)
Patients (n = 49) a
Age, yr, M (SD)	44.0 (14.3)
Sex
Male	31 (63.3)
Female	18 (36.7)
Diagnosis
Schizophrenia	34 (69.4)
Bipolar and depressive disorder	12 (24.5)
Other psychiatric disorder	3 (6.1)

^a One patient was interviewed twice by two different interns; the repeated demographic data were deleted.

The median scores for the nine CS domains ranged from 3.0 to 4.0, as rated by the three raters in two assessment sessions. The mean total score ranged from 29.2 to 31.4, indicating that the interns, on average, achieved approximately 65% to 70% of the total score (Table 2).

Table 2.

Raters’ Scores on the Nine GKCSAF CS Domains and Total Score

Domain	Mdn (Q1–Q3)
	Session 1			Session 2
	Rater 1	Rater 2	Rater 3	Rater 1	Rater 2	Rater 3
1. Builds a Relationship	4.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (4.0–4.0)	3.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (4.0–4.0)
2. Opens the Discussion	4.0 (3.0–4.3)	4.0 (3.0–4.0)	4.0 (4.0–4.0)	4.0 (3.0–4.0)	3.0 (3.0–4.0)	4.0 (4.0–4.0)
3. Gathers Information	3.0 (3.0–4.0)	3.0 (3.0–4.0)	4.0 (4.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (4.0–4.0)
4. Understands the Patient’s and Family’s Perspective	4.0 (3.0-5.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)
5. Shares Information	4.0 (3.0–4.0)	3.0 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	3.0 (3.0–4.0)	4.0 (3.0–4.0)
6. Reaches Agreement	3.0 (2.0–3.3)	3.0 (3.0–4.0)	3.0 (3.0–4.0)	3.0 (2.0–4.0)	3.0 (3.0–4.0)	3.0 (3.0–4.0)
7. Provides Closure	3.0 (2.0–4.0)	3.0 (2.0–4.0)	3.0 (3.0–4.0)	3.0 (1.0-4.0)	3.0 (2.0–4.0)	3.0 (3.0–4.0)
8. Demonstrates Empathy	3.5 (3.0–4.0)	4.0 (3.0–4.0)	4.0 (3.0–4.0)	3.0 (3.0–4.0)	4.0 (3.0–4.0)	3.0 (3.0–4.0)
9. Communicates Accurate Information	3.0 (3.0–4.0)	3.0 (2.0–4.0)	3.0 (3.0–4.0)	3.0 (2.0–4.0)	3.0 (2.0–3.3)	3.0 (3.0–3.0)
Total score, M (SD)	30.8 (6.1)	30.4 (5.7)	31.4 (4.2)	29.2 (5.3)	30.9 (6.1)	31.1 (3.8)

Note. CS = communication skills; GKCSAF = Gap–Kalamazoo Communication Skills Assessment Form; Mdn = median; Q = quartile.

Interrater Reliability

All the CS domains had poor agreement (weighted κ = .08–.30). The domains Demonstrates Empathy, Shares Information, and Communicates Accurate Information had the lowest agreement (weighted κs = .08, .10, and .11, respectively), and the domains Reaches Agreement, Provides Closure, and Understands the Patient’s and Family’s Perspective had fair but the highest agreement, relatively (weighted κs = .30, .28, and .22, respectively). The percentage of agreement of the nine CS domains was quite low (3.0%–19.0%). The agreement of the total scores was poor (ICC = .22, 95% CI [.10–.35]; Table 3).

Table 3.

Agreement of the Nine GKCSAF CS Domain Scores and Total Score

Domain	Interrater Reliability		Intrarater Reliability
Domain	Weighted κ	% Agreement	Weighted κ	% Agreement
1. Builds a Relationship	.16	15.0	.58	63.3
2. Opens the Discussion	.14	19.0	.37	58.0
3. Gathers Information	.15	13.0	.27	56.0
4. Understands the Patient’s and Family’s Perspective	.22	19.0	.27	60.0
5. Shares Information	.10	19.0	.57	69.3
6. Reaches Agreement	.30	18.0	.56	58.0
7. Provides Closure	.28	3.0	.73	59.3
8. Demonstrates Empathy	.08	19.0	.51	57.3
9. Communicates Accurate Information	.11	12.0	.68	60.7
Total score, ICC (95% CI)	.22 [.10, .35]		.69 [.60, .77]

Note. CI = confidence interval; CS = communication skills; GKCSAF = Gap–Kalamazoo Communication Skills Assessment Form; ICC = intraclass correlation.

There were no significant differences between the three raters’ scores on the Understands the Patient’s and Family’s Perspective and Communicates Accurate Information domains (p ≥ .147), whereas there were significant differences on the other domains (p ≤ .023). The largest differences in these CS domain scores were found between Raters 1 and 3. Regarding the total scores, there were significant differences among the three raters (p < .001), especially between Raters 1 and 2 and between Raters 1 and 3 (p < .050; Table A.1 of the Appendix). In the two assessment sessions, Rater 1 had the lowest total score (M = 29.2, SD = 5.3) and Rater 3 had the highest (M = 31.4, SD = 4.2; Table 2).

Intrarater Reliability

All the CS domains had poor to intermediate agreement (weighted κs = .27–.73). The CS domains Opens the Discussion, Gathers Information, and Understands the Patient’s and Family’s Perspective had poor agreement between the two assessment sessions (weighted κs = .27–.37), and the other six CS domains had intermediate agreement (weighted κs = .68–.73). Additionally, the percentage of agreement of the nine CS domains was acceptable (56.0%–69.3%). The agreement of the total scores was fair (ICC = .69, 95% CI [.60–.77]; Table 3).

There were no significant differences in the eight CS domains (p ≥ .101), except for the domain Demonstrates Empathy (p = .035). There were no significant differences in total scores between the two assessment sessions (p = .139; Table A.1).

Weighted κ scores for Raters 1–2, 1–3, and 2–3 ranged from poor to fair (weighted κs = .08–.44). Demonstrates Empathy and Communicates Accurate Information were notably low, and Reaches Agreement and Provides Closure scored highest (Appendix Table A.2). ICC values across all pairs were suboptimal (ICC = .19–.47, 95% CI [.09–.55]; Table A.2).

Other Psychometric Properties

The GKCSAF showed good internal consistency (Cronbach’s α = .85). The SEM and MDC for interrater reliability of the total score were 5.1 and 17.2 points, respectively; for intrarater reliability, they were 0.3 and 0.8 points, respectively (Appendix Table A.3). The nine CS domains exhibited minimal floor (1.0%–9.7%) and ceiling (5.7%–11.0%) effects (Appendix Table A.4).

Discussion

Our study showed poor interrater reliability for the nine CS domains of the GKCSAF (weighted κs = .08–.30; agreement percentage = 3.0%–19.0%). Seven of the nine CS domains showed significant differences in raters’ scoring. Furthermore, the total score in our study also revealed poor interrater reliability (ICC = .22), with significant difference observed among the three raters. The results indicate that was a notable inconsistency in raters’ evaluations, both across the individual domains and in the total score.

These findings were inconsistent with those of two previous studies that reported poor to good interrater reliability for the nine CS domains (ICCs = .23–.82 in Amaral et al. [2016]; ICCs = .53–.80 in Peterson et al. [2014]). However, in previous studies, the raters evaluated medical residents’ CS in simulated interviews with standardized patients. In such structured environments, predefined scripted interviews may promote uniform communication behaviors, helping raters to focus on expected behaviors and facilitating a more consistent assessment. Consequently, structured interviews in simulated settings might result in higher reliability in the evaluation of residents’ CS.

Three possible factors contribute to the inadequate interrater reliability of the nine CS domains. First, raters may have differing perspectives on the importance of the GKCSAF’s CS items. Given that each domain contains two to five behavioral descriptions, raters might prioritize certain descriptions over others, influencing their scoring decisions and affecting the overall CS domain scores. Second, variability in rater leniency or strictness could lead to inconsistent ratings. Some raters may be more lenient or stringent in their evaluations, contributing to score discrepancies. Last, the GKCSAF’s rating scale may contribute to the issue because of its lack of detailed definitions for each behavioral description. This ambiguity can result in varied interpretations among raters.

Despite providing training to raters, these challenges persist. To address these issues, future modifications should include providing explicit definitions for each item and behavioral description in the GKCSAF, along with establishing clear scoring guidelines. Implementing a more structured scoring approach, such as assigning specific weighted scores to particular descriptions or setting a fixed number of points for certain behaviors, could mitigate these discrepancies. Such refinements are expected to reduce variance in ratings and thereby improve the GKCSAF’s interrater reliability.

The GKCSAF demonstrated poor interrater reliability for both the nine CS domains and the total scores, indicating that relying on a single rater’s scores is not likely to produce reliable results. To address this issue, future studies may consider using multiple raters’ scores as the final score. According to a previous study, at least three raters were suggested to achieve more stable results on the GKCSAF (Murray et al., 2019). However, this requires further empirical evidence to substantiate. Adequate training for raters is also important to improve interrater reliability, particularly for the CS domains with poor interrater reliability.

Our study showed poor to intermediate intrarater reliability for the nine CS domains (weighted κs = .27–.73), with an agreement percentage of 56.0% to 69.3%. There were no significant differences in eight of the nine CS domains. To our knowledge, our study is the first to present information about the intrarater reliability of the GKCSAF’s nine CS domains. It provides preliminary evidence that the GKCSAF has acceptable intrarater reliability for six CS domains. Thus, the results for these six CS domains indicate that they are adequate for use in both clinical and research contexts. However, the results for the other three CS domains with low intrarater reliability should be interpreted conservatively.

Our study showed fair intrarater reliability for the GKCSAF total score (ICC = .69). There were no significant differences in total scores between the two assessment sessions. One prior study reported high intrarater reliability (ICC = .94) in simulated medical interviews (Amaral et al., 2016). However, the interval between their assessment sessions is unknown. These inconsistent results may be due to a more structured context (simulated interview), shorter interview times (an average of 7.5 min), fewer CS contents that needed to be evaluated, and less random error than in the evaluation process in the previous study (Amaral et al., 2016). Thus, the intrarater reliability in the Amaral et al. (2016) study may be overestimated and should be examined further. Additionally, our study involved occupational therapy interns and real clients with mental illness, which were more specific and better reflected actual clinical situations. Our results suggest that GKCSAF total scores from the same rater provide consistent results.

This study has four limitations. First, our study used convenience sampling. The potential sampling bias may have affected the results and, thereby, their generalizability. Second, the involvement of instructors in selecting clients may have introduced selection bias. Such biases may affect the representativeness of the sample, thereby posing challenges to generalizing the study findings to other psychological settings. Although measures were taken to minimize these effects, future research might consider alternative methods to further reduce the potential for such biases. Third, our study lacks data on clients’ admission duration and did not use standardized assessment to confirm clients’ communication abilities, hindering a thorough analysis of their impact on our findings. Future research should gather these data to improve understanding. Fourth, the CS contexts of our study focused only on evaluation interviews. Therefore, the reliability of the GKCSAF in other CS contexts, such as treatment, remains unknown. Last, the raters rated the interns’ CS solely on the basis of the interview transcripts, without considering nonverbal CS. Therefore, to further validate our results, future studies should provide raters with video recordings to assess occupational therapy personnel’s CS.

Implications for Occupational Therapy Practice

The study revealed poor interrater reliability for the GKCSAF, suggesting that different raters may evaluate the CS domains and overall performance of the same intern differently. However, rater training is crucial to align raters’ understanding and application of assessment criteria. Rater training promotes scoring consistency, reduces subjectivity, and enhances reliability and validity, ensuring a standardized evaluation process, thereby improving overall assessment quality and credibility. Therefore, standardizing the training and practice of raters is crucial to enhance the GKCSAF’s reliability. However, we found acceptable intrarater reliability, indicating that rating by the same rater are consistent across multiple trials or sessions. Hence, occupational therapists should aim to use the same raters when evaluating interns’ CS to ensure consistency in results.

Conclusion

The interrater reliability of the GKCSAF in assessing the nine CS domains among occupational therapy interns was poor, indicating substantial variations in how different raters evaluate the same intern. However, intrarater reliability for overall CS performance was fair, suggesting more consistency over time by the same rater. These findings emphasize the necessity for cautious interpretation of the CS ratings obtained through the GKCSAF and highlight the importance of reliable and standardized measures in assessing CS within the field of occupational therapy.

Footnotes

Acknowledgments

We acknowledge all the occupational therapists in the Department of Psychiatry, Kaohsiung Chang Gung Memorial Hospital, who assisted with data collection. This trial was supported by a Kaohsiung Chang Gung Memorial Hospital Research Grant (CMRPG8M0811) and a grant from the National Science and Technology Council (110-2511-H-002-016-MY2). San-Ping Wang and Ching-Lin Hsieh contributed equally to this work and serve as corresponding authors.

Appendix

Table A.4.

Floor and Ceiling Effect of the GKCSAF

Domain	%
Domain	Floor Effect	Ceiling Effect
1. Builds a Relationship	1.0	8.0
2. Opens the Discussion	1.0	8.3
3. Gathers Information	2.0	10.7
4. Understands the Patient’s and Family’s Perspective	0.3	11.0
5. Shares Information	0.7	6.7
6. Reaches Agreement	6.3	5.7
7. Provides Closure	9.7	8.3
8. Demonstrates Empathy	1.0	5.7
9. Communicates Accurate Information	4.7	10.0

Note. Floor and ceiling effects were calculated by determining the percentage of interns scoring the lowest and highest possible scores on the assessment tool, respectively. GKCSAF = Gap–Kalamazoo Communication Skills Assessment Form.

References

Amaral

A. B. C. N.

Rider

E. A.

Lajolo

P. P.

Tone

L. G.

Pinto

R. M. C.

Lajolo

M. P.

, & Calhoun

A. W.

(2016). Development of a Brazilian Portuguese adapted version of the Gap-Kalamazoo Communication Skills Assessment form. International Journal of Medical Education, 7, 400–405. https://doi.org/10.5116/ijme.583a.df42

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).

Armitage

Berry

, & Matthews

J. N. S.

(2008). Statistical methods in medical research. Wiley.

Calhoun

A. W.

Rider

E. A.

Peterson

, & Meyer

E. C.

(2010). Multi-rater feedback with gap analysis: An innovative means to assess communication skill and self-insight. Patient Education and Counseling, 80, 321–326. https://doi.org/10.1016/j.pec.2010.06.027

Conger

A. J.

(1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322–328. https://doi.org/10.1037/0033-2909.88.2.322

J. F.

, & Longnecker

(2010). Doctor-patient communication: A review. Ochsner Journal, 10, 38–43.

Hayes

J. R.

, & Hatch

J. A.

(1999). Issues in measuring reliability: Correlation versus percentage of agreement. Written Communication, 16, 354–367. https://doi.org/10.1177/0741088399016003004

Kluger

A. N.

, & DeNisi

(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254–284. https://doi.org/10.1037/0033-2909.119.2.254

Koo

T. K.

, & Li

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

10.

Lai

M. M. Y.

Roberts

Mohebbi

, & Martin

(2020). A randomised controlled trial of feedback to improve patient satisfaction and consultation skills in medical students. BMC Medical Education, 20, 277. https://doi.org/10.1186/s12909-020-02171-9

11.

Makoul

(2001). Essential elements of communication in medical encounters: The Kalamazoo consensus statement. Academic Medicine, 76, 390–393. https://doi.org/10.1097/00001888-200104000-00021

12.

Mokkink

L. B.

Terwee

C. B.

Knol

D. L.

Stratford

P. W.

Alonso

Patrick

D. L.

et al. de Vet, H. C. (2010). The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology, 10, 22. https://doi.org/10.1186/1471-2288-10-22

13.

Murray

Hall

Williams

G. C.

McDonough

S. M.

Ntoumanis

Taylor

et al. Matthews, J. (2019). Assessing physiotherapists’ communication skills for promoting patient autonomy for self-management: Reliability and validity of the Communication Evaluation in Rehabilitation tool. Disability and Rehabilitation, 41, 1699–1705. https://doi.org/10.1080/09638288.2018.1443159

14.

O’Donohue

Aguey-Zinsou

Yule

Fairhurst

, & Debaets

(2023). Mental health consumer perspectives of a person-centred multidisciplinary care planning meeting on a rehabilitation inpatient unit. International Journal of Mental Health Nursing, 32, 1701–1712. https://doi.org/10.1111/inm.13192

15.

Peterson

E. B.

Calhoun

A. W.

, & Rider

E. A.

(2014). The reliability of a modified Kalamazoo Consensus Statement Checklist for assessing the communication skills of multidisciplinary clinicians in the simulated environment. Patient Education and Counseling, 96, 411–418. https://doi.org/10.1016/j.pec.2014.07.013

16.

Sharrack

Hughes

R. A.

Soudain

, & Dunn

(1999). The psychometric properties of clinical rating scales used in multiple sclerosis. Brain, 122, 141–159. https://doi.org/10.1093/brain/122.1.141

17.

Shea

S. C.

(2017). Psychiatric interviewing: The art of understanding. Elsevier.