Abstract
The concept of continuing competence in health care encompasses clinical knowledge and skills, professional behavior, judgment, reflection, evidence-based principles, and ethics (Epstein & Hundert, 2002). It is generally accepted that professional development through continuing education supports continuing competence of health care providers. Assessment, in some capacity, has been promoted as a way to demonstrate the effects of professional development on continuing competence. Specifically, the Institute of Medicine (2003) recommended that “health professions boards should move toward requiring licensed health professionals to demonstrate periodically their ability to deliver patient care . . . through direct measures of technical competence, patient assessment, evaluation of patient outcomes, and other evidence-based assessment methods” (p. 7).
The recommendation for competence assessment of health care providers is not unique. A recent poll found that most Americans favor periodic assessment of health care professionals (National Board of Certification and Recertification of Nurse Anesthetists, 2013). Groups such as the Citizen Advocacy Center (CAC; 2011) and AARP (Swankin et al., 2006) have also advocated for increased regulation and more rigorous methods of determining recertification, including third-party assessment. Although assessment may seem prudent, U.S. health care providers are historically resistant to practice-based and written assessments as a means for recertification for reasons such as the lack of validity, limited research, and high cost of assessment for continuing competence (CAC, 2004, 2011; Doroghazi, 2015; Sandhu et al., 2015; Teirstein & Topol, 2015; Zambricki et al., 2015).
Continuing Competence Assessment in Occupational Therapy
Most U.S. occupational therapy practitioners live in states where professional development is required to maintain licensure, and a large majority of practitioners maintain their voluntary certification through the National Board for Certification in Occupational Therapy (NBCOT®), which also requires professional development to recertify (American Occupational Therapy Association [AOTA], 2015a; NBCOT, 2015a). However, there is no mandatory assessment of continuing competence for occupational therapy in the United States. Occupational therapists are expected to self-assess their strengths and deficits regarding knowledge, skills, and attributes for quality service delivery (AOTA, 2015b; NBCOT, 2015a); there are concerns about relying on self-assessment to determine continuing competence needs. Accurate self-assessment of learning needs is lacking across all types of learners and different professionals; practitioners and students tend to overestimate their abilities and underestimate their limitations (CAC, 2011; Dunning et al., 2004).
The NBCOT Navigator® online platform provides virtual continuing competence assessment and professional development tools and activities for certified occupational therapy practitioners (NBCOT, 2015b). The platform uses serious games (e.g., games played for assessment or learning) with the intent of increasing participants’ motivation during game play (McNamara et al., 2015). Once logged into Navigator, participants access a list of recommended assessment tools based on their profiles and responses to a self-reflection questionnaire. Participants then engage in assessment of client-centered and evidence-based practice (EBP) knowledge and skills using several different types of tools for specific practice areas, such as mini practice quizzes, case simulations, match games, and a four-module set designed to teach therapists how to develop an answerable clinical question. Therapists receive feedback in the form of a performance quartile or a score depending on the activity.
Professional development units (PDUs) are awarded for completion of Navigator activities when scoring is above the median score of the therapist’s peers. EBP is supported through the provision of research resource lists that are individually tailored to areas of need on the basis of Navigator performance. The resource lists consist of recommended readings (e.g., peer-reviewed publications) or search strings for reference searches. All certified occupational therapy practitioners have free access to the ProQuest (Ann Arbor, MI) Nursing and Allied Health Source and the ProQuest Medline Indexing System as part of their certification renewal, thus giving them access to the resources and allowing unlimited evidence-based literature searches.
Navigator offers an innovative option for occupational therapists’ competence assessment, yet little is known about participants’ perceptions of this new program. Knowing participants’ perceptions of Navigator’s usefulness will provide the groundwork for future research investigating how competence assessment may assist occupational therapy practitioners to identify professional development needs and implement changes in their practice regarding those needs. The purpose of this study was to explore Navigator participants’ perceptions of the online competence assessment tools and evidence-based resources. The following research questions guided the study:
How do occupational therapists use Navigator’s online competence assessment tools and evidence-based resources?
What are occupational therapists’ perceptions of how participation in an online competence assessment and use of Navigator’s evidence-based resources influence practice?
What are the relationships between occupational therapists’ perceptions of participation in an online competence assessment and two variables: years working as an occupational therapist and highest professional degree?
Method
The study used a cross-sectional research design with survey methodology. Approval was received from the University of Florida institutional review board. The reported findings are part of a larger study investigating research utilization of certified occupational therapists, including research utilization behaviors and use of evidence-based practices of therapists who have used Navigator and those who have not.
Participants and Procedure
Of the 35,960 occupational therapists certifying or recertifying with NBCOT in March 2016, 10% (3,596) were randomly selected to receive a recruitment email in May 2016. To obtain data for the larger study, half of the occupational therapists in the group were Navigator participants and half were nonparticipants. An email with a link to the online questionnaire and informed consent was sent to 511 occupational therapists who indicated they were interested in more information. Two additional emails were sent as reminders 2 wk and 4 wk later. As an incentive, participants received 1 PDU through NBCOT for completing the study. This report describes the findings from the responses of only the participants who used Navigator.
Instrument
The questionnaire was developed after a review of the literature on continuing and professional competence, EBP, and research utilization. A draft of the questionnaire was reviewed by one occupational therapist with expertise in continuing competence (i.e., an occupational therapy educator with 30 yr of experience) and two researchers with expertise in survey methodology. On the basis of their feedback, the questionnaire was revised and then piloted with six local occupational therapists. The final questionnaire was presented online using the Qualtrics (Provo, UT) Research Suite online survey platform. The questionnaire included 32 items about occupational therapists’ perceptions of Navigator and use of Navigator’s evidence-based resources. The questionnaire had 7 demographic items, 6 dichotomous questions (e.g., “A list of evidence-based resources [e.g., references for journal articles] is provided after completion of some Navigator activities. Did you obtain copies of any of the resources provided in the list?”), 7 open-ended questions (e.g., “Please explain how the Navigator activities helped you to identify where you needed to improve in your research knowledge and/or research skills”), and 12 multiple-choice items using a Likert scale (e.g., “Participating in Navigator activities helped me identify where I need to improve in my clinical knowledge,” “I changed my practice to use more evidence-based approaches after participation in Navigator activities”).
Data Analysis
Data were exported into IBM SPPS Statistics (Version 22; IBM Corp., Armonk, NY) for analysis. Descriptive statistics were used for the demographic, multiple-choice, and multiple-answer items. The χ2 test of independence was performed to examine the relationship between responses on eight perception items (e.g., changes in clinical knowledge, clinical skills, research knowledge, research skills, and use of evidence-based approaches) and two categorical variables: years working as an occupational therapist and highest professional degree. For the analysis, years working as an occupational therapist were grouped into ≤5 yr and ≥6 yr. Likert scale data from perception items were simplified by combining strongly disagree, disagree, and neutral choices into one grouping and agree and strongly agree choices into another grouping. Open-ended questions were analyzed using inductive content analysis (Elo & Kyngäs, 2008). Responses were read line by line and coded separately by the author and a graduate occupational therapy student. An iterative process, which included regular meetings to discuss individual analyses, was used to finalize the codes, group them into subcategories, determine main categories, and develop final themes.
Results
Of the 511 occupational therapists who were sent emails with questionnaire links, 453 responded to the questionnaire, for a response rate of 88.6%. Forty participants were not currently working in the field, leaving a sample of 413 participants. Of these participants, 64.4% (n = 266) indicated that they had used Navigator; data from this sample are reported here. Reported percentages vary according to item response because not all participants responded to every question.
Demographics
Twenty-five participants were initially certified in 2016, with the remainder (n = 241) recertifying. Of those recertifying, the majority held a master’s degree as their highest professional degree (n = 169; 70.1%) and had worked 0–5 yr as an occupational therapist (n = 107; 44.4%). Approximately one-third of recertifying participants identified pediatrics as their practice area (n = 88; 36.5%). The most frequently identified work setting was long-term care or skilled nursing (n = 68; 28.2%). Table 1 contains additional demographic information.
Participant Demographics (N = 241)
Note. Percentages may not total 100 because of rounding.
Perceptions of Online Competence Assessment
A greater percentage of participants agreed or strongly agreed that participating in Navigator activities helped them to identify needed improvements in clinical knowledge (n = 211; 79.9%) and clinical skills (n = 160; 60.8%) than who agreed or strongly agreed that participating in Navigator activities helped them to identify needed improvements in research knowledge (n = 99; 38.8%) and research skills (n = 64; 25.2%). A smaller proportion of participants agreed or strongly agreed that they changed their practice to use more evidence-based approaches after participation in Navigator activities (n = 59; 23.4%) than those who agreed or strongly agreed that they considered making changes to their practice to use more evidence-based approaches after participation in Navigator activities (n = 86; 44.6%). Of participants who indicated that they maintained a professional development plan (n = 154; 57.9%), slightly more than half (n = 79; 51.6%) were likely or extremely likely to modify their plan on the basis of their participation in Navigator activities. The majority of participants stated they were likely or extremely likely to participate in Navigator activities again within the next year (n = 180; 72.3%).
Two significant statistical relationships regarding self-reported changes using Navigator were present. The first relationship was between years working as an occupational therapist and identification of areas of improvement in clinical skills, χ2(1, N = 240) = 6.73, p = .01. Less experienced occupational therapists (i.e., those who had worked 5 yr or less) were more likely to agree that Navigator helped them identify clinical skill areas that needed improvement than experienced occupational therapists (i.e., who had worked 6 yr or longer). The second was the relationship between highest degree and changes to use more EBP approaches, χ2(1, N = 241) = 5.70, p = .017. Participants with bachelor’s degrees as their highest degree were more likely to agree that use of Navigator changed their practice to use more evidence-based approaches than participants with master’s degrees or higher as their highest degree.
Use of Evidence-Based Resources
Of the 98 participants (45.6%) who obtained copies of the evidence-based resources, 72.5% (n = 71) read at least one of the resources. Of those who read, 93.0% (n = 66) indicated the evidence-based resources were useful or extremely useful and 88.7% (n = 63) reported they were likely or extremely likely to apply what they learned from the resources to their practice. Of the 46 participants (22.4%) who conducted a literature search using search terms provided at the end of Navigator activities, 80.4% (n = 37) read at least one of the resources obtained from the literature search. Of those who read, 81.1% (n = 30) responded that the resources were useful or extremely useful to their practice and 78.4% (n = 29) were likely or extremely likely to apply what they learned from the resources to their practice.
Responses to Open-Ended Questions
Content analysis of responses to open-ended questions yielded three themes: reflective learning, self-directed learning, and application of learning.
Reflective Learning.
Participants who agreed or strongly agreed that Navigator activities helped them identify areas of needed improvement in their clinical knowledge, clinical skills, research knowledge, or research skills were asked to explain how the activities were helpful. Responses were primarily focused on how Navigator helped participants identify their strengths and weaknesses regarding knowledge and skills (n = 130), including the use of Navigator as a way to reflect on knowledge and skill needs (n = 25). The following comments illustrate these findings: “Identify if I am staying current to best practice and what may be outdated thinking/knowledge,” “The activities pointed out areas in which I’m lacking current/updated research and evidence-based practice treatment approaches,” and “I was able to identify which questions I didn’t know the answers to and therefore where my weaknesses were for practicing.”
Self-Directed Learning.
Participants described using Navigator as a basis for initiating a learning process in which they reviewed their clinical knowledge and skills (n = 45), enhanced clinical knowledge and skills in their current practice area (n = 28), learned about a different practice area (n = 17), or continued education in weak knowledge and skill areas (n = 16). Obtaining evidence-based resources from Navigator or ProQuest (n = 21), reviewing knowledge or learning new knowledge (n = 19), learning more about EBP (n = 11), and keeping current with evidence (n = 16) were identified as ways in which Navigator helped participants to address research knowledge and skill needs.
Application of Learning.
Applying their learning from Navigator to change aspects of their practice was identified by participants (n = 57). In addition, many participants described incorporating evidence into practice (n = 117) or using readings and resources provided through Navigator as a component of this practice change (n = 63). Participants also expressed feeling successful in their ability to make practice changes associated with their learning (n = 23) or stated that they have shared or were planning to share their learning with colleagues or patients (n = 25).
Discussion
Although a majority of participants agreed that Navigator helped them identify professional development needs for their clinical knowledge and skills, they described less influence of Navigator on identifying improvements needed for research knowledge and skills. Participants possibly engaged more with the clinically focused assessment tools than with those centered on research evidence (i.e., the four-part module that educates participants to develop an answerable clinical question using the PICO [patient, problem, or population; intervention; comparison; outcome] method); however, data on specific assessments were not collected. Previous studies have demonstrated that occupational therapy practitioners who engage in EBP training have significantly improved confidence in EBP (Brangan et al., 2015; Cahill et al., 2015; Szucs et al., 2016) and are more likely to rely on research literature during clinical decision making (Graham et al., 2013). When participants were asked how they had changed or considered changing their practice to use more evidence-based approaches or how they planned to apply learning from the evidence-based resources to their practice, most described gathering, reading, or reviewing the evidence-based literature. These actions support the development of an answerable clinical question, which is a first step to making changes in practice through the use of EBP.
Therapists with less experience were more likely to identify clinical skill needs than therapists with more experience. The reasons for this finding are unknown, but it may be related to the influence of accumulated experience on practitioners' skill levels. Therapists with fewer years of experience may demonstrate decreased flexibility, ability to individualize approaches to intervention, and use of research-based knowledge in their decision making than do experienced colleagues (Kuipers & Grice, 2009). Previous studies have found that more experienced participants are less likely to use research in practice or adopt an EBP approach (Cameron et al., 2005; Dysart & Tomlin, 2002; Hitch, 2015). The conflicting findings across studies may be the result of an overemphasis on experience as an influence in practice expertise, in particular expertise related to EBP. Experience is just one component in the development of expertise, with other factors, such as personal qualities (e.g., motivation) and knowledge, contributing as well (King, 2009).
The finding that therapists with bachelor’s degrees were more likely to change their practice to use more evidence-based approaches than those with master’s degrees or higher may be indicative of the type of education provided to therapists in their entry-level programs. Most therapists with bachelor’s degrees graduated before the early to mid-2000s; after that time there was an increased emphasis on EBP in occupational therapy curricula. It is possible that therapists with bachelor’s degrees may have been more likely to change their practice based on Navigator assessment tools because the EBPs included in the tools were new to them, whereas therapists with master’s degrees or higher learned about those EBPs in their entry-level or postprofessional programs and were already using them. The majority of participants had practiced 5 yr or less, which may be an important factor in determining the appeal of Navigator to occupational therapists. Younger occupational therapists may be more comfortable using technology for competence assessment because they have taken their graduate school entrance exams and NBCOT certification exam on computers and may be more familiar with online assessment than seasoned therapists (Boevé et al., 2015).
Responses to the open-ended questions indicate that Navigator assisted participants to consider the relevance of their practice patterns and intervention choices in relation to current evidence. Previous studies of occupational therapists’ perceptions of EBP and research utilization suggested that therapists generally have positive views of EBP, the use of research to support clinical decisions, and the use of research to support the scientific basis of occupational therapy practice (Cameron et al., 2005; Hitch, 2015; Salls et al., 2009). However, most therapists do not report using research evidence to justify an intervention (Cameron et al., 2005; Graham et al., 2013). Through use of Navigator, occupational therapy practitioners may be exposed to EBPs of which they were unaware, thus initiating the knowledge translation process (i.e., the application of knowledge to practice in context; Cramm et al., 2013). Future studies should explore whether Navigator affects practitioners’ use of evidence in making clinical decisions, including how practitioners translate new EBP knowledge to practice.
Implications for Occupational Therapy Practice
The study findings suggest that Navigator provides skill and knowledge assessment opportunities for occupational therapy practitioners, with the following implications regarding continuing competence related to occupational therapy practice:
Occupational therapy practitioners may benefit from using online competence assessment to understand their strengths and limitations regarding clinical and research knowledge and skills.
Results of online competence assessments may inform professional development plans, which can be modified to address clinical and professional competence needs.
Because the assessment tools provided in Navigator are based on current evidence in occupational therapy, the tools may expose practitioners to EBPs applicable to their practice settings, influencing continuing education choices and possibly initiating the knowledge translation process.
Sharing of new knowledge gained from online competence assessment tools with consumers and colleagues may increase client-centered practices and shared decision making by occupational therapy practitioners.
Limitations
Selection bias is a limitation of this study. Although attempts were made to obtain a random sample, the therapists who participated may have been more interested in participating in an online competence assessment than those who did not. Reliance on self-report may have introduced bias into the survey, because practitioners may have over- or underestimated their behavior. Last, the sample of participants may not be representative of all occupational therapists who use Navigator, because the largest group of participants had 5 yr or less experience.
Conclusion
The study findings suggest that participants perceived that Navigator assisted them in identifying continuing competence needs and, to a lesser extent, evidence-based resources to address those needs. Therapists described using reflection, self-direction, and application of learning to practice in association with the online competence assessment tools. Evidence-based resources were useful to therapists who read them, and the majority of therapists who read the resources indicated that they were likely to apply what they learned to practice. Continued research into the role of Navigator as an online competence assessment platform and potential uses as a dissemination tool for EBPs in occupational therapy is needed.
Footnotes
Acknowledgments
The author thanks graduate student Jennifer Lotz for her assistance in analyzing the open-ended questions. This project was funded by the National Board for Certification in Occupational Therapy.