Abstract
We describe the development and evaluation of a clinical research training program designed specifically for such health professionals as occupational and physical therapists. Outcomes of program success included trainees’ self-rating of research skills before and after the program, as well as submission of a formal grant application to a grant competition for program participants. At program completion, participants reported improvements in their research skills, with the most gain in formulating research questions and writing a testable hypothesis and the least gain in understanding statistics. Of the 21 participants, 43% submitted a grant proposal to a competitive intramural grant program. In the next year, grantees of the program will continue to be mentored by the program mentors while conducting their research projects. Given the initial successes, this program represents a promising model for providing research training to practicing clinicians.
Although third-party payers increasingly demand evidence to support the effectiveness of treatments that health care professionals provide, professions such as occupational therapy lack a sufficient evidence base. Recent national summits and outcome centers urged the development of a research environment in which practitioners would be key collaborators in outcomes research, because their work directly depends on research knowledge and tools developed from research studies (Frontera et al., 2006; Kielhofner, Hammel, Finlayson, Helfrich, & Taylor, 2004). In recognition of the value of practitioners’ knowledge and clinical research questions, our university’s mission endorses providing clinical research training for health professionals. In this article, we discuss the development, implementation, and program outcomes of a novel clinical research training program at the University of Michigan. The program evaluation identifies outcomes of preliminary program successes, potential for improvement and expansion, and a model that could be replicated at other universities.
Background
In 2007, the National Institutes of Health (NIH) awarded a Clinical and Translational Science Award (CTSA) to the University of Michigan. The CTSA provides academic health centers funding to help translate research into practice more quickly (National Center for Research Resources, 2009. One goal of the CTSA is to provide research education and career development training to future clinical researchers from a variety of disciplines, including nurses, dentists, and rehabilitation health professionals (NIH, 2008). Although several NIH mechanisms (such as K awards) exist to train people to become independent scientists, these opportunities are less available to practicing clinicians. In recognition of the potential of clinicians to make significant and meaningful contributions to the evidence on which their practice is based, the University of Michigan resolved to design training for practicing clinicians who could play critical roles in “team science.” In addition, the university, as a whole, and the Department of Physical Medicine and Rehabilitation (PM&R), in particular, emphasized the need to more effectively incorporate research into clinical practice among all practicing clinicians.
Previous PM&R research identified national needs for clinical research training (Frontera et al., 2006; Stucki, 2007) and structured mentoring (Galicia, Klima, & Date, 1997). The University of Michigan’s PM&R Department held goals for continuing education for its staff and had concurrently instituted other measures to facilitate clinical research, such as including standardized outcome measures into treatments and keeping computerized records for future data analyses. Consistent with these institutional and departmental efforts, we sought to develop a program to provide clinical research training to clinicians (such as occupational therapists and physical therapists) currently practicing in the university’s hospitals and clinics who may not have had previous experience with research. The steps we took to develop and evaluate the program included (1) determining the perceived need for clinical research training in PM&R disciplines, (2) establishing the feasibility of a focused didactic and experiential clinical research course for PM&R clinicians, (3) identifying the impact of the program on participants’ perceptions of clinical skills acquired, and (4) identifying the impact of the program on participants’ formulation and submission of clinical translational-focused grants.
Needs Assessment Guiding Program Creation
To fulfill needs at both university and departmental levels, we developed a proposal for a focused clinical research skills course for PM&R practitioners, with the encouragement of the director of the Michigan Institute for Clinical and Health Research (MICHR, sponsored by the CTSA award). In formulating our program proposal, we recognized the need to identify both fundamental clinical research skills and outcomes that would reflect trainees’ acquisition of these skills, with trainees’ preparation of grant proposals as the culmination of the program. We intended at the outset to measure program feasibility and outcomes.
To secure institutional support for the program, officials at MICHR agreed to help cost share this program with the PM&R Department and facilitated cost-sharing agreements with other departments. We approached department chairs with the idea and expectation that their clinicians would participate in the program. Proposed expenses included costs for program implementation and a new grant-funding mechanism specifically for participants. In addition to these costs, participating departments agreed to support a buy-out of the clinician’s time if he or she was funded with an award. In some cases, this buy-out would include hiring temporary workers to fill in for missed clinical care time. Further support for this effort included the donated time of our program evaluator (Patricia B. Mullan), who developed and interpreted results of the pre- and post-course measures for the program and course participants.
The program focused on one department, PM&R, given the high interest and cooperation of the clinical director. Concentrating on one department helped us tailor clinically relevant examples for use in the program. For example, a session on evaluating research evidence featured articles on constraint-induced therapy, given that the audience would likely include physical and occupational therapists. These clinically relevant examples were intended to enhance understanding of how to evaluate research and apply it to clinical practice. If the attendees had represented other disciplines, such as nursing, examples would have been tailored to that discipline.
To further guide a responsive design for the program, our needs assessment included an e-mail survey of clinicians in all inpatient and outpatient PM&R clinics to assess attitudes and perceived abilities in conducting research. Fifty clinicians of 260 (19%) responded to the survey, which queried 11 areas pertaining to research. For each area, clinicians rated their experience on the following scale: 0 = little or no experience, would need guidance to perform this task; 1 = some experience, have had some instruction in this area but would need further instruction to perform this task efficiently; 2 = quite a bit of experience, am comfortable performing this task independently; 3 = a great deal of experience, could mentor others in this area. Figure 1 summarizes respondents’ self-ratings in four areas of research knowledge. Survey respondents reported least experience in understanding statistics; 90% of respondents reported some or little to no experience in this area. Interpreting clinical research data and developing research hypotheses were also areas in which most respondents reported some or little to no experience (88% and 86%, respectively). The area in which respondents reported the most experience was reading and critiquing literature; however, many respondents (74%) reported only some or little to no experience with this area. In addition, respondents identified high level of interest in learning each of these areas. Between 84% and 92% of respondents reported interest in participating in seminars on these topics if they could find time in their work schedules to attend. Given the institutional survey results, we concluded that a need for the proposed program existed within the PM&R department.
Experience level reported by practicing clinicians prior to development of the Practice-Oriented Research Training (PORT) program (N = 50).
Program Design
We designed the program primarily for practicing physical and occupational therapists. The first and second authors of our team (Murphy and Kalpakjian) developed this program in two phases (Table 1). Phase 1 consisted of a series of didactic seminars covering fundamentals of clinical research. Phase 2 focused on providing clinicians with more in-depth research training. This second phase involved a longer series of didactic seminars in which the principles and processes of clinical research were presented in a step-by-step fashion. In addition, Phase 2 held the expectation for completing outside work and meetings with program mentors related to an eventual grant application submission.
Content of Phase 1 and Phase 2 Sessions
Phase 1 consisted of three 2-hr sessions offered weekly. Phase 1 constituted a continuing education activity for PM&R Department clinicians who were paid for their time after regular work hours to attend. These sessions were open to all university staff and faculty, with no cost to participate. Before implementing Phase 1, participants were surveyed regarding the time at which the program should be offered. On the basis of their responses, we offered sessions once a week in the evenings and catered all sessions because they occurred over dinner hours.
Phase 2 consisted of nine 1-hr sessions offered weekly, structured to reflect a progression of steps from literature review to implementation of a research project. The first and second authors (Murphy and Kalpakjian) delivered some sessions and invited expert speakers from the university to deliver specific sessions. As with Phase 1, we used evening class times and provided meals. In Phase 2, we created a central Web-based site (CTools) that contained all slides from sessions and course materials, including links to university research resources for program participants. We communicated with participants outside of classroom time via e-mail and announcements from our course Web site. The program required a nominal fee to help offset costs of food and the required textbook on clinical research design (Hulley, Cummings, Browner, Grady, & Newman, 2007). For each session, our syllabus identified specific sections of the book as related readings.
A particularly innovative aspect of the program was a newly developed grant competition available only to Phase 2 program participants. Consequently, we structured the Phase 2 training in a manner to concurrently build a grant proposal and complete an institutional review board (IRB) application. Mentors (primarily Murphy and Kalpakjian) were identified early in the program to assist participants with proposal writing. We encouraged participants to meet with mentors early in the program. To develop a grant proposal, we introduced additional expectations of Phase 2 participants. We required participants to meet with department administrators to secure a letter of institutional support. The letter of support outlined the particular buy-out situation and other resources that would be provided to participants (e.g., computer workstations, office supplies). Participants also needed to meet with grant administrators to formulate a budget for the project. Other expectations included regular meetings or e-mail communications with a mentor or mentors outside of scheduled seminar time. The mentor’s role was to provide feedback on refining the project idea, writing the proposal, building a study team, and writing the IRB proposal. Participants were required to submit an IRB application. We also directed participants to seek help using the university’s statistical consult services for guidance with power analyses and statistics.
The grant proposal took the form of a five-page application congruent with the NIH structure (i.e., specific aims, significance, research design), which included the clinical background of the investigators and future projects that might evolve from the pilot work being proposed. The projects proposed needed to be conducted in a year’s time, with the expectation of continued mentoring from the program.
Methods to Evaluate Program Outcomes
We evaluated program outcomes in two ways: (1) participants’ self-rating of their level of skill in defined areas of clinical research and (2) submission of a grant application meeting the program requirements. During Phase 1, we administered a preprogram assessment of participants’ self-rating of their current skill level regarding 14 areas of clinical research. The areas of clinical research skills identified included conducting a literature review, developing a research question and hypotheses, selecting a sample, identifying research methods and statistical analysis, writing a proposal, and presenting research findings. Each skill was rated on a scale ranging from 1 (need further basic instruction) to 4 (able to perform independently). The scale was based on the program evaluator’s experience in using level of independent performance ability as a metric guiding participants’ needs assessment, which then informs the need for medical training programs (Mullan, Weissman, Ambuel, & Von Gunten, 2002). At the program’s completion, trainees completed a postassessment of these skills.
Submitting a grant proposal for competition in the CTSA program constituted the second program outcome. Proposal submission required a high level of effort to complete tasks beyond course contact time to write and refine a research proposal, submit an IRB application, and complete administrative aspects of the submission.
Results
Of the 38 trainees participating in Phase 1, 21 elected to continue through completion of Phase 2. Participants who completed the entire program included 11 physical therapists, 7 occupational therapists, 1 therapeutic recreation clinical specialist, and 2 research staff (1 research coordinator and 1 lab manager).
Self-Ratings of Clinical Research Skills
Figure 2 shows participants’ pretest (n = 23) and posttest (n = 11) ratings of their clinical research skills. Unfortunately the individual pairing of pretest and posttest responses proved problematic, because most attendees did not remember their “code,” which they used to identify themselves at pretest. As a group, the participants demonstrated a gain in all categories, with the most gain in the area of formulating a research question (mean difference of 1.36 points), writing a testable hypothesis (mean difference of 1.35 points), selecting measures to use in their study (mean difference of 1.30 points), and identifying a question in the published literature that a research study might fill (mean difference of 1.20). The least gain was seen in the areas related to statistical analysis, such as conduct a statistical analysis and interpret results (mean difference of 0.41 points), selecting the statistical analysis to test hypotheses (mean difference of 0.52 points), and estimating sample size (mean difference of 0.70). The least gain was also seen in the area of presenting research findings (mean difference of 0.59).
Pre- and posttest ratings of outcome areas (n = 23 on pretest; n = 11 on posttest).
Qualitative feedback from attendees at the end of the didactic and grant writing portion of the program indicated that the program was generally positively received, with the most useful aspects of the program identified as the mentors and their availability, writing the grant proposal, orientation to the fundamentals of research, and sharing of our own proposals that have been successful. Suggestions for improvement included (1) a more systematic orientation to each process of the proposal and IRB application coordinated with the didactic presentation topics and (2) more clarity on how to prioritize tasks (i.e., administrative, science and regulatory).
Grant Proposal Submission
In addition to the self-reported gains from trainees in their level of autonomy, the program evaluation included the number and proportion of trainees who completed grant applications. Of the Phase 2 participants, nine (43%) submitted grant applications. One participant of the nine submitted a grant proposal as an individual, and the remaining eight participants submitted grants in teams for a total of four applications. The titles and focus of the four proposed pilot studies are given in Table 2.
Topics of Grant Proposals Submitted by Participants
The other 12 participants in the program met with mentors initially and intended to complete proposals, but found time too limited or did not have sufficient focus on a specific research question early enough in the program to complete the proposal. The participants who completed grant proposals were encouraged to include a faculty member (i.e., “faculty mentor”) on their team who would provide both guidance in the development of the project and support during its execution. Faculty mentors were typically those with whom the participants had already been working; their inclusion on the proposed research was easily achieved.
To complete the grant application, the teams met with mentors on a biweekly basis across a 3-mo period, sending updated drafts between sessions with mentor feedback received before biweekly meetings. Participants who completed grant applications typically had project ideas in mind when they began the program and worked with mentors to refine these ideas and the research design to carry out the project. Sessions with mentors focused on thinking through the research question, writing specific aims and hypotheses, and determining the details of the research design (e.g., sample characteristics, data collection methods, analytic methods). IRB applications were also completed during this process. Teams worked directly with program mentors working through the IRB application, with varying degrees of support needed. For instance, one grant submission from a single clinician was closely related to her mentor’s large research study, and she worked primarily with her faculty mentor and met periodically with her program mentor.
Grant Application Reviews
A study section review panel was convened by MICHR and included faculty and statistical reviewers who independently scored applications using a study evaluation form commonly used by MICHR’s scientific review group in study sections for university grant competitions. Our institution convened an external review panel, who reviewed the applicants’ work, without prior knowledge of whether they would be funded, and presented their reviews at a study section.
By contrast with a typical study section, grant applicants were asked to attend and to submit peer reviews of all other applications using the same study evaluation form. Murphy and Kalpakjian were present at the study section meeting and structured the meeting as a learning experience. The applicants were shown the process of how grants are typically reviewed in study sections and were given the opportunity to provide comments about other applications as well as to ask questions about their own proposal. The CTSA budget support enabled funding of all four applications submitted.
Postaward Period
After notification that funding was received, a postaward meeting was held. The postaward meeting objectives were twofold: (1) providing information on the tasks needing to be done in the grant “start-up” period, including a link with staff resources and mentors to assist in these tasks, and (2) outlining a structure of support that the program will provide for the year of funding as well as expectations of the grantees. Specifically, program mentors planned to meet with grantees about their research projects, in addition to meetings they would have with faculty mentors on their study teams. Grantees would file quarterly reports with program mentors to update them on progress. At the end of the year period, grantees would be expected to disseminate their findings. A research day would be held for all physical therapist and occupational therapist clinicians and faculty of participating departments in which grantees would present their work. Grantees were encouraged to budget travel costs for one clinical conference or scientific meeting and to prepare a poster or podium presentation. Mentors were also to work on manuscript writing with grantees during this period.
Discussion
The need for training to promote clinical research translation is reflected in the NIH’s creation of the CTSA program. Existing recent research and policy statements affirm the relevance of this type of training in PM&R. Instructional methods appropriate for meeting these needs include formal didactic training, mentoring, and applied experience. In this program evaluation study, we established the evidence of consensus about the need for a structured program from clinicians before the development of our program. Evidence of institutional support for the program included the award of CTSA-sponsored intramural funding for the development of the program. Additional evidence of feasibility and support included PM&R’s encouragement of the course development and other departmental chairs in related disciplines, provision of release time, and financial support.
The responses of the clinician–trainees to the baseline assessment of clinical skills corroborated the evidence of the need for the program among the trainees at our institution. The participation of busy clinicians in both phases of the program attests to the feasibility of a PM&R–focused clinical skills research training program. The support for the program attested to the departmental and institutional recognition of the relevance and importance of a training program for clinical research.
In terms of impact on clinical research skills, our program evaluation showed that these trainees perceived gains in all of the research skills that the program was intended to address. Gains made in identifying gaps in the literature, formulating research questions, and writing hypotheses are likely the result of mentoring and the crafting of a research proposal that drew heavily on such skills. The perception of smaller gains in skills related to statistics and presenting findings should be acknowledged, suggesting that these skills require greater hands-on and didactic experience.
The more stringent measures of the program’s success included the participants’ completion of clinical research grants on a short, defined timeline. Given the high level of effort involved, it was encouraging that 43% of participants submitted a grant proposal and completed all other necessary tasks (e.g., submitting an IRB application, formulating a budget, formalizing the arrangement of a clinical buy-out in the department to undertake the research project). However, not all trainees completed grant applications. Our postcourse review suggests that participants who identified research questions early in the course were most likely to complete grant proposals. In addition, the formation of research teams among clinicians already working together in a clinic appeared to facilitate successful grant submissions.
Challenges and Future Directions
We believe the success of a program such as this one depends heavily on institutional support. Without the support of the CTSA, university, and participating departments, we would not have been able to offer a program-specific grant competition. The grant competition was designed for practicing clinicians who typically have heavy patient-care duties. The PM&R Department was instrumental in allowing a buy-out of clinical time. The main challenge of the program for program mentors (i.e., Murphy and Kalpakjian) and Phase 2 participants was the amount of time required outside of course contact time. In the future, we plan to provide more extensive time for grant preparation. Our results also suggest encouraging group submissions.
As a result of the success of this program, other departments at the university have asked to cost share the program so that their staff and faculty can participate. In response, we have planned program revisions to include clinical faculty from other departments in the medical school who do not have previous research experience. An increased heterogeneity of participants in terms of disciplines will require mentors who will be able to provide time for this purpose. We will need to evaluate the impact of the program for this more diverse faculty group. Inclusion of this diversity will enable us to determine whether this program facilitates collaboration across disciplines. Other future plans to facilitate course participation include an increase in online resources.
Conclusions
This article presents a feasible model for providing research training to practicing clinicians. Although the preliminary results are encouraging, we acknowledge some limitations in the measurement of program success. Limitations include selection bias, given that clinicians more interested in research may have been more likely to have participated in the needs assessment and in the course. In addition, the current study lacks a comparison group. We believe that the use of an actual grant competition as a measurable outcome is in keeping with best practices in program evaluation, which urge the use of outcome measures that mirror performance expectations in the work setting. The grant submission outcome measure appears to constitute a means for evaluating at least the short-term outcome of the program on the trainees’ skills and motivation. We also believe the completion of the grant submission process offers preliminary support that the program was meeting its objectives for research training. We recognize that these represent only short-term outcomes, not the long-term impact of the program’s effectiveness.
Existing research on educational reform in medical education identifies predictors of success as including institutional support and mentoring. Our program evaluation suggests that institutional support, in the form of both institutional and interdepartmental funding, as well as focused training and mentorship, can work to promote interest and skills in clinical research among disciplines involved in PM&R.
Footnotes
Acknowledgments
We thank Patricia Moulis for collecting survey data and assisting in program organization and Jose Kottoor for support of the clinical training of clinicians in the Occupational Therapy and Physical Therapy Division at the University of Michigan. Grant UL1RR024986 from the National Center for Research Resources (NCRR) provided support for the project. The content is solely the responsibility of the authors and does not necessarily represent the official views of NCRR or the National Institutes of Health. Murphy and Kalpakjian were recipients of K01 Mentored Research Scientist Career Development Awards from the National Center for Medical Rehabilitation Research at the time this article was written (HD045293 [Murphy] and HD046602 [Kalpakjian]).