Abstract
Keywords
A major challenge facing health professionals is the sheer quantity of new research being published each year. In the Medline bibliographic database, for instance, approximately 700,000 references were added in 2010 alone (U.S. National Library of Medicine, 2011). This volume of research places tremendous pressure on health care practitioners, including occupational therapists and occupational therapy assistants, as they attempt to incorporate research evidence into practice (Bastian, Glasziou, & Chalmers, 2010; Holm, 2000; Ottenbacher, Tickle-Degnen, & Hasselkus, 2002). Methods for efficiently locating, accessing, filtering, and interpreting research are critical if practitioners are to integrate research evidence with their clinical expertise, client values, and information from the clinical context (Straus, Richardson, Glasziou, & Haynes, 2005). Systems for synthesizing this information have become essential.
The long tradition of using literature reviews to help readers grasp the breadth of a topic is unlikely to slow, and systematic reviews have become an important method for rigorously synthesizing the literature (Mulrow, Cook, & Davidoff, 1997). Systematic reviews involve the use of explicit, reproducible, and uniformly applied criteria in searching for and selecting articles for review, appraising risk of bias within individual studies, and synthesizing the results of primary studies to provide a concise summary of the available research evidence (Crowther & Cook, 2007). When possible, systematic reviews use meta-analysis, a statistical method that combines the results of two or more primary studies and provides a more precise estimate of effect than results from individual studies (Bennett, Leicht Doyle, & O’Connor, 2010).
Systematic reviews can be used to synthesize studies with different methodologies, depending on the research question of interest (Gough, Oliver, & Thomas, 2012). The most frequently conducted type of systematic review synthesizes studies about the effects of an intervention and includes randomized controlled trials (RCTs) alone or a mixture of randomized and nonrandomized studies. Systematic reviews may address other clinical questions, however, such as those about diagnosis, long-term outcomes, or economic factors (Gough et al., 2012). More recently, qualitative research has been the subject of systematic reviews (Barnett-Page & Thomas, 2009), and concerted efforts have been made to integrate qualitative and quantitative research in systematic reviews (Thomas et al., 2004).
Robust systematic review can improve the dissemination of research evidence, hasten the assimilation of research into practice, help clarify conflicting results between studies, and increase the generalizability of findings (Crowther & Cook, 2007). Yet, systematic reviews can also be subject to publication bias, in which primary studies are more likely to be published if interesting or positive results are obtained; another potential bias is the inclusion of flawed primary studies or studies with poor reporting of primary data (Jüni, Altman, & Egger, 2001). Such bias affects the extent and nature of primary studies available for inclusion in systematic reviews.
Despite these issues, systematic reviews are rapidly growing in number, and an estimated 11 new systematic reviews are published each day (Bastian et al., 2010). This growth has been nurtured by the Cochrane Collaboration, an international collaboration that produces and disseminates systematic reviews about the effects of health care interventions (Bero & Rennie, 1995). Although occupational therapy practitioners are actively involved in the Cochrane Collaboration, a comparatively small amount of research has specifically tested the effectiveness of occupational therapy interventions that might then be synthesized in systematic reviews. This scarcity is attributable in part to occupational therapy being a “research emergent” profession, with a young research tradition and limited research infrastructure (Ilott, Taylor, & Bolanos, 2006).
To maximize the value of existing occupational therapy research, practitioners first need to know where this research might be found. Only a small number of studies evaluating the effectiveness of occupational therapy interventions have been published in occupational therapy journals. Much of the research on the effect of occupational therapy interventions is multidisciplinary in nature and is published in a variety of health journals (Andresen, Tang, & Barney, 2006; Bennett et al., 2007; Potter, 2010).
To overcome this complexity of publication sources, we developed the OTseeker database (Occupational Therapy Systematic Evaluation of Evidence, available at www.otseeker.com; Bennett, McCluskey, Hoffmann, & Tooth, 2011). OTseeker contains citations of systematic reviews and critical appraisals of RCTs to improve access to high-quality research evidence on the effects of interventions relevant to occupational therapy. The articles contained within OTseeker are located through regular and comprehensive searches of the following databases: Medline, CINAHL, ERIC, Embase Rehabilitation and Physical Medicine, AMED, PsycInfo, the Cochrane Library, CancerLit, Ageline, and Turning Research Into Practice (TRIP) and by handsearching 13 occupational therapy journals regularly. In total, articles are indexed from approximately 1,100 journals.
For a systematic review to be eligible for inclusion in OTseeker, it must contain a methods section and at least one RCT that meets the following criteria: (1) The study must compare at least two interventions using an RCT or crossover design, (2) the study must be published in a peer-reviewed journal, (3) the intervention must be designed for application or use with human participants who are likely to be seen by an occupational therapist, (4) the trial must have used random allocation, and (5) at least one of the interventions examined could be part of current or future occupational therapy practice (McCluskey, Bennett, Hoffmann, & Tooth, 2010). Thus, although studies of interventions indexed in OTseeker need to be relevant to occupational therapy, they do not need to specifically involve occupational therapy.
Systematic review findings can benefit the occupational therapy profession and clients. Not only are systematic reviews an important study design for informing practice, but reviews can inform occupational therapy researchers who define future research agendas and can assist administrators and purchasers who develop evidence-based policies and guidelines (Crowther & Cook, 2007; Murphy, Robinson, & Lin, 2009). The OTseeker database provides occupational therapy practitioners with a single resource for locating systematic reviews and RCTs. Our previous studies have aimed to inform therapists about the status of evidence contained in OTseeker in a number of practice areas, including pediatrics (Hoffmann et al., 2008), stroke rehabilitation (Hoffmann et al., 2007), and work-related injury prevention and management (McCluskey et al., 2005). Yet none of these studies provide an overview of the content of all systematic reviews relevant to occupational therapy. When a descriptive analysis of the content of OTseeker was last published (Bennett et al., 2007), 939 systematic reviews were indexed in OTseeker. However, that study provided only a descriptive analysis of systematic reviews relevant to occupational therapy and did not identify systematic reviews specifically involving occupational therapy.
The aims of the current study were (1) to report the number of systematic reviews indexed in OTseeker that are broadly relevant to occupational therapy by year of publication, type of analysis used in the review, publishing journal, intervention categories involved, and diagnostic categories involved and (2) to determine the number of systematic reviews that specifically involved occupational therapy interventions and when and where this subset of reviews was published. Findings will help inform occupational therapy clinicians and educators, researchers, managers, and policymakers of the scope of research available from systematic reviews.
Method
Design and Data Collection Procedures
We used a cross-sectional descriptive study design to identify all systematic reviews contained in OTseeker during a search in December 2011. Initially, we described the systematic reviews in terms of
Date of publication: We allocated reviews into 5-yr periods starting from 1980.
Type of analyses reported: We searched the abstracts and titles of the systematic reviews to determine whether meta-analytic methods were used and whether reviews included both randomized and nonrandomized studies.
Key sources and journals: We identified the titles of journals and other sources indexed in OTseeker that have published systematic reviews.
Intervention categories: OTseeker has 35 intervention categories—for example, consumer education, behavioral interventions, movement training, and cognition.
Diagnostic categories: OTseeker has 31 diagnostic categories—for example, neurological (e.g., stroke), mental health (e.g., affective disorders), and musculoskeletal (e.g., back or neck) conditions. A full list of the intervention and diagnostic categories used in OTseeker is available at www.otseeker.com/inthelp.aspx.
Next, we analyzed all systematic reviews to identify how many specifically involved occupational therapy and their year of publication and to determine in what sources this subset was published. We used two approaches to identify systematic reviews that specifically involved occupational therapy. First, we conducted a search across the systematic reviews for the phrases occupational therapy or occupational therapist in the article title, abstract, journal title, and key words. Second, Bennett searched the full text and tables of all remaining systematic reviews for occupational therapy or abbreviations of this phrase, such as occup ther. Systematic reviews in which at least one author was clearly identified as an occupational therapist were also included in the subset of systematic reviews specifically involving occupational therapy. Studies were not included when a named intervention, such as hand therapy or falls prevention, may have been provided by an occupational therapist but the term occupational therapy was not included anywhere in the full text of the systematic review to avoid misclassification of reviews of interventions that may have been undertaken by other health professions.
Data Analysis
We used descriptive statistics to analyze the data. Data are presented as frequencies and percentages.
Results
Systematic Reviews in OTseeker by Year and Type of Analysis
In December 2011, 1,940 systematic reviews were indexed in the OTseeker database. Figure 1 presents the frequency of systematic reviews published in 5-yr increments between 1980 and 2011. The 5-yr increment with the smallest number of published systematic reviews was 1980 to 1984, with only 6 (0.3%) systematic reviews published in that period. The period with the highest number of systematic reviews (n = 775, 39.9%) was 2005 to 2009. Of the 1,940 systematic reviews indexed in OTseeker, 463 (23.9%) clearly indicated in their abstract that they synthesized data using meta-analysis. The remaining reviews synthesized data narratively. Systematic reviews in this study either restricted their inclusion criteria to RCTs alone or synthesized data from both randomized and nonrandomized studies.
Number of systematic reviews contained in OTseeker published between 1980 and 2011.
Systematic Reviews in OTseeker by Journal or Other Source Title
The 1,940 systematic reviews analyzed were published across 496 journals, the majority of which were medical and rehabilitation journals. The Cochrane Database of Systematic Reviews contained the highest number of systematic reviews indexed in OTseeker (n = 407, 21.0%). The only occupational therapy journal included in the top 10 sources most frequently publishing systematic reviews was the American Journal of Occupational Therapy (AJOT), with 29 systematic reviews (see Table 1). Only 53 systematic reviews (2.7%) indexed in OTseeker were published in occupational therapy journals, and these were published in 10 different journals.
Top 10 Sources of Systematic Reviews Relevant to Occupational Therapy (N = 1,940)
Systematic Reviews in OTseeker by Intervention Category and Diagnostic Group
Table 2 presents the number of systematic reviews indexed in OTseeker by intervention category. The most frequently represented intervention categories were consumer education, psychosocial techniques, and exercise. The intervention categories with the fewest systematic reviews included play, purposeful activity, and instrumental activities of daily living.
Frequency of Systematic Reviews Relevant to Occupational Therapy, by Intervention Category (N = 1,940)
Note. CBT = cognitive–behavioral therapy.
Because each article had as many as five codes for intervention categories, percentages do not equal 100%.
The most frequently represented diagnostic categories were neurological diagnoses, mental health, and musculoskeletal or connective tissue conditions (Table 3). The small number of systematic reviews were available for the diagnostic categories of renal disorders, visual and hearing disorders, congenital disorders, and immune system dysfunction.
Frequency of Systematic Reviews Relevant to Occupational Therapy, by Diagnostic Category (N = 1,940)
Because each article had as many as five codes for diagnostic categories, percentages do not equal 100%.
Systematic Reviews in OTseeker Specifically Involving Occupational Therapy by Date and Journal
A subgroup of 125 reviews were identified from the 1,940 reviews that contained the terms occupational therapy or occupational therapist in the domains of article title, abstract, journal title, or key words. A second subgroup of 265 reviews were identified from the remaining 1,815 reviews in which the terms occupational therapy or occupational therapist or abbreviations such as occup ther were identified in the main text of an article or in a table. In total, these two searches identified 390 (20.1%) systematic reviews specifically involving occupational therapy. Few of the systematic reviews specifically involving occupational therapy were published between 1980 and 1984 (n = 3, 0.2%), but this number increased to 151 (7.8%) between 2005 and 2009. The 390 systematic reviews specifically involving occupational therapy were published in 134 different journals, with the Cochrane Database of Systematic Reviews containing the highest number (n = 77, 19.7%), followed by AJOT (n = 29, 7.4%).
Discussion
Systematic Reviews Relevant to Occupational Therapy
This study is the first to comprehensively describe the nature and content of systematic reviews relevant to occupational therapy. Findings reveal a steady increase in the number of systematic reviews published over time, with the greatest number published between 2005 and 2009 (n = 775). This exponential increase has also occurred in other rehabilitation and medical fields (Bader & Ismail, 2004; Engberg, 2008; Johnston, 2003; Moseley, Herbert, Sherrington, & Maher, 2002). Contributing to this increase is the substantial development in techniques, standards, and procedures for systematic reviews over the past 30 yr (Mullen & Ramirez, 2006). This increase has occurred alongside the growth of the evidence-based practice movement and the creation of the Cochrane Collaboration, which actively supports the production and dissemination of systematic reviews (Bero & Rennie, 1995; Dirette, Rozich, & Viau, 2009; Engberg, 2008).
The use of meta-analysis is also on the rise, with nearly one-quarter of reviews indexed in OTseeker clearly indicating in their abstract that data were synthesized using meta-analysis. This figure is conservative because we were unable to determine with certainty which systematic reviews contained a meta-analysis without reading the full text of each review. Through the use of meta-analysis, it is possible to enhance the precision of results by statistically combining data from included studies and presenting data in forest plots, which help the reader understand trends (Egger, Smith, & Phillips, 1997; Liberati et al., 2009). Despite the advantages of meta-analysis, it is not always possible to statistically combine results of individual studies. When included studies are too clinically or methodologically heterogeneous, meta-analysis may not be possible (Higgins, Thompson, Deeks, & Altman, 2003). This problem occurs commonly in rehabilitation trials that involve testing across a range of different settings with markedly different treatment protocols and outcome measures (Massy-Westropp & Masters, 2003; van den Ende, Steultjens, Bouter, & Dekker, 2006).
Meta-analyses of the effects of interventions often combine data from RCTs, whereas systematic reviews that synthesize data narratively may either restrict the included studies to RCTs or include nonrandomized studies. When a review includes nonrandomized studies, it is particularly important to read the full review to identify which types of studies the authors have drawn their conclusions from. The use of a systematic review methodology does not necessarily mean that conclusions from systematic reviews can be relied on for informing practice decisions. Readers must consider the type and rigor of the individual studies within the review, the potential for bias in the review methods used, and the clinical (and not just statistical) significance of results found. It is helpful when study authors provide this information for readers in the abstract. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Liberati et al., 2009) is a widely endorsed guideline for reporting systematic reviews and recommends that the abstracts of reviews provide a “balanced and realistic assessment of the review’s findings” (p. 69). However, abstract word limits often prohibit the inclusion of detailed descriptions of study methods.
Regardless of which type of synthesis or combination of studies is used, readers need to be aware of the potential for bias in reviews and the possibility of unjustified conclusions or recommendations (Scott, Greenberg, Poole, & Campbell, 2006). Commonly used approaches for determining the risk of bias or methodological quality of systematic reviews include the Assessment of Multiple Systematic Reviews (AMSTAR), an 11-item tool with good reported face and content validity (Shea et al., 2007), and the Critically Appraised Skills Program Checklist (Public Health Resource Unit, England, 2006), which was adapted from a pivotal article on this topic (Oxman, Cook & Guyatt, 1994). Explanations of how to assess bias in systematic reviews are also included in many evidence-based practice textbooks (e.g., Bennett et al., 2010).
The systematic review content indexed in OTseeker covers a diverse range of intervention and diagnostic categories. A large number of reviews address interventions such as consumer education, psychosocial techniques, and exercise and target diagnostic groups such as neurological conditions (e.g., stroke), mental illness, and musculoskeletal or connective tissue injuries. The interventions and diagnostic topics most often studied may represent areas in which occupational therapists frequently practice or areas that are more feasible for undertaking research such as RCTs. Very few citations exist for some interventions, such as play, purposeful activity, and instrumental activities of daily living, and for diagnostic groups such as renal disorders, visual and hearing impairments, and congenital disorders. The fact that some intervention and diagnostic categories are much broader than others also affects the number of systematic reviews within each category, which must be considered when interpreting these data.
Systematic reviews relevant to occupational therapy are published across a broad range of journals. Searches of occupational therapy journals alone will miss the majority of reviews. Few systematic reviews (n = 53) about the effects of interventions were published in occupational therapy journals, with AJOT containing the largest number (n = 29). This number has increased since commencement of the American Occupational Therapy Association’s Evidence-Based Practice Project, which includes a program of systematic reviews of multidisciplinary scientific literature (Lieberman & Scheer, 2002).
Systematic Reviews Specifically Involving Occupational Therapy
Although this study identified 1,940 systematic reviews relevant to occupational therapy, only 390 could be clearly identified as specifically involving occupational therapy. Several factors explain this relatively small proportion. First, it was difficult to identify systematic reviews that could be legitimately categorized as specifically involving occupational therapy. Many articles were not clearly identifiable using terms denoting occupational therapy. Relying on the use of occupational therapy in the text and key words of systematic reviews potentially excluded reviews that did not use these terms but focused on specific interventions that often involve occupational therapists, such as hand therapy reviews (Gerritsen et al., 2002), and broad service delivery interventions, such as case management. Of concern is the labeling of systematic reviews as “reviews of occupational therapy.” It was evident in this study that a number of systematic reviews declared in their title that they were “reviews of occupational therapy” when in fact they incorporated studies carried out by other health professions, not specifically occupational therapy. Using phrases such as relevant to occupational therapy, involving occupational therapy, or informing occupational therapy may be more legitimate descriptors and can reduce confusion when communicating this type of information with other health professions. Clarity about the specific contribution of occupational therapy in future research publications will not only enable more efficient identification and indexing of occupational therapy research but also enhance our communication with other disciplines.
Second, the small number of systematic reviews that consider the effects of interventions specifically involving occupational therapy may be attributable to the profession having developed a research tradition relatively recently compared with disciplines such as medicine. Our profession does not have a long history of using systematic review methods (Ilott et al., 2006; Nelson & Mathiowetz, 2004; Ottenbacher, 1990). Finally, in some practice areas it is difficult to conduct primary studies such as RCTs to assess the effectiveness of occupational therapy interventions, and the scarcity of primary studies has implications for the conduct of systematic reviews (Ilott et al., 2006). For the profession of occupational therapy to continue to grow, creative means for tackling the barriers to undertaking effectiveness research are required to increase the quantity and quality of RCTs and hence systematic reviews in key areas of practice (Gutman, 2009).
To ensure that the research that does exist is incorporated into systematic reviews, one strategy is for therapists and researchers to participate in multidisciplinary teams that write systematic reviews. The Cochrane Collaboration offers a welcoming, interprofessional means for anyone who is interested to contribute to the development of systematic reviews. Involvement may include participating in writing a review, being a peer reviewer or translator, or being a consumer representative. To understand more about this collaboration and how to participate, see www.cochrane.org.
The small subset of reviews involving occupational therapy identified in this study were published across 134 different journals, with the Cochrane Library containing the greatest number. Andresen et al. (2006) also noted the propensity for reviews to be spread across hundreds of journals in a previous study that evaluated the scope of occupational therapy literature available through searches of CINAHL and Medline databases between 1996 and 2002. Andresen and colleagues found that 25% (n = 868) of eligible occupational therapy articles were research articles and that, of that number, almost half (43%) were published in non–occupational therapy journals. More recently, Potter (2010) demonstrated a trend in the occupational therapy literature toward an increasingly scattered publication pattern, although the analysis did not examine patterns according to research type. The scattering of RCTs and systematic reviews across hundreds of journals has been identified as a problem for other health professions, and with continuing growth in the number of publications and journals publishing them, the problem of scatter will only intensify (Hoffmann, Erueti, Thorning, & Glasziou, 2012). As a profession, occupational therapy needs to identify solutions that will help manage this scatter.
Authors of systematic reviews that involve occupational therapy publish in non–occupational therapy journals for several reasons. For example, publishing their work in multidisciplinary journals with high readership and high impact factors may ensure that their findings reach their intended target audience or be useful for their academic careers (Walker, 2005). Publication in non–occupational therapy journals also builds a collaborative body of knowledge from which multiple disciplines can draw (Dirette et al., 2009). As a result, occupational therapists are not able to rely on occupational therapy journals as the sole source of research evidence and must maintain a broad approach to reading that includes a number of multidisciplinary journals. However, it is not feasible for busy clinicians to scan the number of journals required to keep up with new research (Hoffmann et al., 2012). More efficient alternatives to finding research include developing skills in searching databases (Corcoran, 2006), including efficient retrieval of particular study types such as systematic reviews, and using targeted resources such as OTseeker (www.otseeker.com) and the REHAB+ database (http://plus.mcmaster.ca/rehab/Default.aspx). The systematic reviews and RCTs included in REHAB+ are sourced from more than 120 journals and filtered for quality, and those that are scientifically sound are rated for relevance and newsworthiness by a panel of practicing occupational therapists and physiotherapists.
Limitations
Aspects of the OTseeker database may have influenced the data we collected for this study. First, as many as five intervention and five diagnostic categories are assigned to each article included in the OTseeker database; therefore, articles may have been counted more than once. Second, systematic reviews included in this study were limited to those that included at least one RCT according to the OTseeker inclusion criteria (determined by two independent raters for OTseeker). Although some systematic reviews of treatment effectiveness relevant to occupational therapy do not include any RCTs, these would not be included in the OTseeker database and therefore were not included in our study. Although it was evident from many abstracts that the reviews included in this study could be categorized as including either RCTs alone or RCTs and nonrandomized studies, the number in each of these two categories was not determined. Third, few systematic reviews indexed in OTseeker are written in languages other than English because non-English reviews are not currently targeted by OTseeker search strategies. The results of this study are therefore limited to English-language reviews. Finally, although the articles in OTseeker were located from approximately 1,100 journals, and although comprehensive searches are used to search for all relevant RCTs and systematic reviews, because of the complexity of occupational therapy practice some systematic reviews relevant to occupational therapy may have been missed.
Conclusion
This study identified 1,940 systematic reviews relevant to occupational therapy. Although systematic reviews have the potential to provide rigorous evidence, readers need to read reviews critically and be aware of the potential for bias arising from review methods used. For example, many systematic reviews of the effects of interventions include both randomized and nonrandomized studies within the review. Although this inclusiveness provides a comprehensive view of the research and important insights, readers need to be aware of the type and rigor of the individual studies contributing to the conclusions of a review and determine whether the review conclusions are justified and appropriate. Longer journal abstracts have the potential to provide more information about study methods, which can help readers identify potential biases in published systematic reviews.
The systematic reviews were scattered across 496 different journals, the majority of which were not occupational therapy journals. This scattered distribution highlights the importance of searching specialist databases that are relevant to occupational therapy (e.g., OTseeker, REHAB+) that aim to reduce the problem of scatter by gathering high-level evidence regarding the effects of occupational therapy into a single location.
Only a small number of systematic reviews (n = 390) specifically involved occupational therapy. Creative means for tackling the barriers to undertaking effectiveness research are required to strengthen the evidence for occupational therapy. Clarity about the contribution of occupational therapy in reports of future studies is also essential for efficient identification of occupational therapy research and communication with other disciplines.
Footnotes
Acknowledgments
The OTseeker database is currently funded by the Motor Accidents Authority of New South Wales, Australia, and the Occupational Therapists Board of Queensland, Australia. Tammy Hoffmann is supported by a National Health and Medical Research Council of Australia Primary Health Care Research, Evaluation, and Development Career Development Fellowship with funding provided by the Australian Department of Health and Ageing.