Abstract
This study found that the reporting quality in abstracts of randomized controlled trials published in the five highest impact occupational therapy journals was moderate to low and recommends that journal editors require authors of RCTs to use the CONSORT-A checklist to promote optimal reporting and to enable occupational therapy practitioners to critically appraise the validity of findings.
The randomized controlled trial (RCT) is the gold standard for evaluating clinical procedures because of its rigorous methodology, prospective data collection, and ability to establish causal conclusions (Hariton & Locascio, 2018; Murad et al., 2016). RCTs provide the highest quality scientific evidence and the most useful basis for designing interventions and clinical guidelines and protocols (Winstein et al., 2016). RCTs play a significant role in evidence-based clinical decision making for occupational therapy practitioners (Lin et al., 2010; Nelson & Mathiowetz, 2004). Thus, detailed reporting in RCT abstracts is crucial to practitioners’ ability to provide high-quality interventions and ensure client safety.
Occupational therapy practitioners assessing the potential applicability of an RCT to their daily clinical practice must critically appraise the validity of the results (Al-Jundi & Sakka, 2017; Miller & Forrest, 2009). However, access to the full text of the article may be limited by time constraints (Harding et al., 2014), institutional skills (Voronin et al., 2011), or lack of resources (Harding et al., 2014). Thus, the only information on the study’s findings that is available to most practitioners is provided in the article’s title and abstract (Barbour et al., 2006; Hopewell et al., 2008, 2012). If the abstract presents inaccurate or incomplete information, practitioners may misinterpret the findings and, in consequence, make ineffective clinical decisions (Barbour et al., 2006; Hopewell et al., 2008). It is essential that the abstracts of published RCTs be complete and accurate.
To tackle the problem of inadequate reporting of RCTs and to support their critical appraisal and interpretation, the Consolidated Standards of Reporting Trials (CONSORT) Group developed the CONSORT Statement (Schulz et al., 2010), a set of minimum recommendations for reporting RCTs. The CONSORT for Abstracts (CONSORT-A; Hopewell et al., 2008) is an extension of the CONSORT Statement that consists of a checklist of 17 essential items authors should provide when reporting the findings of an RCT in the abstract.
Researchers have used the CONSORT-A checklist to examine reporting quality in RCT abstracts published in endodontics (Fang et al., 2020), cardiovascular diseases (Shaqman et al., 2020), psychiatry (Song et al., 2017), surgery (Speich et al., 2019), and oncology (Ghimire et al., 2014), among other fields. However, to our knowledge, no studies have evaluated reporting quality in RCT abstracts in the field of occupational therapy. Therefore, we used the CONSORT-A checklist to analyze reporting quality in RCT abstracts published between 2008 and 2018 in the occupational therapy journals with the five highest impact factors.
Method
Journal Selection
We conducted the search for journals in the Web of Science on September 28, 2019. Because this database had no category for occupational therapy, we used the search term “occupational therapy.” We dismissed acronyms as suitable search terms, mainly because of the difficulty in formulating an effective and reproducible search strategy. After obtaining the search results, we listed, in descending order, the journals with an impact factor of 1 or higher in 2018 according to Journal Citation Reports. The following journals had the highest impact factors and were included in this study: American Journal of Occupational Therapy (AJOT), Australian Occupational Therapy Journal (AOTJ), Canadian Journal of Occupational Therapy (CJOT), Scandinavian Journal of Occupational Therapy (SJOT), and Physical and Occupational Therapy in Pediatrics (POTP; see Figure 1). To identify RCT articles and their abstracts, we conducted separate searches for each journal in Scopus using the full journal title in the source title field and the publication date range of calendar years 2008 to 2018.
Flow diagram of the article selection process.
Article Selection and Data Extraction
We transferred the documents retrieved from Scopus to a comma-separated values file format, selecting the following information for each article: first author’s name, authors’ affiliations, publication year, journal, number of citations in other articles, abstract, title, and keywords. Two authors (Daniel Prieto- Botella and Leticia Moreno-Campos) screened the titles and abstracts manually to select the RCTs and to minimize the effects of errors in the database. They identified articles as RCTs if “randomized controlled trial” or a similar term was present in the title or abstract. Disagreements between the two authors were resolved in consultation with a third author (Eva- María Navarrete-Muñoz). Feasibility studies, protocols, and conference papers were excluded. Prieto-Botella updated the number of citations for each article on October 28, 2020.
To ensure consistent data collection, Prieto-Botella and Moreno-Campos used an ad hoc questionnaire to record the following data for each article: number of authors, structured abstract (yes or no), abstract length (≤150, 151–200, >200 words), first author’s geographic area (North America, Europe, other countries), number of author institutions, sample size, trial design (parallel or cluster), participant health conditions, and occupational therapy domain (occupations, performance skills, other domains) based on the third edition of the Occupational Therapy Practice Framework: Domain and Process (American Occupational Therapy Association [AOTA], 2014). We grouped health conditions into six categories: (1) neurological pathologies (cerebral palsy, congenital hemiparesis, multiple sclerosis, spastic diplegia, stroke); (2) developmental disorders (autism, developmental delay, developmental coordination syndrome, gross motor delays, fetal alcohol syndrome, handwriting difficulties, attention deficit hyperactivity disorder); (3) mental disorders (dementia, intellectual disability, unspecified mental disorder); (4) physical pathologies (carpal tunnel syndrome, interphalangeal joint extension deficit, osteo- and rheumatoid arthritis, tendon laceration, low vision, macular degeneration); (5) chronic and age- related pathologies (functional and cognitive decline, cancer, chronic obstructive pulmonary disease, unspecified chronic health conditions); and (6) no pathology.
Reporting Quality Evaluation
We checked the title and abstract of all included articles for adherence to the CONSORT-A checklist. The CONSORT-A checklist consists of 17 items categorized into eight sections: (1) title, (2) authors, (3) trial design, (4) methods (participants, interventions, objective, outcome [methods], randomization, and blinding), (5) results (numbers randomized, recruitment, numbers analyzed, outcome [results], and harms), (6) conclusions, (7) trial registration, and (8) funding (Hopewell et al., 2008). To quantify adherence to CONSORT-A, we assigned 1 point to each item; thus, total scores ranged from 0 to 17 points, with higher scores representing closer adherence to the CONSORT-A checklist and therefore higher quality reporting in the abstracts.
Articles included in the study were randomly assigned to Prieto-Botella and Moreno-Campos, who independently rated adherence to the CONSORT-A checklist. We calculated interrater reliability for a random sample of 34 articles; Cohen’s κs were ≥.80, indicating excellent agreement. In cases of disagreement, the raters met with Navarrete-Muñoz to discuss the reasons for the discrepancy and reach agreement on the rating.
Data Analysis
Statistical analyses were performed with R (Version 4.0.2). We used frequencies and percentages for bibliometric features of the five occupational therapy journals, characteristics of the included articles, and adherence to CONSORT-A. In addition, we conducted bivariate analyses to explore the relationships between article characteristics and median CONSORT-A scores after checking the scores for normal distribution using the Kolmogorov–Smirnov test. For the bivariate analyses, we used the Kruskal–Wallis and Wilcoxon tests to calculate the median and interquartile range (IQR), and we set significance at p = .05.
Results
Table 1 lists the bibliometric features of the occupational therapy journals included in this study. The journals published a total of 3,409 articles from 2008 to 2018, of which 78 articles (2.3%) reported RCTs. AJOT published the highest proportion of these RCTs (62.8%), followed by SJOT (15.4%), AOTJ (10.3%), POTP (10.3%), and CJOT (1.3%). These 78 articles were cited 1,518 times, for a mean of 19.5 citations per article. AJOT received the highest number of citations (n = 1,080; M = 22.0 citations per article).
Bibliometric Features of the Five Highest Ranked Occupational Therapy Journals in the 2018 Journal Citation Reports
Notes. IF = impact factor (Journal Citation Reports); NA = not applicable; RCTs = randomized controlled trials; SCIE = Science Citation Index Expanded (Web of Science); SSCI = Social Sciences Citation Index (Web of Science).
The Web of Science ranking category for all journals was Rehabilitation, unless otherwise noted.
Total citations for RCT articles divided by no. of RCT articles.
Web of Science Pediatrics category.
Table 2 presents the characteristics of the 78 articles and median CONSORT-A scores for the abstracts. In general, most of the articles had five or fewer authors (73.1%) and five or fewer author institutions (78.2%); slightly more than half (51.3%) were written by first authors from North America. The abstracts were predominantly structured (82.1%), and over one-third (38.5%) had 150 words or less. Approximately two-thirds (67.9%) of the RCTs had a parallel design, and 37.2% had a sample size of at least 50 participants. The RCTs evaluated interventions for participants with a range of health conditions; the highest pro portion of participants (28.2%) had a neurological disorder. The interventions focused mainly on occupations (55.1%) and performance skills (39.7%).
Relationships Between Characteristics of the RCT Abstracts (N = 78) and Median CONSORT-A Scores
Notes. CONSORT-A = Consolidated Standards of Reporting Trials for Abstracts; IQR = interquartile range; RCT = randomized controlled trial.
CONSORT-A scores (continuous nonparametric variable) compared to categorical (Kruskal–Wallis test) and dichotomous (Mann–Whitney U test) variables.
N = 72; 6 abstracts did not report sample size.
CONSORT-A scores differed significantly by journal (p = .021), number of author institutions (p = .015), first author’s geographic area (p = .006), and abstract length (p ≤ .001). AJOT (Mdn = 8.0, IQR = 7.0–9.0) and CJOT (Mdn = 7.0, IQR = 7.0–7.0) had the lowest CONSORT-A scores. Articles describing RCTs conducted in collaboration by authors from five or more institutions had a higher median CONSORT-A score (Mdn = 9.0, IQR = 8.0–10.0) than those by authors from fewer institutions. Articles in which the first author was from Europe had a higher median CONSORT-A score (Mdn = 10.0, IQR = 8.0–11.0) than those with first authors from other geographic areas. Abstracts of 200 words or more had a higher median score (Mdn = 10.0, IQR = 9.0–11.0) than those with fewer words. No significant associations were found between median CONSORT-A score and trial design, structured abstract, number of authors, occupational therapy domains, or health conditions.
Figure 2 displays the percentage of the 78 articles that adhered to each of the 17 items of the CONSORT-A checklist. Overall, the abstracts showed moderate to low adherence to the checklist (Mdn = 8, IQR = 7–9). The items with high adherence were those pertaining to authors (98.7%), interventions (87.2%), objective (96.2%), outcome (methods; 88.5%), randomization (94.9%), and conclusions (100.0%). The items with moderate adherence were title (50.0%) and participants (57.7%), whereas those with low adherence were blinding (20.5%), numbers randomized (35.9%), and outcome (results; 38.5%). Items with very low adherence (i.e., reported in few of the abstracts) were trial design (12.8%), recruitment (5.1%), numbers analyzed (10.3%), harms (1.3%), trial registration (2.6%), and funding (11.5%). The abstracts’ CONSORT-A scores ranged from 4 to 13 points.
Percentage of abstracts that reported each of the 17 CONSORT-A items (N = 78).
Discussion
Ours is the first study to evaluate reporting quality in RCT abstracts in the field of occupational therapy. We found that the reporting quality in the abstracts was moderate to low. Adherence to the CONSORT-A checklist differed significantly by journal, number of author institutions, first author’s geographic area, and abstract length.
Although RCTs are considered the hallmark of high-quality scientific evidence used to guide clinical interventions (Hariton & Locascio, 2018), RCTs accounted for only a small fraction of the articles published in the five highest impact occupational therapy journals between 2008 and 2018. However, we noted an increase, although gradual, in the number of RCTs published after the most recent version of the CONSORT guidelines was released in 2010. To some extent, the low proportion of RCTs in occupational therapy journals may indicate an emphasis on qualitative studies intended to illuminate personal experiences and perspectives of individuals or specific groups (Upton et al., 2014; Wressle & Samuelsson, 2015) over clinical experimental studies evaluating the efficacy of occupational therapy interventions. In addition, occupational therapy researchers may find it difficult to obtain funding to conduct RCTs, and they may submit RCT manuscripts to non–occupational therapy journals with higher impact factors. Further analysis of the research output on the clinical application of occupational therapy is required to provide a more comprehensive overview of evidence for the efficacy of occupational therapy interventions.
In addition, and in line with a growing concern expressed by the authors of several studies, the low production of RCTs supporting the efficacy of occupational therapy may be a direct consequence of poor scientific knowledge and research skills among occupational therapy practitioners (Lindström & Bernhardsson, 2018; Prieto-Botella et al., 2021; Thomas & Law, 2013; Upton et al., 2014). Our finding that European first authors and involvement of a greater number of institutions were linked to higher reporting quality in RCT abstracts suggests that international collaboration can play an important role in producing higher quality research in occupational therapy.
Most of the 78 RCTs were published in AJOT, but the abstracts of those published in SJOT, AOTJ, and POTP showed higher adherence to CONSORT-A. Intriguingly, although AJOT and CJOT recommended that authors use the CONSORT guidelines to maximize transparency and scientific rigor, the RCT abstracts in these two journals had the lowest CONSORT-A scores. Considering that the five journals’ editorial policies on aims, scope, and requirements were very similar, the only convincing explanation for the differences in reporting quality in their RCT abstracts is differences in their author instructions for abstracts. AJOT’s and CJOT’s manuscript submission guidelines in 2018 limited the abstract to a maximum of 150 words. We found that abstracts written with 150 words or fewer had the lowest CONSORT-A scores.
Although we agree that it is possible for abstracts limited to 150 words to follow the reporting standards of the CONSORT Statement and report sufficient information to assess the internal and external validity of a study (Gutman, 2010; Gutman & Murphy, 2012), it is unlikely that such short abstracts can do so for RCTs. Our observation that RCT abstracts of 200 words or more had greater adherence to CONSORT-A checklist is consistent with the results of previous studies (Fang et al., 2020; Ghimire et al., 2014; Jin et al., 2016; Shaqman et al., 2020; Song et al., 2017). Thus, we support the recommendation by the CONSORT Group (Hopewell et al., 2008) and authors of studies similar to ours (Fang et al., 2020; Fleming et al., 2012; Song et al., 2017) that RCT abstracts be between 250 and 300 words in length to include all items on the CONSORT-A checklist.
Our study suggests that abstract length should be considered a crucial element of the reporting quality of health research RCTs. Although the reporting quality of abstracts does not necessarily reflect the quality of the study, it is vitally important that RCT abstracts provide sufficient information about the study for readers to critically appraise its validity before deciding whether to read the full article (Song et al., 2017). Accordingly, in line with the JAMA Network (2018) guidelines, the updated AJOT guidelines for authors published in 2020 raised the limit for abstracts to 250 words for original research articles, including RCTs (AOTA, 2020).
Consistent with the findings of previous studies, we observed that a considerable proportion of the abstracts included in our study did not report details on blinding, numbers randomized, and outcome (results), and few provided information about trial design, recruitment, numbers analyzed, harms, trial registration, or funding (Berwanger et al., 2009; Fang et al., 2020; Ghimire et al., 2014; Gutman & Murphy, 2012; Hays et al., 2016; Jin et al., 2016; Shaqman et al., 2020; Song et al., 2017; Speich et al., 2019). In terms of reproducibility and replicability, the failure to specify study methodology in RCT abstracts may involve risk of bias that can affect the internal validity of the study as a result of flaws in the interpretation of study data (Savović et al., 2012; Schulz et al., 1995); in terms of application, poor reporting quality in RCT abstracts can lead to ineffective clinical decision making (Fleming et al., 2012; Ghimire et al., 2014; Song et al., 2017). Thus, even journals that endorse the CONSORT guidelines in their instructions to authors should have more active editorial policies to ensure that authors implement those guidelines—for example, a requirement that authors provide a completed CONSORT checklist when submitting their manuscript. Since January 1, 2021, AJOT requires authors to complete a presubmission checklist to help improve the reporting quality of research published in this journal. The changes in AJOT’s editorial policy for abstracts can serve as an excellent precedent for other occupational therapy journals to follow.
Limitations
This study has several limitations. First, our results are based on analysis of only the five highest impact occupational therapy journals, which limits the generalizability of our results to other occupational therapy journals. However, because of the low number of scientific journals specifically aimed at occupational therapy research, our results may sufficiently reflect general trends in the reporting quality of RCT abstracts in occupational therapy journals. In addition, we acknowledge that our journal selection criteria prevented us from including OTJR: Occupation, Participation and Health, one of the most important journals in occupational therapy. We conducted a post hoc analysis of RCT abstracts published during the study period in OTJR to check whether its inclusion would have changed our findings; we observed no relevant differences.
Second, we evaluated reporting quality in RCT abstracts based exclusively on the information provided in each abstract and title. We did not check this information against the full article, which was beyond the scope of this study.
Third, data extraction and article selection were performed manually by two researchers. To minimize misclassification bias, the researchers were randomly assigned the RCT abstracts, scored them independently, and were blinded to study identification details. Interrater agreement was excellent (≥.80). Moreover, we used an effective and reproducible methodology for study selection, data extraction, and scoring of reporting quality.
Implications for Occupational Therapy Research and Practice
The findings of this study have the following implications for occupational therapy research and practice: More RCTs are needed to increase the body of evidence on the clinical effectiveness of occupational therapy and to provide a solid basis for evidence-based practice. Researchers and journal editors should require that abstracts for RCTs follow the CONSORT-A checklist to ensure reporting quality and avoid the potential for biased interpretations. In line with CONSORT-A recommendations, occupational therapy journals should allow RCT abstracts of up to 250 to 300 words. Occupational therapy journal editors should require RCT authors to complete the CONSORT checklist as part of the submission process.
Conclusion
The quality of reporting in RCT abstracts in the five highest impact occupational therapy journals from 2008 to 2018 was far from optimal, especially in the methodological domain. When RCT abstracts fail to report sufficient detail, readers may misinterpret findings and potentially make ineffective clinical decisions. We recommend that occupational therapy journals require authors and peer reviewers to use the CONSORT for Abstracts checklist to promote adequate reporting and transparency in RCT abstracts.
Footnotes
Acknowledgments
Paula Fernández-Pires, Daniel Prieto-Botella, and Desirée Valera-Gran share first authorship of this article. The authors declare no conflict of interest. Alicia Sánchez-Pérez is on the editorial board of the American Journal of Occupational Therapy and had no role in the manuscript peer review process.