Abstract
Background:
In recent years, the number of articles in The American Journal of Sports Medicine (AJSM) has risen dramatically, with an increasing emphasis on evidence-based medicine in orthopaedics and sports medicine.
Hypothesis:
Despite the increase in the number of articles published in AJSM over the past decade, the methodological quality of articles in 2011-2013 has improved relative to those in 2001-2003 and 1991-1993.
Study Design:
Meta-analysis.
Methods:
All articles published in AJSM during 2011-2013 were reviewed and classified by study design. For each article, the use of pertinent methodologies, such as prospective data collection, randomization, control groups, and blinding, was recorded. The frequency of each article type and the use of evidence-based techniques were compared relative to 1991-1993 and 2001-2003 by use of Pearson χ2 testing.
Results:
The number of research articles published in AJSM more than doubled from 402 in 1991-1993 and 423 in 2001-2003 to 953 in 2011-2013. Case reports decreased from 15.2% to 10.6% to 2.1% of articles published over the study period (P < .001). Cadaveric/human studies and meta-analysis/literature review studies increased from 5.7% to 7.1% to 12.4% (P < .001) and from 0.2% to 0.9% to 2.3% (P = .01), respectively. Randomized, prospective clinical trials increased from 2.7% to 5.9% to 7.4% (P = .007). Fewer studies used retrospective compared with prospective data collection (P < .001). More studies tested an explicit hypothesis (P < .001) and used controls (P < .001), randomization (P < .001), and blinding of those assessing outcomes (P < .001). Multi-investigator trials increased (P < .001), as did the proportion of articles citing a funding source (P < .001).
Conclusion:
Despite a dramatic increase in the number of published articles, the research published in AJSM shifted toward more prospective, randomized, controlled, and blinded designs during 2011-2013 compared with 2001-2003 and 1991-1993, demonstrating a continued improvement in methodological quality.
Keywords
Over the past decade, since the initial report on the methodological quality of research in The American Journal of Sports Medicine 4 (AJSM), evidence-based medicine has become the standard in medical research. This effect has been felt in orthopaedics as well, with an increasing emphasis on evidence-based medicine in orthopaedics journals.21,25,31 One of the most telling signs of the growing emphasis of evidence-based medicine in orthopaedics has been the increase in the amount of orthopaedics research over the past 10 years. Hui et al 19 reported that between 2000 and 2011, the number of orthopaedics journals increased from 39 to 61 and the number of orthopaedics articles increased from 5161 to 10,087. Even within AJSM, there has been significant growth over the past decade. Since the last paper on the methodological quality of research in AJSM (Brophy et al 4 ), there has been an increase from 6 to 12 issues per year.
With an increase in the number of issues and articles, there is a risk for dilution of methodological quality. Indeed, underneath the umbrella of evidence-based medicine there exists a wide range of scientific rigor, and several papers have brought to light concerns over the methodological quality of orthopaedics research.2,3,5,10,13,22,23 Therefore, along with the increase in the quantity of evidence-based orthopaedics research, there has been a shift toward efforts to increase quality as well.6,8,14,26,30,33 At the time of the last report in AJSM, there was an overall increase in methodological quality from 1991-1993 to 2001-2003, as represented in part by randomized, prospective controlled trials increasing from 2.7% to 5.9% of all analyzed studies over the 10-year period. 4 Over the subsequent 10 years, there has been significant growth in AJSM. The purpose of the current study was to evaluate whether this increase in publication quantity has affected the methodological quality of research published in the journal, as measured by parameters of study design commonly associated with high-quality research, including but not limited to randomization, blinding, and prospective data collection. 12 The study was designed to test the hypothesis that the methodological quality of articles published in 2011-2013 has improved relative to that in 2001-2003 and 1991-1993.
Methods
To ensure comparability to the previous paper on methodological quality, the current study followed an identical design. All articles published in AJSM from 2011-2013 were analyzed and compared with all articles published in the same journal from 2001-2003 and 1991-1993. The type of study, the use of multi-investigator site collaboration, and the identification of a funding source were recorded for each article during the 3 study periods. Each study was classified according to one of the following types: case report; case series; epidemiological/population description; case control; prospective cohort; prospective (concurrent) cohort; meta-analysis/systematic review; randomized, prospective clinical trial; prospective clinical trial (not randomized); review/instructional; in vivo animal; in vitro animal; cadaveric (human); biomechanical; electromyography (EMG); human cellular/genetic; and other studies. Most of the articles categorized as “Current Concepts Reviews” by AJSM were included under the review/instructional category in this study because most of the articles simply discussed the consensus opinion on a certain topic. Some of the Current Concepts Reviews were true systematic reviews, which were categorized under the meta-analysis/systematic review category. Each article was also classified according to whether a specific hypothesis was tested, as well as according to the use of prospective or retrospective data collection, the use of randomization or a control group, and the blinding of patients, care providers, and/or those assessing outcomes.
In a design similar to the previous study, the anatomic area studied, the type of sport involved, and whether the study focused on a particular patient sex were recorded. Each study was assigned to one of the following anatomic areas: spine, shoulder, elbow, hand, hip, knee, foot and ankle, head, other, or not applicable. The classification of study by type of sport was as follows: American football, baseball, basketball, golf, ice hockey, lacrosse, soccer, swimming, tennis, track and field, throwing sports, varied/other, or not applicable.
The initial data set was compiled and analyzed by a single evaluator and then reviewed by a second author for accuracy. A consensus opinion of the authors was used to determine the final classification of each article. Statistical significance of any change between the 3 study periods was determined by the authors using Pearson χ2 testing with a significance level of P = .05 in Microsoft Excel (v 14.4.6). The null hypothesis was that there would be no significant difference in the proportion of studies using a certain methodological parameter for the 3 study periods.
Results
The number of articles published in AJSM increased dramatically from 402 in 1991-1993 and 423 in 2001-2003 to 953 articles in 2011-2013.
The distribution of articles by type suggested that the increased number of articles did not decrease the quality of research published in AJSM (Table 1). While the frequency of case series increased from 15.3% of articles in 2001-2003 to 30.2% of articles in 2011-2013 (P < .001), case reports continued to decrease, from 15.2% of articles in 1991-1993 to 10.6% of articles in 2001-2003 and 2.1% of articles in 2011-2013 (P < .001). Randomized, prospective clinical trials continued to increase, from 2.7% of articles in 1991-1993 to 5.9% of articles in 2001-2003 and 7.4% of articles in 2011-2013 (P = .007). Meta-analysis/literature review studies increased from 0.2% in 1991-1993 and 0.9% in 2001-2003 to 2.3% of articles in 2011-2013 (P = .01). Human/cellular genetic studies increased from 0.2% in 1991-1993 and 1.4% in 2001-2003 to 2.2% of articles in 2011-2013 (P = .029).
Distribution of the Type of Article Published by The American Journal of Sports Medicine a
Results are reported as percentage of articles published in The American Journal of Sports Medicine during the 3 study periods.
The methodology used in published articles indicated that the quality of research appeared to be improving over the study period (Table 2). The use of retrospective data collection decreased from 51.2% in 1991-1993 to 35.5% in 2001-2003 and to 25.4% in 2011-2013 (P < .001). Prospective data collection increased from 14.9% in 1991-1993 to 25.4% in 2001-2003 and 39.1% in 2011-2013 (P < .001). Articles testing an explicit hypothesis rose from 41% in 1991-1993 and 59.3% in 2001-2003 to 66.4% in 2011-2013 (P < .001). Articles with controls increased from 14.9% in 1991-1993 and 20% in 2001-2003 to 35.5% in 2011-2013 (P < .001). Finally, randomization increased as well, from 3.5% in 1991-1993 to 7.3% in 2001-2003 and 15.8% in 2011-2013 (P < .001). Over the 22-year time period, the frequency of multi-investigator site collaboration increased from 0.25% in 1991-1993 and 0.47% in 2001-2003 to 3.5% of articles in 2011-2013 (P < .001). The frequency of articles identifying a funding source increased as well, from 19.7% in 1991-1993 and 28.2% in 2001-2003 to 48.5% in 2011-2013 (P < .001).
Distribution of Research Methodology Used in Articles Published by The American Journal of Sports Medicine a
Results are reported as percentage of articles published in The American Journal of Sports Medicine during the 3 study periods.
Analysis of the anatomic focus of articles again showed a fairly stable distribution over the 22-year time period (Figure 1). The knee was again the most commonly studied area of investigation, with 47.3% of articles in 1991-1993 and 2001-2003 and 44.8% of articles in 2011-2013 (P = .6). The number of articles studying the shoulder continued to increase over this time period, from 10.4% in 1991-1993 and 17.6% in 2001-2003 to 20.4% in 2011-2013 (P < .001). Articles focused on the hip increased from 2.0% in 1991-1993 and 0.7% in 2001-2003 to 9.5% in 2011-2013 (P < .001). Articles studying the head also increased, from 0.5% in 1991-1993 and 1.2% in 2001-2003 to 2.7% in 2011-2013 (P = .01).
Focus by anatomy of articles published in The American Journal of Sports Medicine. NA, not applicable. *Statistically significant change.
There has been less focus of research by sport over time (Table 3), with an increasing percentage of articles (72.6% in 2011-2013) not addressing any particular sport (P < .001). There was a decrease in the percentage of articles focused on American football, track and field, throwing athletes, and tennis. There was not a significant increase for any single sport.
Focus by Sport of Articles Published in The American Journal of Sports Medicine a
Results are reported as percentage of articles published in The American Journal of Sports Medicine during the 3 study periods. NA, not applicable.
Of the articles that mentioned patient sex (Figure 2), there was an increase in the number of articles with both male and female subjects from 49.5% in 1991-1993 to 63.9% of articles in 2011-2013 (P < .001). Of the articles that studied patients from a single sex, the subjects were still more often male, but over the 22-year time period these studies decreased in frequency from 24.1% in 1991-1993 to 14.8% of articles in 2011-2013 (P < .001). Articles studying only female subjects decreased as well, from 7.7% in 1991-1993 to 2.5% in 2011-2013 (P < .001).
Focus by sex of articles published in The American Journal of Sports Medicine. Statistically significant changes occurred in each category. NA, not applicable.
Discussion
Over the past 10 years since the initial study on methodological quality, AJSM has expanded significantly, with more published articles in 2011-2013 compared with 1991-1993 and 2001-2003 combined. While this shift in the volume of publication makes it challenging to interpret trends in quality, certain trends emerge. The frequency of case reports dropped dramatically. Importantly, the proportion of randomized, prospective clinical trials has also continued to increase, now accounting for 7.1% of published articles in AJSM. This value is similar to a recent study estimating that randomized controlled trials currently account for 6% of current sports medicine literature. 17 Regardless, randomized, prospective clinical trials still represent a small proportion of the total published research, with case series and descriptive studies representing 30.2% and 13% of articles, respectively. Meta-analysis/systematic review articles have more than doubled over the past 10 years, although the quality of orthopaedics meta-analyses in general continues to be called into question.9,13
On a positive note, the past 10 years also demonstrate a continued increase in the transparency of orthopaedics research funding, with now close to half of articles published in AJSM reporting a source of funding. Given the potential for bias associated with industry funding, especially in light of a recent study by Khan et al 20 finding a significant positive association between clinical trial outcome and source of funding, it is imperative that research authors report funding.
At the time of the last paper on methodological quality, there was a shift toward more prospective and randomized research designs published in AJSM from 1991-1993 to 2001-2003. 4 This trend has continued to be borne out over the past 10 years, with the articles published in 2011-2013 representing a dramatic increase in the use of prospective data collection, hypothesis testing, controls, randomization, and blinding. There has also been a continued increase in the overall proportion of randomized, prospective clinical trials and multi-investigator site trials, both of which embody an important step toward higher methodological quality. The findings of the current study, when examined in conjunction with 2 recent studies demonstrating a chronological increase in the overall level of evidence in AJSM,8,29 show that sports medicine research appears to be moving in the right direction.
Nevertheless, there remains significant room for improvement 15 as even articles with study designs associated with high methodological quality are not without bias. 7 Furthermore, the numbers of randomized, prospective clinical trials and multi-investigator sites continue to represent a small proportion of the total published research. 18 This is particularly important as it has been shown that orthopaedic surgery and sports medicine lag behind other medical disciplines in the percentage of collaborative, multicenter trials. 5 Up to 40% of the research published in leading general medical journals and 13% to 20% of research in other surgical subspecialty journals was the result of multicenter collaborations 5 compared with the much-improved but still paltry 3.5% over 2011-2013 for AJSM. Collaborative studies have been shown to be cited more often than other types of research. 11 Thus, despite important steps toward improving the overall methodological quality of orthopaedic sports medicine research, efforts should continue in this area.
A greater proportion of articles in 2011-2013 included both male and female participants in their study design relative to 2001-2003. Even with the discrepancy in injury rates between male and female subjects, for example, in noncontact anterior cruciate ligament tears, 27 a disparity remains among articles focusing on patient sex, with male-only research outnumbering female-centric research by almost 6 times, suggesting potential for a more equitable distribution.
There has been less sport-focused research over time, with significant declines in the percentages of articles focusing on American football, track and field, throwing athletes, and tennis. Regarding anatomic studies, the increase in articles focused on the hip and head is not surprising given recent advances in hip arthroscopy 1 and the growing awareness of and concern about traumatic brain injuries.16,32
As medicine moves toward pay-for-performance, the need for high methodological quality in sports medicine literature becomes even more important. 28 To accurately assess orthopaedic outcomes, there must first be well-constructed trials demonstrating well-accepted measures of surgical success. Of course, it should be the priority of all orthopaedic surgeons to provide patients with evidence-based care and to avoid unnecessary health care expenditure. As these ideals become entrenched in value-based care, it remains to be seen whether pay-for-performance implementation will improve outcomes. Given concerns over the cost of health care, however, value-based care is likely here to stay and, along with it, an opportunity for the orthopaedic sports medicine community to push the health of our patients forward.
There are important limitations of the current study. Sampling only three 3-year intervals over the 22-year period, and sampling only from AJSM, introduces the inherent possibility of selection bias. In light of the dramatic increase in the number of publications over the past 20 years, however, it would have been infeasible to study every year in every sports medicine journal. Three-year periods seemed to provide an adequate, but manageable, sample size to capture overall trends over the time frame. Similarly, AJSM has the highest impact factor of orthopaedics journals, and while not being able to capture the entire body of sports medicine research, AJSM again seemed to provide for sufficient volume of high-quality articles to assess the overall state of sports medicine literature. Another limitation of the current study is the generalization of several qualitative parameters as a proxy for methodological quality. Ideally, the study would include more parameters of quality such as type of funding (public, private, government, industry), type of outcome scores used (subjective patient-reported vs objective clinician-measured), use of statistician, geography of single-center and multicenter collaboration (country, continent), and use of a methodological quality questionnaire (Coleman, Delphi, Jadad, Cochrane, Detsky, Quality Appraisal Tool, Consolidated Standards of Reporting Trials [CONSORT], Strengthening the Reporting of Observational Studies in Epidemiology [STROBE], Newcastle, CLEAR-NPT). Unfortunately, these data are not readily available for the previously studied time periods. The same issue prevented comparison of the levels of evidence in the meta-analyses and systematic reviews, as well as the very limited number of such studies 10 and 20 years ago. Not every randomized, controlled trial is inherently high quality, and therefore future studies may benefit from the use of a validated methodological quality questionnaire like CONSORT or STROBE to more quantitatively assess trends in orthopaedic sports medicine literature. Finally, the study did not assess types of studies (therapeutic, prognostic, diagnostic, economic) and levels of evidence, 34 which are significant parameters of research quality.
In conclusion, the quality of research published in AJSM has not been diluted and appears to actually have improved, despite a doubling in volume of articles published. Over the next 10 years, the challenge will be to continue to push the orthopaedic sports medicine literature toward higher methodological quality. Compliance of orthopaedics research with CONSORT or STROBE guidelines has been shown to be relatively poor. 22 Given the improvement in methodological quality among medical journals that adopted a checklist such as CONSORT, 24 it may be helpful to consider a formal methodological checklist as requirement for publication of clinical trials. Adoption of such a system could further improve the methodological quality in orthopaedic sports medicine research.
Footnotes
One or more of the authors has declared the following potential conflict of interest or source of funding: R.H.B. is a member of the editorial board of The American Journal of Sports Medicine.
An online CME course associated with this article is available for 1 AMA PRA Category 1 Credit™ at 
. In accordance with the standards of the Accreditation Council for Continuing Medical Education (ACCME), it is the policy of The American Orthopaedic Society for Sports Medicine that authors, editors, and planners disclose to the learners all financial relationships during the past 12 months with any commercial interest (A ‘commercial interest’ is any entity producing, marketing, re-selling, or distributing health care goods or services consumed by, or used on, patients). Any and all disclosures are provided in the online journal CME area which is provided to all participants before they actually take the CME activity. In accordance with AOSSM policy, authors, editors, and planners’ participation in this educational activity will be predicated upon timely submission and review of AOSSM disclosure. Noncompliance will result in an author/editor or planner to be stricken from participating in this CME activity.