Analysis of the Quality of Prospective Randomized Controlled Trials for Treatment of Boxer’s Fractures

Abstract

Background

This study aimed to review level I and II therapeutic studies on boxer’s fractures to measure variation in quality among the highest level study designs.

Methods

We used quantitative measures of study quality to evaluate prospective randomized controlled trials (RCTs) of treatments of boxer’s fractures. A search of PubMed, using terms “boxer’s fracture” and “fifth metacarpal neck fracture” identified 164 articles from 1961 to 2019. From this list, we identified 6 RCTs. Two observers classified each trial according to 3 systems: the Oxford Levels of Evidence, the modified Coleman Methodology Score, and the revised Consolidated Standards of Reporting Trials (CONSORT) score.

Results

The 2 reviewers were consistent in their use of the Oxford Levels of Evidence (100% agreement). The differences between the average modified Coleman Methodology scores and the average CONSORT scores assigned by the 2 observers were not significant (46.2 vs 45.3 points, κ = 0) and (13.7 vs 14.3 points, κ = 0.33), respectively. Both observers rated all the studies as level I and as unsatisfactory according to the Coleman Methodology Score (100% and 100%), and less than half as unsatisfactory according to the CONSORT score (50% and 17%). Areas of deficiency included randomization, blinding, group comparability, clinical effect measurements, and allocation into treatment arms.

Conclusion

Classifying orthopedic scientific reports according to the levels of evidence implies a degree of respect for level I and II studies that may not always be merited. Our data suggest that the quality of higher level studies, namely those involving boxer’s fractures, varies and may often be unsatisfactory when critically evaluated.

Keywords

boxer’s fracture fifth metacarpal neck fracture prospective randomized controlled trials quality review analysis study design

Introduction

The Oxford Levels of Evidence are commonly assigned to medical publications and tend to carry a large degree of weight when accessing the validity of study conclusions and study design.^1-4 However, these conclusions may be problematic as high levels of evidence do not always correlate to a high-quality scientific study.³ Conversely, studies with lower levels of evidence are often undervalued even when they may be higher quality scientific studies in terms of study design or even clinical impact.⁵

Fractures of the fifth metacarpal neck, or boxer’s fractures, are common injuries, especially in the young male population.^6-8 Traditionally, boxer’s fractures have been treated nonoperatively with closed reduction and splint immobilization.^9,10 However, a recent systematic review of several prospective randomized controlled trials demonstrated that there is no clear benefit to reduction and cast immobilization when compared with soft wrap techniques.¹¹

The purpose of this study was to review the quality of high-level scientific studies (level I and II prospective randomized controlled trials) in the treatment of boxer’s fractures. We hypothesized that the methods of these randomized controlled trials would be sufficient to support the conclusion that boxer’s fractures do not need reduction and can simply be treated with soft wrap techniques.

Materials and Methods

A search of PubMed,¹² with the use of combinations of the terms “boxer’s fracture” and “fifth metacarpal neck fracture,” identified 164 articles published from 1961 to 2019. From this list, we identified 6 prospective randomized controlled trials that specifically addressed the nonoperative management of boxer’s fractures.^13-18 Two observers (both board-certified hand and upper extremity surgeons with MD degrees) classified each trial according to 3 systems: the Oxford Levels of Evidence,¹ the modified Coleman Methodology Score (0-100 point scale),^3,19-21 and the revised Consolidated Standards of Reporting Trials (CONSORT) score.²²

Each observer first assigned an Oxford Level of Evidence for each paper.¹ Using this system, level I studies were defined as high-quality randomized controlled trials with greater than 80% follow-up. Level II studies were defined as low-quality studies, such as randomized controlled trials with less than 80% follow-up, or prospective/retrospective cohort/case-control studies without randomization, among other considerations.

Each observer then assigned a modified Coleman Methodology Score to each study.^3,19-21 Initially, the Coleman Methodology Score was developed to grade the methodology of clinical studies on patellar and Achilles tendinopathy.^19-21 This system was subsequently modified in a quality analysis study of lateral epicondylitis clinical trials to best access the quality of prospective randomized controlled trials.³ Using this modified Coleman Methodology Score, 15 individual criteria were used to create a solitary score for each study. The total score had a range of 0 to 100 points. The criteria and scoring system are illustrated in Table 1, based on the work performed by Cowan et al.³ We used the categorical rating system, which defines each study to be “excellent” if the score is 85 to 100 points, “good” for 70 to 84 points, “fair” for 55 to 69 points, and “poor” for less than or equal to 54 points. A rating of “fair” or “poor” was deemed unsatisfactory (Table 1).

Table 1.

Modified Coleman Methodology Score.³

Criteria	Points
Inclusion criteria
Not described	0
Described without %s given	3
Enrollment rate <80%	6
Enrollment rate >80%	9
Power
Not reported	0
>80%, methods not described	3
>80%, methods described	6
Alpha error
Not reported	0
<0.05	3
<0.01	6
Sample size
Not stated or <20	0
20-40	3
41-60	6
>60	9
Randomization
Not randomized	0
Modified/Partial
Not blinded	2
Blinded	4
Complete
Not blinded	6
Blinded	8
Follow-up
Short term (<6 mo)
Patient retention <80%	0
Patient retention 80%-90%	2
Patient retention >90%	4
Medium term (6-24 mo)
Patient retention <80%	2
Patient retention 80%-90%	4
Patient retention >90%	6
Long term (>24 mo)
Patient retention <80%	4
Patient retention 80%-90%	6
Patient retention >90%	8
Patient analysis
Incomplete	0
Complete	3
Complete and intention-to-treat based	6
Blinding
None	0
Single	2
Double	4
Triple	6
Similarity in treatment
No	0
Similar cointerventions	3
No cointerventions	6
Treatment description
None	0
Fair	3
Adequate	6
Group comparability
Not comparable	0
Partially comparable	3
Comparable	6
Outcome assessment
Written assessment by patient with assistance	0
Written assessment by patient without assistance	2
Independent investigator	4
Recruited patients	6
Description of rehabilitation protocol
Not reported	0
Not adequately described	2
Well described	4
Clinical effect measurement
Effect size
Not reported	0
<50%	2
50%-75%	4
>75%	6
or Relative risk reduction
Not reported	2
<25%	3
>25%	6
or Absolute risk reduction
Not reported	0
<10%	3
>10%	6
No. of patients to treat
Not reported	0
Reported	4

Finally, each observer then assigned a revised CONSORT score to each paper. The revised CONSORT score is a 22-item checklist that assesses various aspects of prospective randomized trials.²² The purpose of this checklist is to provide a standardized approach to compare the conduct of trials and the validity of their results.²² Within the 22-item checklist, the trial is given either 1 point for each criterion met or 0 points if the criteria were not met, with a maximum score of 22 points. The categorical rating system assigns each study to be “excellent” if the score is 18 to 22 points, “good” for 13 to 17 points, “fair” for 8 to 12 points, and “poor” for less than or equal 7 points. A rating of “fair” or “poor” was deemed unsatisfactory (Table 2).

Table 2.

CONSORT Checklist of Items to Include When Reporting a Randomized Trial.³

Paper section and topic	Item	Description
Title and abstract	1	How participants were allocated to interventions (eg, “random allocation,” “randomized,” or “randomly assigned”)
Introduction: Background	2	Scientific background and explanation of rationale
Methods
Participants	3	Eligibility criteria for participants and the settings and locations where the data were collected
Interventions	4	Precise details of the interventions intended for each group and how and when they were actually administered
Objectives	5	Specific objectives and hypotheses
Outcomes	6	Clearly defined primary and secondary outcome measures and, when applicable, any methods used to enhance the quality of measurements (eg, multiple observations, training of assessors)
Sample size	7	How sample size was determined and, when applicable, explanation of any interim analyses and stopping rules
Randomization—sequence generation	8	Method used to generate the random allocation sequence, including details of any restrictions (eg, blocking, stratification)
Randomization—allocation concealment	9	Method used to implement the random allocation sequence (eg, numbered containers or central telephone), clarifying whether the sequence was concealed until interventions were assigned
Randomization—implementation	10	Who generated the allocation sequence, who enrolled participants, and who assigned participants to their groups
Blinding (masking)	11	Whether participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment. When relevant, how the success of blinding was evaluated
Statistical methods	12	Statistical methods used to compare groups for primary outcome(s); methods for additional analyses, such as subgroup and adjusted analyses
Results
Participant flow	13	Flow of participants through each stage (a diagram is strongly recommended). Specifically, for each group report the numbers of participants randomly assigned, receiving the intended treatment, completing the study protocol, and analyzed for the primary outcome. Describe protocol deviations from study as planned, together with reasons
Recruitment	14	Dates defining the periods of recruitment and follow-up
Baseline data	15	Baseline demographic and clinical characteristics of each group
Numbers analyzed	16	No. of participants (denominator) in each group included in each analysis and whether the analysis was by “intention-to-treat.” State the results in absolute numbers when feasible (eg, 10/20, not 50%)
Outcomes and estimation	17	For each primary and secondary outcome, a summary of results for each group, and the estimated effect size and its precision (eg, 95% confidence interval)
Ancillary analyses	18	Address multiplicity by reporting any other analyses performed, including subgroup and adjusted analyses, indicating those prespecified and those exploratory
Adverse events	19	All important adverse events or side effects in each intervention group
Discussion
Interpretation	20	Interpretation of the results, taking into account study hypotheses, sources of potential bias or imprecision, and the dangers associated with multiplicity of analyses and outcomes
Generalizability	21	Generalizability (external validity) of the trial findings
Overall evidence	22	General interpretation of the results in the context of current evidence

Note. CONSORT = Consolidated Standards of Reporting Trials.

For continuous variables, the Student t-test was used to assess the interobserver reliability of each scoring system. For categorical variables, such as those seen in the Modified Coleman and CONSORT scores, the kappa statistic was used to evaluate the interobserver reliability of each scoring system. Kappa scores range from −1 to 1, with 1 indicating perfect agreement and a kappa of 0 indicating agreement equivalent to chance.²³ A categorical rating of the reliability, first described by Landis and Koch,²⁴ was employed. This rating system defines a score of less than 0 as “less than chance agreement,” 0.01 to 0.20 as “slight agreement,” 0.21 to 0.40 as “fair agreement,” 0.41 to 0.60 as “moderate agreement,” 0.61 to 0.80 as “substantial agreement,” and 0.81 to 0.99 as almost “perfect agreement.”

Results

Oxford Levels of Evidence

There was 100% agreement between the observers according to the Oxford Levels of Evidence (κ = undefined; P = undefined). Both observers 1 and 2 agreed that all 6 publications fulfilled the criteria to be considered level I.

Modified Coleman Methodology Score

The differences between the mean modified Coleman Methodology scores assigned by the 2 observers were not significant (45.0 compared with 45.3 points; P = .96). The 2 observers had agreement equivalent to chance regarding the modified Coleman Methodology scores, according to the Landis and Koch²⁴ categorical ratings (κ = 0; P = undefined), but did agree on 83% of studies.

When combining the assessments of both observers, the mean modified Coleman Methodology Score was 45.2 points (range, 34-67 points). Observer 1 assigned 5 (83%) of the studies a “poor” categorical rating and 1 (17%) of the studies a “fair” categorical rating. Observer 2 assigned all 6 (100%) of the studies a “poor” categorical rating. No study was assigned an “excellent” or “good” categorical rating. The modified Coleman Methodology Score breakdown for each reviewed study is listed below (Table 3).

Table 3.

Modified Coleman Methodology Score Breakdown by Reviewed Studies.

Criteria (max score available)	Average score	Total % score earned
Inclusion criteria (9)	5.50	61
Power (6)	3.75	63
Alpha error (6)	2.25	38
Sample size (9)	5.75	64
Randomization (8)	4.67	58
Follow-up (8)	3.67	46
Patient analysis (6)	3.00	50
Blinding (6)	0.3	14
Similarity in treatment (6)	5.75	96
Treatment description (6)	3.25	54
Group comparability (6)	2.25	38
Outcome assessment (6)	1.00	17
Description of rehabilitation protocol (4)	2.50	63
Clinical effect measurement (6)	0.67	11
No. of patients to treat (4)	0.33	8
Total score	45.2

Revised CONSORT Statement

The differences between the 2 observers for the mean revised CONSORT Statement scores were not significant (13.7 compared with 14.3 points; P = .72). The 2 observers had a “fair” agreement regarding the revised CONSORT Statement scores according to the categorical rating (κ = 0.33; P = .27) but did agree on 67% of studies.²⁴

The mean of the revised CONSORT Statement scores for the 2 observers was 14.0 (range, 8-17 points). Observer 1 assigned 3 (50%) of the studies a “good” categorical rating and 3 (50%) of the studies a “fair” categorical rating. Observer 2 assigned 5 (83%) of the studies a “good” categorical rating and 1 (17%) of the studies a “fair” categorical rating. No study was assigned an “excellent” or “poor” categorical rating. The CONSORT score breakdown for each reviewed study is listed below (Table 4).

Table 4.

CONSORT Score Breakdown by Reviewed Studies.

Criteria	Average score
1. Title and abstract	1.0
2. Introduction: Background	1.0
3. Methods: Participants	1.0
4. Methods: Interventions	1.0
5. Methods: Objectives	0.42
6. Methods: Outcomes	0.75
7. Methods: Sample size	0.75
8. Methods: Randomization—sequence generation	0.33
9. Methods: Randomization—allocation concealment	0.08
10. Methods: Randomization—implementation	0.00
11. Methods: Blinding	0.25
12. Methods: Statistical methods	0.83
13. Results: Participant flow	0.42
14. Results: Recruitment	0.83
15. Results: Baseline data	0.58
16. Results: Numbers analyzed	0.58
17. Results: Outcomes and estimation	0.83
18: Results: Ancillary analyses	0.83
19. Results: Adverse events	0.25
20. Discussion: Interpretation	0.75
21. Discussion: Generalizability	0.75
22. Discussion: Overall evidence	0.75
Total score	14.0

Note. CONSORT = Consolidated Standards of Reporting Trials.

Discussion

Medical professionals often look to prospective randomized controlled trials to be gold standard scientific reports. Consequently, even astute readers might fall into the trap of thinking that simply because a study qualifies as being high-level evidence, it is inherently a high-quality study and thus has well-founded methods and results. This study was conducted to assess the quality of high-level prospective randomized control trials, specifically in the treatment of boxer’s fractures. We hypothesized that the current body of literature would be sound enough to support the conclusion that boxer’s fractures do not need reduction and can simply be treated with soft wrap techniques. However, similar to other studies analyzing the quality of evidence for scientific studies,³ we found that the high-level evidence available was often unsatisfactory.

All 6 of the published studies that met our inclusion criteria were considered to be level I trials according to the Oxford Levels of Evidence. The modified Coleman Methodology Score further assessed the strengths and flaws of each study. Although these trials were considered high-level studies according to the Oxford Levels of Evidence, we found most studies were rated “poor” according to the modified Coleman Methodology Score, with only 2 (33%) being rated “fair” by 1 observer. The major limitations to the analyzed studies were numerous: Most failed to implement appropriate blinding, few had group comparability, most neglected to report clinical effect measurements, and only 1 reported the number of patients needed to treat. The only common strengths in these studies were reporting alpha error, having appropriate power, and not allowing cointerventions.

Our results regarding the revised CONSORT Statement were slightly better than the results for the modified Coleman Methodology Score. All the studies were rated as “good” or “fair,” with most of the studies rated “good” (50% and 83%) and the remainder rated “fair” (50% and 17%). Nearly all studies were able to earn full points for simple categories, such as having an appropriate title and abstract, background, inclusion criteria, and explanation of statistical methods. However, most neglected to state objectives or hypotheses. More importantly, almost every study failed to adequately conduct or properly explain randomization, generate appropriate allocation into treatment arms, or conduct suitable blinding. When critically assessed, these flaws in study design are oversights and can have harmful effects on the quality of study design.

The strengths of this study were the independent analyses of 2 qualified observers. Furthermore, we implemented 3 grading criteria that are quite elaborate and have been demonstrated in the past to adequately analyze scientific studies.^3,19-22 Using the Modified Coleman Methodology Score and the revised CONSORT Statement, our reviewers were able to grade each study with statistically insignificant variability, according to Student t-test analysis using a quantitative score for each method. However, 1 weakness of our studies was the poor interobserver reliability according to the kappa statistic for categorical rating systems between scoring methods. This can most likely be explained due to the limited number (6) of high-level trials available for review regarding nonoperative boxer’s fracture management. Nevertheless, both observers did agree on 67% and 83% of studies for each method of analysis, which some authors argue is a better assessment of interobserver reliability than the kappa statistic when raters are well trained and little guessing is likely to exist.²⁵

Our data and analysis show that high-level studies do not always ensure that appropriate criteria are met, at least concerning the modified Coleman Methodology Score and revised CONSORT Statement. Regarding our hypothesis, we believed that current literature supporting nonoperative management with soft wrap techniques for boxer’s fractures would be high quality, but we found this information unsatisfactory when critically reviewed. This finding highlights a potential opportunity for future high-quality, high-level studies analyzing nonoperative management of boxer’s fractures. This review, similar to those published in the past,³ serves as a further reminder to critically assess scientific studies that influence clinical decision-making, regardless of study design or level of evidence.

Footnotes

Ethical Approval

This study was approved by our institutional review board.

Statement of Human and Animal Rights

The making of this study did not require the use of patient informed consent, given the study design.

Statement of Informed Consent

This was not an experimental study involving humans or animals and thus did not need to undergo institutional review.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Cole M. Patrick

Gilberto A. Gonzalez

John C. Dunn

References

Oxford levels of evidence. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009.

Cook

Guyatt

Laupacis

, et al. Clinical recommendations using levels of evidence for antithrombotic agents. Chest. 1995;108(4 suppl):227S-230S

Cowan

Lozano-Calderon

Ring

Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 2007;89(8):1693-1699.

Winters

Eskes

Weir

, et al. Treatment of medial tibial stress syndrome: a systematic review. Sports Med. 2013;43(12):1315-1333.

Carr

AJ.

Evidence-based orthopaedic surgery: what type of research will best improve clinical practice?

J Bone Joint Surg Br. 2005;87(12):1593-1594.

Chung

Spilson

SV.

The frequency and epidemiology of hand and forearm fractures in the United States. J Hand Surg Am. 2001;26(5):908-915.

Hunter

Cowen

NJ.

Fifth metacarpal fractures in a compensation clinic population. Journal of Occupational and Environmental Medicine. 1971;13(12):603-604.

Nakashian

Pointer

Owens

, et al. Incidence of metacarpal fractures in the US population. Hand (N Y). 2012;7(4):426-430.

Theeuwen

Lemmens

van Niekerk

JL.

Conservative treatment of boxer’s fracture: a retrospective analysis. Injury. 1991;22(5):394-396.

10.

Westbrook

Davis

Armstrong

, et al. The clinical significance of malunion of fractures of the neck and shaft of the little finger metacarpal. J Hand Surg Eur Vol. 2008;33(6):732-739.

11.

Dunn

Kusnezov

Orr

, et al. The boxer’s fracture: splint immobilization is not necessary. Orthopedics. 2016;39(3):188-192.

12.

PubMed. https://pubmed.ncbi.nlm.nih.gov/.

13.

Braakman

Oderwald

Haentjens

MH.

Functional taping of fractures of the 5th metacarpal results in a quicker recovery. Injury. 1998;29(1):5-9.

14.

Hansen

TB.

The treatment of fractures of the ring and little metacarpal necks. A prospective randomized study of three different types of treatment. J Hand Surg Br. 1998;23(2):245-247.

15.

Kuokkanen

Mulari-Keranen

Niskanen

, et al. Treatment of subcapital fractures of the fifth metacarpal bone: a prospective randomised comparison between functional treatment and reposition and splinting. Scand J Plast Reconstr Surg Hand Surg. 1999;33(3):315-317.

16.

McMahon

Woods

Burge

PD.

Initial treatment of closed metacarpal fractures. A controlled comparison of compression glove and splintage. J Hand Surg Br. 1994;19(5):597-600.

17.

Statius Muller

Poolman

van Hoogstraten

, et al. Immediate mobilization gives good results in boxer’s fractures with volar angulation up to 70 degrees: a prospective randomized trial comparing immediate mobilization with cast immobilization. Arch Orthop Trauma Surg. 2003;123(10):534-537.

18.

van Aaken

Fusetti

Luchina

, et al. Fifth metacarpal neck fractures treated with soft wrap/buddy taping compared to reduction and casting: results of a prospective, multicenter, randomized trial. Arch Orthop Trauma Surg. 2016;136(1):135-142.

19.

Coleman

Khan

Maffulli

, et al. Studies of surgical outcome after patellar tendinopathy: clinical significance of methodological deficiencies and guidelines for future studies. Scand J Med Sci Sports. 2000;10(1):2-11.

20.

Jakobsen

Engebretsen

Slauterbeck

JR.

An analysis of the quality of cartilage repair studies. J Bone Joint Surg Am. 2005;87(10):2232-2239.

21.

Tallon

Coleman

Khan

, et al. Outcome of surgery for chronic Achilles tendinopathy. A critical review. Am J Sports Med. 2001;29(3):315-320.

22.

Moher

Schulz

Altman

, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1:2.

23.

Viera

Garrett

JM.

Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360-363.

24.

Landis

Koch

GG.

The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.

25.

McHugh

ML.

Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282.