Quotation errors related to the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) study

Literature search

A literature review was conducted using EMBASE, Google Scholar, Medline, Ovid Cited Reference search and Web of Science databases. We included literature that referenced the ProFHER trial in their bibliography, using publications from March 2015 to November 2019. The final search was conducted on the 1st November 2019. We excluded duplicate articles, non-English language articles, textbooks, PhD theses and technical tip videos.

Literature evaluation

Evaluation of the remaining full text articles was independently conducted by two reviewers. Quotation errors were assessed and classified as either major or minor errors, according to a previously validated framework. ¹² All errors were then discussed and any disparities between the two datasets were reviewed and reclassified with agreement of both reviewers. The senior author was consulted when a consensus could not be met by the primary reviewers.

Major errors were defined as references that did not substantiate, were unrelated to or contraindicated the findings of the ProFHER trial. Minor errors were defined as either trivial numerical inaccuracies or statements that oversimplified or generalised the findings of ProFHER but did not considerably change the original assertion. Multi-referencing refers to the use of multiple references to support the same quotation.

Data analysis

Data were collected in a pre-designed Microsoft Excel sheet (Microsoft 2019). The paper's title, type and class of error were recorded. For each error, the rationale for error classification was noted, and whether the error statement was mono- or multi-referenced as well as the number of ProFHER citations in the paper.

A kappa co-efficient was calculated to measure interobserver reliability when assessing for quotation errors.

Major errors

Of the 10 major errors (Table 2), 6/10 (60%) concerned statements not supported by the ProFHER paper.^13–18 For example one paper incorrectly cited ProFHER as reporting that certain fracture patterns had a worse prognosis. ¹⁵

OPEN IN VIEWER

Table 2.

Breakdown of major errors.

Error type	Quotation error	Correct finding from ProFHER	Reference
Unsubstantiated	70% of secondary surgery occurred within the first year	This was not mentioned in the ProFHER paper	Launonen et al. 13
Unsubstantiated	The literature supports the conservative management of non-displaced factures of the humerus	ProFHER included only displaced fractures	Antonios et al. 14
Unsubstantiated	Certain fracture patterns have worse prognosis	This was not mentioned in the ProFHER paper	Hudgens et al. 15
Unsubstantiated	Outcomes when comparing surgical versus non-surgical management are similar regardless of fracture pattern	ProFHER only included displaced fractures. Fractures that had a clear indication for surgery were excluded	Gracitelli et al. 16
Unsubstantiated	ProFHER is against surgical fixation	ProFHER did not demonstrate inferior outcomes with surgical treatment – they were not significantly different and a number of fractures that had a clear indication for surgery were excluded	Meunier et al. 17
Unsubstantiated	ProFHER is unclear if younger patients may benefit from surgical fixation	An age group subanalysis was performed that found no significant difference between >65 and <65 in either group	von Keudell et al. 18
Unrelated	Patients from nursing homes are treated conservatively with the primary aim being pain relief	This was not mentioned in the ProFHER paper	Spross et al. 19
Unrelated	A certain subset of patients with high grade acromioclavicular joint separation may benefit from conservative management	This was not mentioned in the ProFHER paper	Cooper et al. 20
Unrelated	Ankle injury management (AIM) trial reference	This was not mentioned in the ProFHER paper	Costa et al. 21
Contraindicated	There were significantly higher complications in the ProFHER surgical group	There were higher numbers of complications related to the shoulder fracture in the surgical group; however, this was not significant	Launonen et al. 22

Papers that cited ProFHER to support assertions that were unrelated to the topic of ProFHER accounted for 3/10 (30%) of major errors.^19–21 One paper cited ProFHER as supporting non-surgical management for patients with acromioclavicular joint separation. ²⁰ This was not relevant to the ProFHER trial.

The remaining 1/10 (10%) major error was a contradiction to the findings reported in ProFHER. The paper quoted a higher risk of complications and re-operation in the surgical group. ²² ProFHER reported a non-significant difference in complications related to the shoulder fracture or its treatment and 9% of patients in both groups required further surgery.

Minor errors

Of the 25 minor errors, 19/25 (76%) oversimplified or generalised the findings of the ProFHER trial. For example, one paper cited ProFHER as reporting no benefit to using operative locked plating to surgically correct displaced PHFs when compared to non-surgical management. ²³ ProFHER described both the use of locking plates and other modalities (such as hemiarthroplasty and intramedullary nails, which accounted for almost 18% of patients) and this was not made clear. The remaining 6/25 (24%) quoted minor numerical or grammatical/spelling inaccuracies that did not change the overall interpretation of ProFHER's results.

Multi-referencing

Of the identified errors 16/35 (46%) were part of groups of references used to support single assertions.

Interobserver agreement

The kappa co-efficient in relation to interobserver reliability was 0.789 indicating substantial agreement between the two independent observers.

Quotation error rate

The quotation error rate (13%) was lower than has been previously shown in the general medical and orthopaedic literature. A systematic review and meta-analysis of quotation accuracy across the medical literature demonstrated an overall quotation error rate of 25.4%, ²⁴ similar to that reported in a 2008 Cochrane review of 20%. ¹¹ The range of quotation errors reported in purely orthopaedic literature was between 7.6 and 41%.^10,25–27

A possible explanation for the low ProFHER quotation error rate was that the ProFHER study is a large and well-known trial with 93% of orthopaedic surgeons who treat PHFs reportedly aware of ProFHER's findings. ⁹ This means that authors of other related research and peer-reviewers for journals were likely to be familiar with the results and could correctly identify most quotation errors.

Ratio of major and minor errors

In our study minor errors outnumbered major errors. The previous literature reported similar numbers of major and minor errors^10,24,25,28 or even higher numbers of major errors.^26,29

Our findings could be explained either by the subjective nature of error classification or by differences in major and minor error definition. Mogull ¹² re-evaluated quotation error rates with standardised set of definitions in 15 studies and demonstrated that six of these studies had similar ratios of major to minor errors while the remaining nine studies had a higher number of major errors than minor errors.

Causes of quotation error

Numerous studies have sought to identify factors that contribute to quotation errors. The following do not correlate with error rate: journal impact factor,^10,25,26,29 number of authors, word count, references per article^25,26,28 or study design.^26,28

A possible cause of quotation errors was multi-referencing. We found 16/35 (46%) identified errors involved multiple references to support a statement. This was lower than the previously reported 65% of errors associated with multi-referencing. ¹⁰ The same study also found major errors were significantly more common when multi-referencing was used. Our low multi-reference rate may have contributed to the observed lower major error rate.

Currently, it is unclear what factors could impact on quotation error rates but these inaccuracies are detrimental to the quality of the medical literature as their presence undermines the reliability of published work if readers are unable to trust cited statements. It is imperative that authors ensure the accuracy of their work and it is their onus to ensure they have not misinterpreted referenced material, ²⁴ especially when citing and quoting high quality evidence and data, such as the ProFHER trial. A number of methods have been advised to help authors reduce the number of quotation errors. These include reading any referenced material in its entirety to ensure isolated statements are not taken out of context, using primary references in preference to indirect references and if possible reducing the number of references used. ²⁴

We recommend that if an error is identified, the reader should contact the corresponding author of the article to inform them of this. Quotation errors are usually inadvertent and so this offers the author the opportunity to correct their work and improve the reliability of the literature. If the authors take no action or if serious errors are encountered with the potential to lead to patient harm then the editor of the journal should be requested to issue a correction or retraction. An alternative to these conventional methods of reporting is to use social media and online resources such as pubpeer.com that allow post publication independent scrutiny of scientific articles. ³⁰

Study limitations

First, our study reported only quotation errors regarding the ProFHER trial and did not address citation errors. Second, the findings regarding the ProFHER trial cannot be generalised to the wider published literature. Further work is necessary to investigate practices that increase the risk of quotation error. For example, no study has examined whether the quotation error rates for highly cited and quoted studies differ from studies that have fewer citations.