Smartphone “Selfies”—A reliable and accurate tool for measuring elbow range of motion

Abstract

Background

To determine if self-taken photographs (“selfies”), performed independently after instruction by video or illustrated handout, would be an accurate and reliable tool for capturing elbow range of motion in patients with elbow contractures.

Methods

Fifty patients presenting with elbow contractures participated in the study. After completion of the selfie, the senior author clinically measured flexion and extension with a goniometer. The angles from the photographs were measured and analyzed.

Results

The agreement between goniometer and “selfie” measurements correlated closely (R² = 0.98) and agreement was excellent in both extension and in flexion with intra-class correlation coefficients of 0.95 (95% CI 0.92 to 0.97) in extension with a mean difference of 2° (95% CI −3° to 7°), and 0.93 (95% CI 0.89 to 0.96) in flexion with a mean difference of 4° (95% CI 0° to 8°). Systematic errors were low in extension, 0° (95% CI, ±11°) and in flexion −3° (95% CI, ±10°). Six patients demonstrated ≥10° difference between clinical and selfie measurements. Ability to take a usable selfie was inversely correlated with age (R² = 0.97).

Discussion

Self-taken flexion–extension photographs are a reliable and accurate tool for measuring elbow range of motion. Errors in the selfie technique are well tolerated and appear to have a negligible effect upon measurements of motion. This important parameter of elbow function can therefore be obtained outside a normal clinic visit, thereby improving frequency of follow-up assessments (and minimizing loss to follow-up) necessary for quality control and research.

Keywords

elbow range of motion elbow contracture photographic measurements self-taken photographs correspondence follow-up

Introduction

The function of the elbow is to position the hand in space. Loss of elbow motion can severely impair the ability to carry out activities of daily living therefore affecting a patient’s quality of life.^1–4 Thus, range of motion (ROM) is one of the most important objective outcomes following elbow trauma or surgery. Obtaining an accurate measurement of elbow ROM is not always possible during clinic follow-up because elbow surgery is often performed at tertiary care centers, making it difficult for patients to return for such appointments. Using digital photography to capture elbow ROM has been shown to be an accurate and reliable means of capturing elbow ROM.^5–7 In our practice, we routinely teach patients who own smartphone how to capture a self-taken ROM photograph (“selfie”) using a wall mirror to show maximum elbow flexion and extension. Although this method has not been validated in patients with elbow contractures, Meislin et al.⁷ did show that healthy volunteers without elbow contractures could reliably use a smartphone and written or video instructions to take selfies of their elbow ROM.

Blonna et al.⁶ validated digital photography as being accurate and reliable for measuring both flexion and extension contractures of the elbow when compared with traditional goniometry. They also showed that instrument validity of the photographic method was less reliant on observer expertise when compared to clinical goniometry.^6,8,9 However, the photographs were taken using a highly consistent technique by an experienced examiner who also positioned the patient with their arm in the same orientation each time prior to taking the photograph. The extent to which the accuracy or reliability of the technique depended on the examiner is unknown. Therefore, we sought to answer the questions: Are self-taken ROM photographs with a smartphone an accurate and reliable method for measuring elbow flexion–extension, and will errors in technique significantly affect measurements?

We hypothesize that photographic “selfies” taken by patients with elbow contractures are a reliable and accurate means of measuring range of flexion–extension and errors in technique will have a negligible effect on measurements. The purpose of the present study is to examine the accuracy and reliability of smartphone photographs, self-taken after video or written instruction, for measuring elbow contractures.

Materials and methods

After IRB approval, 50 consecutive patients with elbow contractures and firm ROM endpoints on clinical examination were enrolled in this prospective randomized clinical trial. Patients were excluded if there was any barrier to communication (such as a language barrier) or if they were younger than 13 years old, unable to flex the ipsilateral shoulder to 90° and hold it in place long enough to take the photograph, or could not tolerate completing the study due to pain. In order to be representative of a general population, a wide spectrum of pathologies were examined; this included patients with anatomic variations such as bony abnormalities, obese habitus, forearm rotation contractures, and soft tissue deformities.

Patients were then randomized to receive either written and illustrated instructions on a form or an instructional video on YouTube (Google, Inc., San Bruno, CA). Following instruction, they were then given a smartphone (iPhone 5c, Apple, Inc., Cupertino, CA) and asked to take two “selfies” of their elbow, utilizing the instructed technique, in a mirror: one in full extension, and the other in full flexion. One investigator (MNS) was responsible for all study-related patient interactions.

Prior to taking photographs, patients completed a questionnaire assessing pain, elbow function and stability, and self-assessed ROM.

Photographic instructions

Instructions were given on a printed sheet and can be seen in Appendix 1 (available as supplementary material online). The YouTube video for the video instructions (“How to Capture Elbow Range of Motion”) can be found here: https://www.youtube.com/watch?v=Nel6qYjCHlM.

After each patient captured their self-taken ROM photographs, the pictures were reviewed by one of the investigators (MNS). Each patient was re-trained in the correct procedure if any variations from the instructions were seen. As a control, a final set of photographs was taken by the same investigator, using the protocol set forth by Blonna et al.,⁶ to compare the results of these photographs against a previously validated standard.

Standard clinical goniometry

Following the completion of all of the pictures, the senior author clinically measured flexion and extension with a long, blinded goniometer (the angle markings covered with foil to avoid potential bias in reporting measurements) once, first in full extension then once in full flexion. This protocol was used in previously published elbow ROM studies.^6,10 The anatomical landmarks used for goniometer placement included the center of the humeral head proximally, the lateral epicondyle in the middle, and ulnar styloid distally. The position of the arm during the measurement of the active ROM was standardized for all the patients as follows^6,10: the patient stood with shoulder forward flexed to 90° such that it was parallel with the floor, forearm supinated as much as possible, and the arm extended to the patient’s maximum. The senior surgeon lined up a goniometer, which had its numbers blinded to him, with the patient’s arm such that the center of rotation positioned over the lateral epicondyle. Without changing the angle of the goniometer, the angle was read and recorded by a second investigator. The patient held the same positioning and flexed their elbow to its maximum. The angle measurement was repeated as described above.

Digital photography-based ROM measurements

The photographs were displayed at random on a computer screen and angles were measured by one of the investigators with the Adobe Acrobat Pro DC angle measurement tool (Adobe Systems Incorporated, San Jose, CA) to simulate the digital angle measurement tool present on most institutional Picture Archiving and Communication System (PACS) image reading software. Anatomical landmarks used for digital angle measurement included the center of the humeral head proximally, the lateral epicondyle in the middle, and ulnar styloid distally. Angles were measured by clicking in order from the humeral head to the lateral epicondyle to the ulnar styloid. The angles were recorded for each patient. Comparison testing between the angle tool of our institutional PACS software and the Adobe Acrobat Pro DC angle measurement tool indicated that the two were comparable. The mean difference between the Adobe versus the PACS tool for 10 random angle measurements was −0.2° ± 0.2°. Photographic measurements were repeated two weeks later in order to calculate intraobserver reliability.

Photos from the video instruction and illustrated handout group were analyzed separately, to see how the two methods compared to the other. After that initial comparison, the groups were pooled to increase statistical power and to assess how selfies faired as a tool to assess elbow ROM.

The instrument validity was determined by comparing the photographic and clinical measurements. If patients were unable to complete photographs based on written or video instruction alone, the pictures completed after being taught by a member of the study team were included in the analysis.

Statistical analysis

Data were analyzed by GraphPad Prism 7.03 (GraphPad Software Inc., La Jolla, CA) using the Bland and Altman analysis.^11–13

Comparison of photographic ROM measurements and clinical values was calculated using intra-class correlation coefficient (ICC) estimates and their 95% confidence intervals based on a mean-rating (k = 2), consistency, two-way random-effects model.^14–16 An ICC of greater than 0.75 is considered to represent excellent reliability.^12,17 According to Scholtes et al.,¹⁸ the reliability of an ICC score depends on the variation in the study population. To avoid a misleading conclusion of good reliability, we have additionally included the mean difference and 95% confidence intervals (in degrees) as described by Keijsers et al.¹⁹

Results

There were 32 men and 18 women, with a mean age of 50 (range 23 to 83). The mean ages of the written and video instruction groups were 52 and 47, respectively, which was not significantly different (p > 0.05). Forty-five patients (90%) reported owning a smartphone. The etiology of the contractures was post-traumatic in 20, osteoarthritis in 21, inflammatory arthritis in 4, heterotopic ossification in 3, and other in 2. There were distortions of surface anatomy in seven patients (14%); these included obese habitus in four (8%), rheumatoid arthritis in one (2%), and olecranon bursitis in one (2%). One (2%) arm was entirely tattooed, obscuring anatomic landmarks. The average flexion–extension arc of motion measured by goniometer was from 34° (range −2° to 86°) to 124° (range 68° to 149°). Three patients (6%) were unable to supinate their forearms to greater than 60°.

Ease of instrument use

Forty out of 50 patients (80%) were able to complete the task on their own by following the written or video instructions (Figure 1). Of the 10 who did not, 2 had received instruction by video and 8 had received a printed form (P = 0.001). Eight out of those 10 patients were able to complete the task after being taught by a member of the team. Of the five people who did not own smartphones, three (60%) were unable to complete the task on their own.

Figure 1.

Graph representing ease of instructions based on patient’s ability to complete the photographs following receiving written or video instruction. Patients able to complete task after first attempt represent the patients who were able to produce a photograph of their reflection in the mirror after receiving instruction. Asterisk represents statistical significance of P<0.0001 between two groups using a Student’s t-test. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

Increasing age strongly correlated with decreasing ability to complete the task (R² = 0.89) and decreasing ability to complete the photographs without a deviation in form from the designated instruction (technical error) (R² = 0.97) (Figure 2). A nominal logistic fit model indicated that there was an interaction variable between age and the type of instruction given in both ability to complete photographs and complete photographs without any technical errors. With regard to being able to complete photographs, both age and type of instruction were statistically related to success.

Figure 2.

This graph represents a patient’s ability to complete photographs in relation to age. Completion of task included all patients who were able to complete a selfie either with perfect form or including some technical errors. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

Technical errors

Technical errors, for which photos were re-taken, occurred in 17/39 of the extension and 13/38 of the flexion photographs. Errors included shoulder not forward flexed to 90° (28 extension, 8 flexion), shoulder being covered by a shirt sleeve (7 extension, 5 flexion), incorrect positioning of the smartphone (2 extension, 10 flexion), dark photos (6 extension, 2 flexion), hand and/or shoulder cut out of photo (6 extension, 1 flexion), blurred photos (5 extension, 2 flexion), hand not being in line with the forearm (5 extension, 9 flexion), forearm not being fully supinated to the patient’s maximum (4 extension, 2 flexion), and rotation of the body such that the coronal plane was not orthogonal to the plane of the mirror and arm (2 extension, 3 flexion). Incorrect positioning of the smartphone was the only factor that caused a difference of ≥10° between photographic and clinical measurement, with an odds ratio of 8.2 (95% CI: 1.4 to 71.3).

Photos were reviewed for deviations in form from the given instructions. Examples of mistakes are shown in Figures 3 and 4. These errors did not have substantial impact on the differences between photographic and clinical measurements. The mean absolute differences in the clinical and photographic methods were 4° ± 3° in extension and 6° ± 5° in flexion prior to teaching the patients how to correct their technical errors and 5° ± 4° in extension and 5° ± 3° in flexion with correction (P ≥ 0.15).

Figure 3.

Example of technical errors that did affect accuracy. (a) Incorrect smartphone placement, limiting the degree to which the patient can flex the elbow. This caused an erroneous 28° disparity in flexion between clinical and photographic methods and (b) correct smartphone placement, with the two methods differing by only 3°. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

Figure 4.

Examples of technical errors that did not affect accuracy. Photographic examples of form variations including (a) hand not in the photograph, (b) a blurry and dark photo, (c) shoulder covered by a shirt sleeve, and (d) hand not maximally supinated. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

In two cases, the photographs demonstrated at least 10° less extension than the clinical measurements, and in both cases it was apparent that they were not fully extending their elbows. In the flexion group, three patients had greater than 10° more flexion measured goniometrically than on the photographs. In two of these, placement of the phone clearly blocked elbow flexion (Figure 3). The postcorrection data were used for instrument validity (below) for these two patients. One morbidly obese patient appeared to have more flexion on the photographs than clinically.

Instrument validity of self-taken ROM photographs

There were no significant differences between the goniometric measurements and those obtained following instruction from a printed form or by video in any group except for a statistically significant 3° difference in flexion in the video instruction group (P = 0.001).

The correlation between clinical and photographic measurements was extremely high (Figure 5). The ICC agreement was 0.95 (95% CI 0.92 to 0.97) in extension with a mean difference of 2° (95% CI −3° to 7°), and 0.93 (95% CI 0.89 to 0.96) in flexion with a mean difference of 4° (95% CI 0° to 8°).

Figure 5.

Systematic errors were low overall. Only extension measurements in two patients (4%) and flexion measurements in four patients (8%) were ≥10° different from the clinical measurements. Systematic error was measured to be 0° (95% CI, ± 11°) with an upper LOA of 11° in extension. In flexion, systematic error was −3° (95% CI, ± 10°) with an upper LOA of 13° (Figure 6).

Figure 6.

Bland–Altman plots showing differences in extension and flexion between clinical goniometry and photographic measurements taken by a medical student. The middle dotted line represents the average error and the top and bottom dotted lines represent the range of measurement error within the 95% confidence interval. Left: Photos captured after patients received written or video instruction and includes photos with form errors. Right: Photos captured after errors were corrected from being taught correct form from a member of the study team. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

Reliability of self-taken photographs

The intraobserver reliability of measuring the self-taken photographs was 0.98 for flexion and extension in a test–retest comparison. In extension, systematic error was 2° (95% CI, ± 6°) with an upper LOA of 8°. In flexion, systematic error was 1° (95% CI, ± 6°) with an upper LOA of 7° (Figure 7).

Figure 7.

Bland–Altman plots demonstrating intrarater reliability in extension and flexion. The middle dotted line represents the average error and the top and bottom dotted lines represent the range of measurement error within the 95% confidence interval. Source: Used with permission of Mayo Foundation for Medical Education and Research. All rights reserved.

Patient performance in relation to pain and function

Absolute differences in measurements obtained photographically and clinically did not correlate with pain (extension: R² = 0.0085; flexion: R² = 0.12) or function (extension: R² = 0.056; flexion: R² = 0.019) in the affected elbow.

Discussion

This study confirmed the hypothesis that photographic “selfies” taken by patients with elbow contractures are a reliable and accurate means of measuring range of flexion–extension. These photos can be taken by patients on their own when provided written or video instructions without direct involvement of a health care professional, filling the gap between missed follow-up and ensuring patients receive the best care possible. This study also determined that the bulk of technical errors in taking the photos did not affect the measurement obtained.

Instrument validity, which is the degree to which the instrument accurately measures what it intends to measure, was extremely high as indicated by ICC values of 0.95 (95% CI 0.92 to 0.97) in extension with a mean difference of 2° (95% CI −3° to 7°), and 0.93 (95% CI 0.89 to 0.96) in flexion with a mean difference of 4° (95% CI 0° to 8°). Blonna et al.⁶ had previously reported extremely high accuracy and reliability of ROM measurements from photographs that had been taken by an orthopedic surgeon during fellowship training. In that study, the fellow positioned each patient and took the photographs, minimizing the likelihood of technical errors. The results of the present study are similar to those reported by Blonna et al.⁶ The present study shows that the selfie photographic method can be performed by patients reliably and accurately.

Patients appeared to perform marginally better with video instructions over the illustrated handout. There was no significant difference between clinical and photographic measurement in selfies captured by the video instruction group. There was a 3° difference in the flexion measurement captured in the illustrated handout group which was statistically significant. Although this was statistically significant it would not be clinically important.⁹

A few studies have reported that a smartphone photograph taken by someone other than the subject is a valid method for documenting normal extension and flexion in patients without elbow pathology.^7,19 Meislin et al. demonstrated similar agreement between goniometric and photographic measurements to those in the present study. However, their study did not examine the accuracy or reliability of photo measurements for patients with contractures. Also, their recommended procedure required a second person to capture the photograph. Keijsers et al.¹⁹ reported that photography and movie based goniometry showed good validity in flexion and extension. They determined that the use of a smartphone application had poor to moderate reliability when compared to photography. The smartphone application, on which they reported, uses the phone’s accelerometer to measure the patient’s arm position and determine the ROM measurement. This differs from the method proposed in the current study. The present study permits patient independence and simply requires the patient to capture a “selfie.”

Our results demonstrate that patients are able to capture selfie photographs of their flexion–extension, and that these photos can be used to obtain accurate and reliable ROM measurements. ROM is an important objective parameter to assess after treating many elbow conditions or conducting research. Shukla et al.²⁰ showed that patient-reported outcomes following elbow and shoulder surgery can be obtained using a Summary Outcome Determination score, which has extremely high agreement between orthopedic providers and the patient. Therefore, follow-up after elbow surgery may be performed effectively by correspondence by having the patient send in a simple questionnaire, ROM photographs, and X-rays taken near home. Additionally, there is a major risk of losing patients to follow-up due to the expense and inconvenience of travel.^21–23 These results could serve to minimize this potential for loss of follow-up. With increasing emphasis on value-based and evidence-based treatments, telemedicine becomes more appealing likely due to reduced cost and reduced demands on time and resources.^24–28 Travel expenses for patients obtaining follow-up care have been reported to be as high as 4% of the total cost burden of undergoing an elective surgical procedure.²⁹

There are some limitations to this study. After initial instruction (by video or instructional handout) only 80% of the total patient population was able to take a photograph of their elbow ROM. This may not seem adequate for patient follow-up rates; however, completion rates were increased to 96% following in-person instruction with technical errors also being reduced.

Older patients had more difficulty obtaining a selfie, with success rate declining with age in a linear fashion. It may also be reasonable to assume that comprehension rate will continue to increase as smartphones become increasingly ubiquitous in daily living. In our own practice, it is not uncommon for older patients to overcome this challenge by sending in their ROM photos with the assistance of a younger, more “tech-savvy” family member. Of note, we observed in the present study that the success rate at completing usable selfies was 100% when analyzing patients under 40.

A variety of technical errors were made which underwent correction after personal instruction. These included incorrect body position, shoulder obstructed by a shirt sleeve, hand and/or shoulder cut out of photo, blurred photos, forearm not being supinated to patient’s maximum, and incorrect positioning of the smartphone. However, none of these significantly compromised intra-rater reliability or instrument validity, with the exception of incorrect positioning of the smartphone for flexion photographs. Specifically, placing the phone between the arm and forearm blocked elbow flexion. For this reason, we recommend rejecting flexion photographs taken with the phone between the arm and forearm. Additionally, one should perhaps be cautious of the accuracy of photographs of people with anatomic variations, including rheumatoid arthritis or morbid obesity due to alteration or obstruction of the bony landmarks. Furthermore, based on the senior author’s experience, we would recommend rejecting extension photos in which the patient is not supinated to their maximum. Maximal supination places the humerus in maximal external rotation which places the elbow in the correct plane of rotation for accurate extension. Our study did not show this to be a significant factor. However, given that there were only three people who made this error, we suspect it was underpowered to demonstrate this effect.

Another possible limitation involved the procedure for the surgeon’s goniometer measurements. The surgeon placed the blinded goniometer against the patient’s arm and then showed the un-blinded side to the study coordinator. On at least one occasion, the surgeon recognized that the goniometer angle moved during this maneuver, and this was confirmed by a repeat measurement that was different from the first one. Slippage of the goniometer after withdrawal from the patient’s arm may have occurred in additional patients. Having a single surgeon perform the clinical measurements may be seen as a limitation. A recent study by Keijsers et al.¹⁹ examined similar photographic ROM measurements about the elbow and used two blinded reviewers for clinical goniometry measurements. However, our study’s purpose was not to compare the results and/or reliability of multiple clinicians’ goniometry measurements, the main purpose was to compare the measurements with the self-taken photographs. Additionally, we used a method previously published by Blonna et al.,^6,10 where the senior author’s measurements were used as the “gold standard.” We believe that measuring the photographs at two separate time points, with two weeks separating the measurements, allowed us to answer our main research question of whether smartphone photographs are a reliable method for capturing ROM in patients with contractures.

Given that the measurements were taken in the same order every time, there is a possibility that there was carryover measurement error in the study. This was done because the experiment was conducted while the patients were waiting for the senior surgeon to review the patients following their initial assessment with a learner (orthopedic resident or fellow), in order to minimize the time burden to the patient. However, if carryover error did exist, one would expect it to be greater in both flexion and extension, in trying to demonstrate a greater arc of motion when being measured by their surgeon, which was not the case. Patients measured greater flexion and less extension by photograph compared to clinical measurements, which we propose would be more likely related to a consistent measurement or landmark discrepancy rather than carryover error.

A final limitation involved the presence of the study team while photographs were taken. While the team did not assist the patient, nor answer questions regarding the technique until the initial photos had been taken, the team’s presence could have altered performance by distracting the patients or making them feel self-conscious. It is possible that the photographs obtained in our study may not be entirely representative of those that would be obtained by mailing out instructions for patients to take them in their own homes.

The next step of this study may be to verify the reliability of this method by sending instructions to patients at home. Also, it may be useful to examine whether this is an accurate and reliable method to capture pronation–supination. Lastly, future studies comparing self-taken photographs to other methods of ROM collection could choose to do a crossover design between groups. This would allow the investigators to collect ROM measurements at numerous time points with numerous methods allowing the subjects to act as their own control.

Supplemental Material

Supplemental material for Smartphone “Selfies”—A reliable and accurate tool for measuring elbow range of motion

Supplemental Material for Smartphone “Selfies”—A reliable and accurate tool for measuring elbow range of motion by Maegan N Shields, Anthony M Vaichinger and Shawn W O’Driscoll in Shoulder & Elbow

Footnotes

Acknowledgements

This study has been presented as a poster at the 2018 American Academy of Orthopaedic Surgeons Annual Meeting in New Orleans, LA. It was also presented as a podium at the Annual meeting of the American Shoulder and Elbow Surgeons in Chicago, IL on 12 October 2018. The authors gratefully acknowledge the statistical analysis guidance provided by the Mayo Clinic Center for Clinical and Translational Science with its supported Grant Number UL1 TR002377 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. The authors of this project would also like to thank Mayo Clinic Media Support Services, Tammy S. Olson, Tiffany Keller-Lam, and James S. Fitzsimmons for their immense help and their contributions to this project.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The senior author and the research foundation with which he is affiliated do receive royalties from Acumed, LLC, Wright Medical, Inc. and Aircast, Inc.

Ethical Approval

Institutional Review Board ID: 13-003378—Validation of Self Digital Photography for Assessing Elbow Range of Motion.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the Mayo Clinic Foundation.

Informed Consent

Patient consent forms approved by Institutional Review Board ID: 13-003378. Informed consent received from all participants.

References

Morrey

Askew

K-N

, et al. A biomechanical study of normal elbow motion. J Bone Joint Surg 1981; 63-A: 872–877.

Keschner

Paksima

. The stiff elbow. Bull NYU Hosp Jt Dis 2007; 65: 24–28.

K-N

Morrey

Biomechanics of the elbow. In: Morrey

(eds). The elbow and its disorders., 5th ed. Philadelphia, PA: Elsevier, 2017, pp. 33–46.

Morrey BF. The elbow and its disorders. 4th ed. Philadelphia, PA: Saunders/Elsevier, 2009, p.1211.

Russo

Burn

Ismaily

, et al.

Is digital photography an accurate and precise method for measuring range of motion of the shoulder and elbow?

J Orthop Sci 2018; 23: 310–315.

Blonna

Zarkadas

Fitzsimmons

, et al. Validation of a photography-based goniometry method for measuring joint range of motion. J Shoulder Elbow Surg 2012; 21: 29–35.

Meislin

Wagner

Shin

. A comparison of elbow range of motion measurements: smartphone-based digital photography versus goniometric measurements. J Hand Surg Am 2016; 41: 510–515.

Cleffken

van Breukelen

van Mameren

, et al. Test-retest reproducibility of elbow goniometric measurements in a rigid double-blinded protocol: intervals for distinguishing between measurement error and clinical change. J Shoulder Elbow Surg 2007; 16: 788–794. e782.

Armstrong

MacDermid

Chinchalkar

, et al. Reliability of range-of-motion measurement in the elbow and forearm. J Shoulder Elbow Surg 1998; 7: 573–580.

10.

Blonna

Zarkadas

Fitzsimmons

, et al. Accuracy and inter-observer reliability of visual estimation compared to clinical goniometry of the elbow. Knee Surg Sports Traumatol Arthrosc 2012; 20: 1378–1385.

11.

Bland

Altman

. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet 1995; 346: 1085–1087.

12.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307–310.

13.

Bland

Altman

. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8: 135–160.

14.

Shrout

Fleiss

. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 420–428.

15.

Haas

. The reliability of reliability. J Manipulative Physiol Ther 1991; 14: 199–208.

16.

Haas

. Statistical methodology for reliability studies. J Manipulative Physiol Ther 1991; 14: 119–132.

17.

Fleiss

. Analysis of data from multiclinic trials. Control Clin Trials 1986; 7: 267–275.

18.

Scholtes

Terwee

Poolman

. What makes a measurement instrument valid and reliable? Injury 2011; 42: 236–240.

19.

Keijsers

Zwerus

van Lith

DRM

, et al. Validity and reliability of elbow range of motion measurements using digital photographs, movies, and a goniometry smartphone application. J Sports Med 2018; 2018: 7906875–7906875.

20.

Shukla

Vaichinger

Shields

, et al. Patient-physician agreement using summary outcome determination scores. Mayo Clin Proc 2018; 93: 32–39.

21.

Singh

Kwoh

Boudreau

, et al. Hospital volume and surgical outcomes after elective hip/knee arthroplasty: a risk adjusted analysis of a large regional database. Arthritis Rheum 2011; 63: 2531–2539.

22.

Lau

Perruccio

Gandhi

, et al. The role of surgeon volume on patient outcome in total knee arthroplasty: a systematic review of the literature. BMC Musculoskelet Disord 2012; 13: 250–250.

23.

Marlow

Barraclough

Collier

, et al. Centralization and the relationship between volume and outcome in knee arthroplasty procedures. ANZ J Surg 2010; 80: 234–241.

24.

Shortell

Rundall

Hsu

. Improving patient care by linking evidence-based medicine and evidence-based management. JAMA 2007; 298: 673–676.

25.

Porter

Teisberg

. Redefining competition in health care. Harv Bus Rev 2004; 82: 64–76. 136.

26.

Kaplan

Porter

. How to solve the cost crisis in health care. Harv Bus Rev 2011; 89: 46–52. 54, 56–61 passim.

27.

Ergina

Cook

Blazeby

, et al. Challenges in evaluating surgical innovation. Lancet 2009; 374: 1097–1104.

28.

Franklin

Budenholzer

. Implementing evidence-based health policy in Washington State. N Engl J Med 2009; 361: 1722–1725.

29.

Fjalestad

Hole

Jorgensen

, et al. Health and cost consequences of surgical versus conservative treatment for a comminuted proximal humeral fracture in elderly patients. Injury 2010; 41: 599–605.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.23 MB