Abstract
Background:
The American Orthopaedic Foot & Ankle Society (AOFAS) score is one of the most common and adapted outcome scales in hallux valgus surgery. However, AOFAS is predominantly physician based and not patient based. Although it may be straightforward to derive statistical significance, it may not equate to the true subjective benefit of the patient’s experience. There is a paucity of literature defining MCID for AOFAS in hallux valgus surgery although it could have a great impact on the accuracy of analyzing surgical outcomes. Hence, the primary aim of this study was to define the Minimal Clinically Important Difference (MCID) for the AOFAS score in these patients, and the secondary aim was to correlate patients’ demographics to the MCID.
Methods:
We conducted a retrospective cross-sectional study. A total of 446 patients were reviewed preoperatively and followed up for 2 years. An anchor question was asked 2 years postoperation: “How would you rate the overall results of your treatment for your foot and ankle condition?” (excellent, very good, good, fair, poor, terrible). The MCID was derived using 4 methods, 3 from an anchor-based approach and 1 from a distribution-based approach. Anchor-based approaches were (1) mean difference in 2-year AOFAS scores of patients who answered “good” versus “fair” based on the anchor question; (2) mean change of AOFAS score preoperatively and at 2-year follow-up in patients who answered good; (3) receiver operating characteristic (ROC) curves method, where the area under the curve (AUC) represented the likelihood that the scoring system would accurately discriminate these 2 groups of patients. The distribution-based approach used to calculate MCID was the effect size method. There were 405 (90.8%) females and 41 (9.2%) males. Mean age was 51.2 (standard deviation [SD] = 13) years, mean preoperative BMI was 24.2 (SD = 4.1).
Results:
Mean preoperative AOFAS score was 55.6 (SD = 16.8), with significant improvement to 85.7 (SD = 14.4) in 2 years (P value < .001). There were no statistical differences between demographics or preoperative AOFAS scores of patients with good versus fair satisfaction levels. At 2 years, patients who had good satisfaction had higher AOFAS scores than fair satisfaction (83.9 vs 78.1, P < .001) and higher mean change (30.2 vs 22.3, P = .015). Mean change in AOFAS score in patients with good satisfaction was 30.2 (SD = 19.8). Mean difference in good versus fair satisfaction was 7.9. Using ROC analysis, the cut-off point is 29.0, with an area under the curve (AUC) of 0.62. Effect size method derived an MCID of 8.4 with a moderate effect size of 0.5. Multiple linear regression demonstrated increasing age (β = −0.129, CI = −0.245, –0.013, P = .030) and higher preoperative AOFAS score (β = −0.874, CI = −0.644, –0.081, P < .001) to significantly decrease the amount of change in the AOFAS score.
Conclusion:
The MCID of AOFAS score in hallux valgus surgery was 7.9 to 30.2. The MCID can ensure clinical improvement from a patient’s perspective and also aid in interpreting results from clinical trials and other studies.
Level of Evidence:
Level III, retrospective comparative series.
Background
Hallux valgus is a common complex deformity involving varus deviation of the first metatarsal and valgus deviation of the first proximal phalanx. There are many surgical techniques involving soft tissue procedures, osteotomies, or fusion of the involved structures. To analyze clinical outcomes, the American Orthopaedic Foot & Ankle Society (AOFAS) Scale is one of the most common and adapted outcome scales in foot and ankle surgery.2,8,9,19 However, the AOFAS scale is predominantly physician-based and not patient based. Patients who are not satisfied with the outcomes of surgery may not always be reflected because of a lack of uniform outcome measures. While it may be straightforward to derive statistical significance, it may not equate to the true subjective benefit of the surgery patient’s experience.18,19
Therefore, it is crucial to use a more accurate measurement that can take into account subjective aspects of individual benefits to tell if patients had a significant improvement, as that will guide our management and monitoring of patients in routine care.1,11 One such approach is using the Minimal Clinically Important Difference (MCID), which is defined by Jaeschke et al as “the smallest difference which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management.” 11 The MCID is vital in accurately determining how these scores truly reflect treatment outcome as well as in the research field.4,8,13,19,20
There is a paucity of literature defining MCID for AOFAS scores in hallux valgus surgery although it could have a great impact on the accuracy of analyzing surgical outcomes of the surgery on a whole. 5 The primary aim of this study was to define the MCID for the AOFAS score in these patients and the secondary aim was to correlate patients’ demographics to the MCID.
Methods
This study was approved by the hospital ethics review board and written informed consent obtained from all recruited patients. We conducted a retrospective cross-sectional study. Between January 2007 and October 2013, 665 patients underwent hallux valgus surgery in our tertiary institution and were included in this study. Patients who underwent concomitant procedures on the lesser toes, midfoot, or hindfoot during the same anesthesia setting were excluded. Among the remaining patients, 20% were lost to follow-up. Subsequently, 446 patients formed the study group, who were all reviewed preoperatively and followed up for 2 years. Patients were identified by using diagnostic codes in the electronic hospital inpatient discharge summaries. A total of 446 patients underwent hallux valgus surgery and completed 2-year follow up. There were 405 (90.8%) female and 41 (9.2%) male patients. The mean age was 51.2 years. The mean preoperative BMI was 24.2. The mean length of hospital stay was 1.9 days. We included various techniques which were Scarf (n = 240, 53.8%), Chevron (n = 89, 19.9%), Lapidus (n = 58, 13.0%), Wilson (n = 19, 4.4%) and Mitchell (n = 40, 8.9%) osteotomies for the surgery. 143 (32.1%) patients underwent Akin procedure as well.
Baseline demographic characteristics (age, gender and BMI) were recorded. Patients were evaluated with the AOFAS scale at each visit. Patients’ demographics, surgical technique, and length of stay were recorded as well. In addition, one anchor question, adapted from the North American Spine Society Low Back Pain Instrument was asked 2 years postoperation: “How would you rate the overall results of your treatment for your foot and ankle condition?” (excellent, very good, good, fair, poor, terrible). This anchor question was chosen because satisfaction is a very important factor in determining a successful surgery. The anchor-based approach is a global transition item that compares the change in patient-reported outcome scores to an external criterion, in this case the anchor question. 4
The MCID was derived from 3 methods from the anchor-based approach: (1) mean difference in 2-year AOFAS scores of patients who answered good versus fair based on the anchor question; (2) mean change of AOFAS score preoperatively and at 2-year follow-up in patients who answered “good”; (3) receiver operating characteristic (ROC) curves were plotted, with the y-axis being sensitivity (proportion of patients who reported improvement in anchor question and had AOFAS scores above the MCID value) and x-axis being specificity (proportion of patients who did not report an improvement in anchor question and had AOFAS scores below the MCID value) to identify the score with the best sensitivity and specificity to discriminate between the “good” and “fair” patients.4,5 The area under the curve (AUC) of an ROC curve represented the likelihood that the scoring system would accurately discriminate these 2 groups of patients.4,14
Additionally, we used a distribution-based approach to calculate MCID. Effect size is a standardized measure of change derived by dividing difference in scores from preoperation to postoperation by the standard deviation (SD) of the preoperative scores.4,6,10 In other words, effect size is a measurement of difference in functional scores, taking into account the variability of patients’ improvement in scores postoperation. A larger effect size would mean a greater proportion of the treatment group being above the MCID compared to a control group. 16 Therefore, with a greater improvement in preoperative to postoperative scores, the effect size will be larger.5,17 It is commonly acknowledged that an effect size of 0.2 is small, 0.5 is moderate, and 0.8 is large.5,11 The MCID is then calculated by multiplying the SD of preoperative scores by 0.5.17,21
The association of demographic characteristics and the result of the anchor question were determined by Wilcoxon rank sum test and chi-square test, respectively, for continuous and categorical variables. One-way analysis of variance was applied to compare the means of preoperative AOFAS, 2-year AOFAS, and AOFAS score change among different satisfaction score groups by anchor question. We also performed a multiple linear regression to study the effect of age, gender, preoperative BMI, and preoperative AOFAS score on the mean of AOFAS score change. The ROC curves were determined by logistic regressions to discriminate between patients answering good and fair for anchor question by AOFAS score change. Statistical analyses were done with R 3.2.2.
Results
Mean preoperative AOFAS score was 55.6 with significant improvement to 85.7 in 2 years (P value <.001). There were no statistical differences between the demographics or preoperative AOFAS scores of patients with good versus fair satisfaction levels. At 2 years, patients who had good satisfaction had higher AOFAS scores than those who had fair satisfaction (83.9 vs 78.1, P < .001) and higher mean change (30.2 vs 22.3, P = .015) (Table 1).
Demographics of Patients Who Underwent Hallux Valgus Surgery, Sub-analyzed.
Abbreviations: AOFAS, American Orthopaedic Foot & Ankle Society; M, mean; SD, standard deviation.
Preoperatively, patients had no difference in their AOFAS scores (P = .416) (Table 2). For the anchor question at 2-year follow up, 93 patients had excellent satisfaction, with an AOFAS score of 92.9; 126 had very good satisfaction, with an AOFAS score of 90.0; 132 had good satisfaction, with an AOFAS score of 83.9; 51 had fair satisfaction, with an AOFAS score of 78.1; 27 had poor satisfaction, with an AOFAS score of 75.6; and 7 had terrible satisfaction, with an AOFAS score of 65.5. (P < .001) (Table 2)
Anchor Question Satisfaction Scores With Relation to AOFAS Scores.
Abbreviations: AOFAS, American Orthopaedic Foot & Ankle Society; M, mean; SD, standard deviation.
Satisfaction based on anchor question “How would you rate the overall results of your treatment for your foot and ankle condition?”
The MCID was calculated with the anchor and distribution methods as mentioned above. For anchor methods, the mean change in AOFAS score in patients with good satisfaction was 30.2. The mean difference in patients who had good versus fair satisfaction was 7.9. Using ROC analysis, the cut-off point was 29.0 with an area under the curve (AUC) of 0.62 (Table 3, Figure 1). For distribution method, the mean effect size was 1.8, corresponding to a large effect size. The effect size method derived an MCID of 8.4, with a moderate effect size of 0.5.
Minimally Clinical Important Difference via Anchor-Based Methods.
Abbreviations: AOFAS, American Orthopaedic Foot & Ankle Society; ROC, receiver operating characteristic.
Receiver operating characteristic curve calculation.
We subanalyzed the effect size by satisfaction scores to look at the responsiveness of the AOFAS score on different satisfaction levels. Effect size increased with satisfaction levels, from effect size of 0.2 to 1.1, 1.4, 1.9, 1.9, and 2.2, respectively, with terrible, poor, fair, good, very good, and excellent satisfaction levels.
Multiple linear regression demonstrated age and preoperative AOFAS score to significantly affect the change in AOFAS score over 2 years. Increasing age (β = −0.129, CI = −0.245, –0.013, P = .030) and higher preoperative AOFAS score (β = −0.874, CI = −0.644, –0.081, P < .001) significantly decreased the amount of change in AOFAS score (Table 4).
Multiple Linear Regression of Mean Change in AOFAS Score.
Abbreviations: AOFAS, American Orthopaedic Foot & Ankle Society; BMI, body mass index.
Therefore, we conducted a subanalysis of patients 50 years old and those 50 years and older. For patients less than 50 years old, the mean change was 33.9, mean difference was 7.6, and ROC cut-off point was 29. For patients 50 years and older, mean change was 22.7, mean difference was 7.0, and the ROC cut-off point was 29. There was no difference in mean change between patients age less than 50 and 50 and above (P = .086) (Table 3, Figures 2 and 3).
Receiver operating characteristic curve, age <50 years.
Receiver operating characteristic curve, age ≥50 years.
Discussion
Hallux valgus surgery provided excellent results among our patients, with a mean increase of 30.3 from preoperative to 2-year AOFAS score taking into account all patients and 93.3% of patients having fair to excellent satisfaction postsurgery. With AOFAS being one of the most frequently used outcome measures, it is vital to accurately determine clinical significance with relation to AOFAS and not just statistical significance.8,9,19 The MCID is an appropriate estimate for a minimum improvement needed for patients to have adequate improvement from their perspective. It can also be used in sample size calculation.13,17,22
Our study consisted of a large number of patients compared to other similar studies that derived specific scoring systems to interpret outcome measures and this increased the accuracy of the MCID.3,12 Also, we used anchor questions that had 6 distinct responses that would show a larger scale of responses, further increasing its precision. 21 Dawson et al 5 was the only paper in literature to our knowledge that derived MCID for AOFAS Score in hallux valgus surgery. The MCIDs reported were 24.75 (mean change) and 17.0 (ROC cut-off point). However, the anchor question used by Dawson et al focused on patients’ pain rather than satisfaction. 5
This study has shown the MCIDs of AOFAS in hallux valgus surgery to be from 7.9 to 30.2. (mean change = 30.2, ROC cut-off point = 29.0, effect size = 8.4, and mean difference = 7.9). We derived different MCIDs from different statistical methods, similar to other studies.12,14,15,21 The purpose of using multiple statistical analyses was to ensure consistency of MCID across all analyses, and it is recommended that we take into account all MCID estimates.
The mean change and ROC methods are most commonly used and reliable measures.11,21 The MCID, which represents the smallest difference between minimal change and no change over time that is clinically meaningful, was 30.2 in our study. The cut-off point in ROC curve, which determines the AOFAS score with equal sensitivity and specificity to discriminate between minimal change and no change, was 29.0 in our study. 4 The AUC for the ROC curve, which is interpreted as the ability to discriminate between minimal change and no change, was 0.62. This corresponds to the ROC curve having discriminatory accuracy, though poor. 14 These 2 methods derived an estimate of 29 to 30 points, which is a clinically useful estimate for MCID in AOFAS score in hallux valgus surgery.
Overall, 56.4% of patients achieved the MCID (increase in AOFAS ≥29). Among them, 94.1% had fair to excellent satisfaction. Among patients who did not achieve MCID, 85.8% had fair to excellent satisfaction. The proportion of patients with fair to excellent satisfaction achieving MCID was higher than those who did not achieve MCID (94.1% vs 85.8%, P = .007). Therefore, there was satisfactory predictive value with the MCID of 29 to 30 points as demonstrated.
Anchor-based methods relate quality-of-life instrument scores to an external marker of clinical change whereas distribution-based methods relate magnitude of score changes to a certain measure of variability, such as standard error. 3 There are studies that favor anchor-based methods because distribution-based methods do not provide direct information about the MCID and they assess minimal detectable change rather than MCID.17,21 There is less evidence to prove that the minimal detectable change is important to the patient. Between the different anchor-based methods, the ROC cut-off point method was suggested to be better as it aimed to minimize misclassification of patients who had significantly improved and patients who were unchanged. 21 Therefore, among all the different statistical methods used, we recommend the ROC cut-off point to be more reliable.
The effect size was 1.8 in our study, which corresponds to a large effect size. With subanalysis of satisfaction levels, effect size was still large (≥0.8) for all satisfaction groups except those who answered terrible satisfaction. However, there were only 7 patients in that category. Because effect size is an indicator of responsiveness, AOFAS was shown in our study to be a responsive tool for patients’ satisfaction. 7
Our study also showed that only age and preoperative AOFAS scores significantly affected the mean change of AOFAS scores. Older age decreased the magnitude of change in AOFAS scores. This may be due to older patients having less potential for improvement given a poorer functional status or mobility preoperatively. Higher preoperative AOFAS scores also decreased the magnitude of change in AOFAS scores, because there is less “room for improvement” with a higher baseline score. Therefore, we should be aware that a smaller increase in AOFAS score is needed for older patients or patients with higher baseline function to achieve significant benefit from hallux valgus surgery.
We acknowledge the limitations of our study. First, the AOFAS scores at various time points of follow-up could be used to report the MCID at 6 months or 1 year after hallux valgus surgery. However, too early follow-up may not accurately portray the full functional status of patients as not enough time is given for patients to fully recover from surgery. Second, different anchor questions and study population characteristics (age, gender, follow-up duration, preoperative scores) may give different MCID values. In this study, our institution’s practice of routinely admitting patients overnight for observation after anesthesia may differ from other institutions. This may give rise to different MCID values needed to assess clinically important improvement. This is a well-known limitation with MCID studies, and further research has to be conducted to develop universally recognized definitions for “important” or “minimum.” 3 Third, there has been some debate regarding the validity of AOFAS score in hallux valgus surgery, though it is popular among clinicians in assessing surgical outcomes.8,9 Last, each anchor-based or distribution-based method has its own limitation in accuracy and there has yet to be an ideal statistical method to calculate MCID. 22 Therefore, a constellation of methods was applied to increase the accuracy in calculation. This study reported the MCIDs for AOFAS in hallux valgus surgery to have a range of 7.9 to 30.2, which leads to the recommendation of another adjunct scoring system to be used in conjunction with AOFAS to better assess hallux valgus surgical outcomes. This will aid in better assessing the true subjective benefit of surgery that the patient experiences.
Conclusion
Our study presented the minimal clinically important difference (MCID) of 7.9 to 30.2 in AOFAS score following hallux valgus surgery, depending on the statistical method. This implies that an improvement of 7.9-30.2 points in the AOFAS score indicates a clinically important improvement in patients’ functional outcome from hallux valgus surgery and is not due to measurement error. The MCID can ensure clinical improvement from a patient’s perspective and also aid in interpreting results from clinical trials and other studies.
Footnotes
Acknowledgements
We would like to thank Ms Cheong Hwei Chi for her assistance.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.