Abstract
Summary
Objective
The best practice tariff for hip and knee replacement in the English National Health Service (NHS) rewards providers based on improvements in patient-reported outcome measures (PROMs) collected before and after surgery. Providers only receive a bonus if at least 50% of their patients complete the preoperative questionnaire. We determined how many providers failed to meet this threshold prior to the policy introduction and assessed longitudinal stability of participation rates.
Design
Retrospective observational study using data from Hospital Episode Statistics and the national PROM programme from April 2009 to March 2012. We calculated participation rates based on either (a) all PROM records or (b) only those that could be linked to inpatient records; constructed confidence intervals around rates to account for sampling variation; applied precision weighting to allow for volume; and applied risk adjustment.
Setting
NHS hospitals and private providers in England.
Participants
NHS patients undergoing elective unilateral hip and knee replacement surgery.
Main outcome measures
Number of providers with participation rates statistically significantly below 50%.
Results
Crude rates identified many providers that failed to achieve the 50% threshold but there were substantially fewer after adjusting for uncertainty and precision. While important, risk adjustment required restricting the analysis to linked data. Year-on-year correlation between provider participation rates was moderate.
Conclusions
Participation rates have improved over time and only a small number of providers now fall below the threshold, but administering preoperative questionnaires remains problematic in some providers. We recommend that participation rates are based on linked data and take into account sampling variation.
Background
Publication of comparative information on patient health outcomes after surgery in order to facilitate quality improvement is becoming increasingly common.1,2 Some health systems have experimented with the use of such information in the design of reimbursement schemes; an approach known as pay-for-performance intended to sharpen the incentives to provide high quality care.3,4
The ‘best practice tariff’ (BPT) for primary hip and knee replacement in the English NHS is an example of a pay-for-performance scheme, emphasising the Department of Health’s ambition to establish patients’ views and self-reported outcomes as a central component of hospital quality assessment and regulation.5,6 Since April 2009, all providers of NHS-funded care have been required to collect patient-reported outcome measures (PROMs) for all patients undergoing unilateral hip and knee replacement (and also for varicose vein surgery and groin hernia repair). PROMs are structured questionnaires that allow patients to report their health status (or health-related quality of life) before and six months after surgery. By comparing these responses, changes in health can be identified and used to better understand differences in the systematic effect that individual hospitals have on improving their patients’ health. 7
The new BPT, implemented in April 2014, links bonus payments to the requirement that providers do not perform statistically significantly worse than the national average with respect to risk-adjusted improvements in patients’ health status. 8 The size of this bonus payment amounts to approximately 10% (550 GBP) of the nominal reimbursement price. However, in order to qualify for this bonus, providers must also satisfy a 50% participation rate criterion. The rationale for linking participation rates to the bonus is that the larger and more representative the evidence base, the greater the ability to identify systematic differences between providers.
The participation rate is calculated as a ratio of the number of completed preoperative PROM questionnaires relative to the number of eligible patients. Preoperative data may be missing because providers fail to ask patients to complete the questionnaire or patients decline to participate.
Providers can appeal if deemed not to qualify for the bonus. There may be three grounds for appeal over the participation rate. First, participation rates are subject to sampling uncertainty. For example, a participation rate of 49% may result from chance variation and not be statistically different from the 50% threshold. This can be assessed by constructing confidence intervals around the crude rates. Second, participation rates for providers with low volumes will be estimated imprecisely and random variation can lead to exceptionally high or low observed participation rates. For example, in the extreme case of only two eligible patients, the observed participation rates can only be 0, 50 or 100%, even though 80% (say) of patients participate nationally. 9 The BPT guidance does not impose a minimum volume threshold for assessing participation rates. Precision-weighted estimates allow for low volumes, yielding participation rates more likely to reflect the true underlying rate.9–12 Third, patients may differ systematically across providers and these differences may impact on the willingness of patients to complete the PROMs survey. 13 This can be accounted for by risk adjustment of the participation rates.
A further issue arises from the way in which participation rates are calculated. Only PROMs questionnaires that can be linked to the patient’s electronic medical record are included in the analysis of health outcomes. Arguably, participation rates should also be calculated using only linked data. But the BPT guidance does not distinguish between PROM responses that can be linked and those that cannot. The consequences of this merit exploration.
The aim of this paper was to explore empirically these issues and their implications for the English PROMs programme based on pre-policy data. We constructed two participation rates: (a) a BPT rate based on all PROM records, and (b) a linked rate based on only those PROM records that could be linked to inpatient records. For these two rates, we determined the number of providers that fell short of the 50% participation rate in three financial years (April 2009 to March 2012) and studied the consistency of participation rates over time. We constructed confidence intervals around crude rates, employed hierarchical modelling techniques to adjust for differences in the precision of the provider estimates and explored the impact of risk adjustment. We draw conclusions about the likely impact of the BPT participation rate threshold introduced from April 2014, discuss the most appropriate definition of the participation rate and provide recommendations about the statistical treatment of participation rates.
Methods
Data
Data on all elective admissions for patients, aged 18 or over, who underwent NHS-funded, unilateral hip or knee replacement between April 2009 and March 2012 were extracted from the Hospital Episode Statistics (HES). Identification was based on the primary procedure codes recorded in the first HES episode of the inpatient spell (OPCS 4.5; see guidance 8 for a full list of relevant procedure codes). These data were used to construct the denominator of the participation rate. HES was also the primary source of information on patient characteristics used for risk adjustment. 13
To construct the numerator we used the number of non-duplicate complete preoperative PROM questionnaires collected as part of the national PROM survey. All NHS and private providers of NHS-funded care are required to administer a preoperative PROMs questionnaire to all patients deemed fit for hip or knee replacement. Consenting patients complete the paper-based questionnaire shortly before the surgery, i.e. during the last outpatient appointment preceding the surgery or on the day of admission. Responsibility for data collection lies with the provider of care.
For the BPT, patients’ health status is assessed using the Oxford Hip Score (OHS) or Oxford Knee Score (OKS).14,15 The OHS/OKS questionnaires consist of 12 items pertaining to functioning and pain. For the PROM response to be considered ‘complete’ no more than two of the 12 OHS/OKS items must be missing.
Calculation of participation rates
The BPT guidance calculates the participation rate as ‘the number of pre-operative PROMs questionnaires completed, relative to the number of eligible HES spells’.
8
This includes all preoperative PROM questionnaires completed, irrespective of whether these could be linked to HES. The BPT participation rate is given by
An alternative is to calculate ‘linked participation rates’ which rely solely on those PROM records that can be linked to HES, i.e.
BPT participation rates are greater or equal to linked participation rates due to their more inclusive definition of the numerator.
Statistical analysis
The analysis aimed to determine how many providers do not achieve the 50% participation rate threshold. The BPT guidance does not mandate adjustment for sampling uncertainty, patient characteristics or the precision of the rates. We compared crude rates with three more refined measures. First, we calculated confidence intervals, based on Wilson’s approach, around crude rates to account for sampling uncertainty.16,17 Second, we estimated hierarchical logistic models and obtained precision-weighted estimates of the participation rates with associated Bayesian credible intervals (see Appendix 1). This is common in performance assessments and (a) recognises that the probability of missing data may be more similar within than across providers, and (b) ‘borrows strength’ from other providers to estimate more reliable participation rates for providers with low volumes of activity.9–12 Precision-weighted estimates were calculated as weighted averages of the crude participation rate and the population mean, where weights reflect the precision of the crude rate (i.e. sample size of the provider). Rates for small providers are shrunken towards the mean, whereas for large providers they are largely unaffected. Third, we calculated risk-adjusted, precision-weighted linked participation rates. BPT participation rates could not be adjusted for patient factors because such information is unavailable for non-linked data.
For each provider, we tested whether the true underlying participation rate was below 50% using one-sided hypothesis tests at the 95% confidence level. Data were analysed separately by financial year (1 April to 31 March of the following year) to examine changes over time. We calculated the correlation of provider participation rates across years using Spearman’s rank correlation coefficient and displayed this using scatter plots. Providers with >100% participation rate were excluded from the statistical analysis to facilitate modelling and visualisation.
Results
The scale of the problem
Figure 1 shows the participation rates at national level for hip and knee replacement surgery. The proportion of complete preoperative PROM questionnaires increased from 65% in 2009/10 to 75% in 2011/12 for hip replacement procedures. The proportion of knee replacement patients with complete preoperative PROM questionnaire was slightly higher (70% in 2009/10 to 83% in 2011/12). When only linked episodes were considered, these rates were substantially lower.
National participation rates for hip and knee replacement surgery in financial years 2009/10 to 2011/12.
Item non-response did not explain the observed patterns. Of those who responded to the preoperative PROM questionnaire, only approximately 1.0% of hip replacement patients and 1.1% of knee replacement answered fewer than 11 of the 12 questionnaire items.
Proportion of providers not meeting the participation rate threshold
Table 1 shows the number of providers that did not meet the 50% threshold in the financial years 2009/10 to 2011/12 under the four different measures. Five main findings emerged and these are drawn out for the hip replacement results. First, as expected, the number of providers that did not meet threshold was much lower when assessed using BPT rates instead of linked rates. Second, in line with the increase in the national participation rates (Figure 1), the number of providers failing to meet the threshold had been declining over time, from 74 (29.1%) providers in 2009/10 to 46 (15.4%) in 2011/12, taking the crude BPT rates. Third, the confidence intervals overlapped the threshold for a substantial number of providers deemed to fall below it. In 2011/12, after accounting for sampling variation, the number of providers below the threshold fell from 46 (15.4%) to 34 (11.4%) (reductions were even greater in 2009/10) (Figure 2). Fourth, precision weighting identified a handful of low volume providers which failed to meet the 50% threshold because their crude rates were imprecisely estimated (Figure 3). Fifth, risk adjustment had an effect on linked participation rates and, depending on the provider’s case-mix, increased or decreased the number of providers identified as failing to meet the threshold (estimated coefficients are in Appendix 1). Similar results were obtained for the knee replacement data.
Uncertainty around crude BPT provider participation rates in financial years 2009/10 to 2011/12 (hip replacement; only providers with <50% participation rate). Note: Providers with participation rates statistically lower than 50% are highlighted dark grey, whereas providers with statistically not significantly lower participation rates are highlighted light grey. Effect of precision-weighting in low- and high-volume hospitals (hip replacement, financial year 2011/12). Note: Dots represent providers. Light grey dots represent crude BPT participation rates; dark grey dots represent precision-weighted BPT participation rates. Number (%) of providers identified as failing the 50% participation rate threshold. Note: The total number of providers may differ across BPT and linked datasets because of issues of subcontracting. Precision weighting was based on statistical analysis of either (a) all providers (when analysing linked participation rates) or (b) only those providers with no higher than 100% participation rates (when analysing BPT participation rates). The percentage of providers failing the participation rate threshold is calculated as the proportion of identified providers over the total number of providers (i.e. in the case of BPT rates including providers with >100% rates). The numbers of providers used in the assessments of BPT participation rates are as follows: hip replacement – 202 (FY 2009/10), 237 (FY 2010/11), 246 (FY 2011/12); knee replacement – 189 (FY 2009/10), 205 (FY 2010/11), 218 (FY 2011/12).
Participation rates over time
Longitudinal comparison suggested that BPT participation rates were moderately correlated over time. Correlation coefficients were 0.68 (2009/10 to 2010/11), 0.54 (2010/11 to 2011/12) and 0.45 (2009/10 to 2011/12) for hip replacement and 0.62 (2009/10 to 2010/11), 0.64 (2010/11 to 2011/12) and 0.50 (2009/10 to 2011/12) for knee replacement. The longitudinal correlations were similar for linked participation rates. Providers falling below the 50% threshold in one year were likely to achieve low participation rates in the following year. Figure 4 illustrates this for the hip replacement data in years 2010/11 and 2011/12.
Correlation of precision-weighted BPT participation rates in 2010/11 and 2011/12 (hip replacement). Note: The solid line shows the estimated slope of regression of BPT participation rates in 2010/11 on rates in 2011/12, i.e. the estimated year-on-year correlation. The dashed line is the 45° line, i.e. perfect correlation. Dots represent providers. Those providers with participation rates statistically significantly lower than 50% in both years are highlighted black.
Discussion
Main findings
As with other patient self-reported surveys, PROMs are prone to different forms of missing data. This has the potential to bias assessments of comparative hospital performance due to small numbers and potentially unrepresentative case-mix. Foreseeing this in designing the BPT for hip and knee replacement, Monitor and NHS England have mandated that provider participation rates must be 50% or more in order to be considered for a bonus. 8
Participation rates, like any other performance metric, are likely to be influenced by random variation and this should be taken into account when assessing whether providers meet the 50% threshold. This can be achieved through one-sided confidence intervals and precision weighting as routinely applied in many performance measurement schemes.9–12 Our results emphasise the importance of such refinements. Accounting for sampling uncertainty is particularly important, substantially reducing the number of providers deemed to have failed to meet the threshold. Precision weighting reduces the number further by correcting rates for providers with low volumes. An alternative is to implement a minimum volume threshold similar to that used in the BPT (i.e. at least 30 cases) when analysing performance in relation to health improvements.
We found that participation rates had been improving over the three financial years from 2009/10 to 2011/12, but that complete preoperative PROMs were still missing for more than 15% of patients, and that linked records are not available for approximately 40% of patients. In line with this, after allowing for sampling variation and precision weighting, the number of providers identified as failing the BPT participation threshold declined over time from approximately 20% in the financial year 2009/10 to 9% in 2011/12 for hip replacement surgery, and 16% (2009/10) to 6% (2011/12) for knee replacement surgery. These improvements in participation rates seem to have applied approximately uniformly across all providers as suggested by high inter-temporal correlation in provider rates. Providers with low participation rates in one year had low participation rates in subsequent years.
A previous study has identified a range of patient characteristics associated with non-response. 13 Our results suggest that risk adjustment may matter for some providers. However, while the case for risk adjustment is generally undisputed in the context of health outcome measurement, it is less clear when analysing participation rates. By adjusting for case-mix one assumes implicitly that non-response is due to patients declining to participate, not providers failing to engage sufficiently with harder to reach patient groups. In contrast, unadjusted rates assign responsibility exclusively to the provider and do not allow for differences in patients’ willingness to participate associated with observed characteristics. Ultimately, it is a policy choice about where to assign responsibility.
Policy implications
We argue that participation rates should be based on only those data that can be linked to the patient’s administrative record. There are three reasons for this. First, it ensures consistency with the data used in the assessment of health outcomes. For example, the BPT outcome assessment excludes all patients undergoing revision surgery, but these patients can only be identified from procedure codes recorded in HES. Second, it allows risk adjustment of the participation rates if deemed appropriate. Third, if the aim of the BPT participation rate threshold is to ensure that outcome assessment is based on as much data as possible, then linkage is crucial and should be incentivised as well. To a large extent, linkage failure is due to incorrect values in the fields used for linkage (e.g. the patient’s NHS number) in the PROM survey. Responsibility for data quality rests with the provider, and the risk of failing to meet the participation threshold will incentivise improvements in coding behaviour. Note that the issue of subcontracting, which affects approximately 1% of all linked hip and knee replacement records, is also a coding issue and is therefore not directly relevant to the argument for or against use of linked data in calculating participation rates.
The large variation in participation rates and their stability over time implies substantial scope for providers to improve participation rates and that it is appropriate to condition bonus payments on participation rates. However, the declining number of providers that would have been affected by this rule raises questions about whether to maintain the threshold at the 50% level. If participation rates kept increasing after the end of our study period, the incentives set by the BPT to improve participation further may be rather weak.
Weaknesses
Our study has limitations. First, linkage of PROM and HES records was based on the linkage algorithm developed by the Health and Social Care Information Centre and improvements to this algorithm are ongoing. 8 Second, the number of eligible patients was based on HES records, which may be subject to miscoding and measurement error. However, the impact on the estimated participation rates is likely to be small. 8 Third, we conducted analysis of BPT participation rates using data on only those providers with rates between 0 and 100%. This had a small effect on the precision-weighted estimates, which were now calculated on the basis of a smaller inter-provider variance and a lower mean across all providers; arguably both are more realistic. Assessments based on crude rates with confidence intervals were unaltered by including subcontracted providers with participation rates above 100%. An alternative modelling strategy would have been to estimate hierarchical Poisson models with the numerator as dependent variable and the denominator as exposure term. Finally, we analysed data that predate the introduction of the BPT. If the BPT leads to changes in data collection then future participation rates may diverge from the reported historic trends.
Conclusions
The implementation of the BPT is inconsistent in the treatment of statistical uncertainty and risk factors, both of which are accounted for in the assessment of outcomes, but not in the assessment of participation rates. Providers deemed not to have met the 50% threshold for participation may have grounds to appeal the penalty.
We believe that penalties for poor participation rates are appropriate but these must be calculated correctly and should aim at increasing the number of data points available for outcome assessments. Furthermore, participation rates should be calculated on the basis of completed and linked questionnaires, as these are the relevant responses in the outcome assessment and providers can help ensure linkage through better coding. Given increasing participation rates in the first three years of the PROM programme, there may be scope for raising the threshold above 50% in order to maintain pressure on providers to continually improve data collection.