Transparency and Public Reporting of Pediatric and Congenital Heart Surgery Outcomes in North America

Abstract

Health care is embarking on a new era of increased transparency. In January 2015, the Society of Thoracic Surgeons (STS) began to publicly report outcomes of pediatric and congenital cardiac surgery using the 2014 Society of Thoracic Surgeons Congenital Heart Surgery Database (STS CHSD) Mortality Risk Model. Because the 2014 STS CHSD Mortality Risk Model adjusts for procedural factors and patient-level factors, it is critical that centers are aware of the important impact of incomplete entry of data in the fields for patient-level factors. These factors are used to estimate expected mortality, and incomplete coding of these factors can lead to inaccurate assessment of case mix and estimation of expected mortality. In order to assure an accurate assessment of case mix and estimate of expected mortality, it is critical to assure accurate completion of the fields for patient factors, including preoperative factors. It is crucial to document variables such as whether the patient was preoperatively ventilated or had an important noncardiac congenital anatomic abnormality. The lack of entry of these variables will lead to an underestimation of expected mortality. The art and science of assessing outcomes of pediatric and congenital cardiac surgery continues to evolve. In the future, when models have been developed that encompass other outcomes in addition to mortality, pediatric and congenital cardiac surgical performance may be able to be assessed using a multidomain composite metric that incorporates both mortality and morbidity, adjusting for the operation performed and for patient-specific factors. It is our expectation that in the future, this information will also be publicly reported. In this era of increased transparency, the complete and accurate coding of both patient-level factors and procedure-level factors is critical.

Keywords

outcomes (includes mortality morbidity)database (all types)practice guidelines professional affairs

The Society of Thoracic Surgeons Congenital Heart Surgery Database

The art and science of assessing outcomes of pediatric and congenital cardiac surgery continues to evolve.^1,2 The Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database (STS CHSD), which was founded in 1994 to support quality improvement in cardiothoracic surgery, is the largest database in North America for pediatric and congenital cardiac surgery.³ From January 1998 through June 30, 2014, STS CHSD contains 331,672 operations reported from 120 of the estimated 125 pediatric and congenital heart surgery hospitals in the United States and 3 of 8 centers in Canada.³ With penetrance of over 95% in the United States, the data in STS CHSD are representative of pediatric and congenital heart surgery in the United States.

Coding in STS CHSD is accomplished by clinicians and ancillary support staff using the International Pediatric and Congenital Cardiac Code, which is used by most international databases of patients undergoing care for pediatric and congenital cardiac disease.^4,5 Definitions of all terms and codes used in the STS CHSD have been standardized and published.⁶ Evaluation of data quality includes intrinsic verification of data (eg, identification and correction of values that are missing or out of range and inconsistencies across fields), along with a formal process of random site audits at 10% of all participating centers each year, conducted by personnel from an independent quality assessment agency and pediatric cardiac surgeons.⁷ Audits include comparison with institutional source material (ie, the medical record) and have documented rates of completeness and accuracy of 100% and 98.7%, respectively, for the primary surgical procedure, and of 100% and 98.8%, respectively, for mortality status at hospital discharge.^3,7

Adjustment for Case Mix in the STS CHSD (Complexity Stratification, STAT Categories, and the STS CHSD Mortality Risk Model)

Assessment of variation in outcomes across pediatric and congenital cardiac surgical programs requires adjustment for differences in case mix across hospitals. Because of the large number of different types of pediatric and congenital cardiac operations (more than 200 individual procedure types, most often performed in various combinations), it is useful to stratify individual operations into groups or categories that are relatively homogeneous with respect to complexity or risk. This methodology, called complexity stratification, has been used by STS CHSD since 2002. The data are analyzed and reported within each stratum. STS CHSD has employed three methods of complexity stratification^8
–10:

The S ociety of T horacic Surgeons - European A ssociation for Cardio- T horacic Surgery Congenital Heart Surgery Mortality Categories (STAT Mortality Categories),

A ristotle B asic Complexity Levels (ABC Levels), and

R isk A djustment for C ongenital H eart S urgery-1 Categories (RACHS-1 Categories)

The STS CHSD initially used the ABC Levels and the RACHS-1 Categories to stratify procedures according to degree of complexity and risk. With the increasing availability of multi-institutional clinical data, the empirically based STAT Mortality Score and STAT Mortality Categories were introduced in STS CHSD in 2010. The STAT Mortality Categories^9,10 are a tool for risk stratification that was developed from an analysis of 77,294 operations entered into the EACTS CHSD (33,360 operations) and the STS CHSD (43,934 patients). Procedure-specific mortality rate estimates were calculated using a Bayesian model that was adjusted for small denominators. Operations were sorted by increasing risk and grouped into five categories (the STAT Mortality Categories) that were designed to be optimal with respect to minimizing within-category variation and maximizing between-category variation. STAT Category 1 is associated with the lowest risk of mortality and STAT Category 5 is associated with the highest risk of mortality. Defining case mix using the STAT Mortality Categories is helpful in assessing the outcomes achieved at a given institution relative to those achieved at the aggregate of all centers contributing data.

In 2014, STS CHSD enhanced its method of reporting of outcomes and began to report outcomes using the 2014 STS CHSD Mortality Risk Model. The 2014 STS CHSD Mortality Risk Model facilitates description of operative mortality adjusted for procedural factors and patient-level factors. The 2014 STS CHSD Mortality Risk Model is used to assess Operative Mortality and adjusts for the variables listed in Table 1.

Table 1.

The STS Congenital Heart Surgery Database Mortality Risk Model: List of Included Variables for Which the Model Adjusts.

Variable
Age-group
Primary procedure^a
Weight (neonates and infants)
Prior cardiothoracic operation
Any noncardiac congenital anatomic abnormality
Any chromosomal abnormality or syndrome
Prematurity (neonates and infants)
Preoperative factors
• Preoperative/preprocedural mechanical circulatory support (IABP, VAD, ECMO, or CPS)
• Shock, persistent at time of surgery
• Mechanical ventilation to treat cardiorespiratory failure
• Renal failure requiring dialysis and/or renal dysfunction
• Preoperative neurological deficit
• Any other preoperative factor^b

Abbreviations: CHSD, Congenital Heart Surgery Database; STS, Society of Thoracic Surgeons; STAT, The S ociety of T horacic Surgeons - European A ssociation for Cardio- T horacic Surgery Congenital Heart Surgery Mortality Categories; IABP, intra-aortic balloon pump; VAD, ventricular assist device; ECMO, extracorporeal membrane oxygenation; and CPS, mechanical cardiopulmonary support system.

^aThe model adjusts for each combination of primary procedure and age-group. Coefficients obtained via shrinkage estimation with STAT mortality category as an auxiliary variable.

^b“Any other preoperative factor” is defined as any of the other specified preoperative factors contained in the list of preoperative factors in the data collection form of the STS CHSD.

Operative mortality is defined in all STS databases as (1) all deaths, regardless of cause, occurring during the hospitalization in which the operation was performed, even if after 30 days (including patients transferred to other acute care facilities) and (2) all deaths, regardless of cause, occurring after discharge from the hospital but before the end of the 30th postoperative day.^11,12

Public Reporting With the STS CHSD

Health care is embarking on a new era of increased transparency.^13
–15 In January 2015, the STS began to publicly report outcomes of pediatric and congenital cardiac surgery using the 2014 STS CHSD Mortality Risk Model (http://www.sts.org/quality-research-patient-safety/sts-public-reporting-online).^16

–19

For all STS CHSD participants who consent to participate in voluntary public reporting, STS Public Reporting Online reports the following:

the overall operative mortality rate for each STS CHSD participant over a 4-year period for all ages and

the operative mortality rate for each STS CHSD participant over a 4-year period for all ages for each of the five STAT Mortality Categories.

The STS Public Reporting Online also includes the following data to provide a clear summary of the operative mortality of an STS CHSD participant:

Number/Eligible: This column of data presents a fraction with the numerator representing the number of observed patient deaths and the denominator representing the number of patients included in the calculation of operative mortality.

Observed: This column of data presents the observed rate of operative mortality as a percentage. This percentage is calculated by dividing the number of observed deaths by the total number of eligible patients included in the calculation.

Expected: This column of data presents the expected operative mortality rate as a percentage. The 2014 STS CHSD Mortality Risk Model is used to estimate the number of expected patient deaths when considering the actual case mix of a given STS CHSD participant (ie, the mix of patients treated as defined by all the variables listed in Table 1).

O/E (95% CI): This column of data presents the observed-to-expected (O/E) operative mortality ratio with 95% confidence intervals (CI). The O/E ratio is the number of observed deaths divided by the number of expected deaths. An O/E ratio >1 means that the STS CHSD participant had more deaths than expected based on the actual case mix of that STS CHSD participant. An O/E ratio <1 means that the STS CHSD participant had fewer deaths than expected based on the actual case mix of that STS CHSD participant. Small differences in the O/E ratio are usually not statistically significant, unless one is dealing with very large sample sizes, which is why the O/E ratio is reported along with 95% CIs. The 95% CIs provide a range of O/E ratios that could represent the underlying true O/E ratio of a given STS CHSD participant. The underlying true O/E ratio of a given STS CHSD participant is the ratio that would be observed hypothetically if the STS CHSD participant operated on a very large number of patients. If the CI is very wide, it means that the O/E ratio of the STS CHSD participant is a less exact estimate of the true underlying O/E ratio of the STS CHSD participant. In general, the width of the CI decreases as the number of patients included in the calculation increases.

Adjusted rate (95% CI): This column of data presents the adjusted mortality rate (AMR) with 95% CIs. The AMR is an estimate of what the rate of operative mortality of a given hospital would be if its observed performance was extrapolated to the overall aggregate case mix of STS CHSD. The 95% CIs provide a range of AMRs that could represent the underlying true adjusted operative mortality rate of a given STS CHSD participant. The underlying true adjusted operative mortality rate of a given STS CHSD participant is the rate that would be observed hypothetically if the STS CHSD participant operated on a very large number of patients. If the CI is very wide, this means that the adjusted operative mortality rate of the STS CHSD participant is a less exact estimate of the true underlying adjusted operative mortality rate of the STS CHSD participant. In general, the width of the CI decreases as the number of patients included in the calculation increases.

In the STS CHSD, the overall star rating of a given STS CHSD participant is based on their overall risk adjusted O/E operative morality ratio for all cardiovascular surgical patients, using the 2014 STS CHSD Mortality Risk Model.

Centers are classified into three categories based on their overall O/E risk-adjusted operative morality ratio:

One Star = higher than expected operative mortality (the 95% CI for their risk-adjusted O/E mortality ratio was entirely above the number 1)

Two Star = same as expected operative mortality (the 95% CI for their risk-adjusted O/E mortality ratio overlapped with the number 1)

Three Star = lower than expected operative mortality (the 95% CI for their risk-adjusted O/E mortality ratio was entirely below the number 1)

Examples of star ratings and 95%CIs are provided in the Table 2.

Table 2.

The STS Congenital Heart Surgery Database Mortality Risk Model: Star Ratings.

Overall Star Rating	Description	Examples of O/E (95% CI)
⋆	Lower and upper limits of confidence interval >1.0	1.2 (1.1-1.4)
⋆⋆	Confidence interval includes 1.0	0.8 (0.7-1.0)
⋆⋆⋆	Lower and upper limits of confidence interval <1.0	0.6 (0.2-0.9)

Abbreviations: CI, confidence interval; O/E, observed-to-expected; STS, Society of Thoracic Surgeons.

When publicly reporting outcomes for centers participating in the Adult Cardiac Surgery Database (ACSD), STS provides data of varying levels of granularity, ranging from point estimates with credible intervals for statistically sophisticated users to star ratings corresponding to average, above average, or below average (based on the work of Professor Judith Hibbard).²⁰ In 2010, STS began public reporting of cardiothoracic surgical performance with the public reporting of performance associated with coronary artery bypass grafting (CABG), using a multidomain composite measure that includes the following four domains:

risk-adjusted mortality,

risk-adjusted any or none morbidity (the occurrence of any [any or none] of five major complications of CABG surgery (stroke, renal failure, prolonged ventilation, reoperation, and infection),

the use of at least one internal mammary artery graft, and

the provision of all four (all or none) National Quality Forum-endorsed medications (preoperative beta blockade; and discharge beta blockade, lipid lowering agents such as statins, and aspirin).

For patients undergoing isolated CABG, STS publicly reports both numerical point estimates with credible intervals based on a Bayesian hierarchical model and a “star rating” category based on the true Bayesian probabilities (eg, 99% for isolated CABG) that the provider is an average, above-average, or below-average performer. In 2013, STS began publicly reporting outcomes after isolated aortic valve replacement (AVR) using a multidomain composite measure that includes the following two domains:

risk-adjusted mortality and

risk-adjusted any or none morbidity (the occurrence of any [any or none] of five major complications of cardiac surgery (stroke, renal failure, prolonged ventilation, reoperation, and infection).

In 2014, STS began publicly reporting outcomes after combined AVR and CABG using the same two domain composites used for isolated AVR. For patients undergoing isolated AVR and combined AVR and CABG, STS publicly reports both numerical point estimates with credible intervals based on a Bayesian hierarchical model and a “star rating” category based on the true Bayesian probabilities (eg, 97.5% for isolated AVR and for combined AVR and CABG) that the provider is an average, above-average, or below-average performer.

In 2015, STS began public reporting of outcomes associated with pediatric and congenital cardiac surgery. In preparation for the public reporting of outcomes associated with pediatric and congenital cardiac surgery, the STS CHSD Task Force and the STS Quality Measure Task Force examined the number of centers identified as programs with mortality that was higher than expected, the same as expected, and lower than expected, using four-year samples with 80%, 90%, 95%, and 99% CIs.¹⁸ Based on this analysis, the decision was made to use 95% CIs to identify outliers. It is important to note that in the public reporting of outcomes at centers participating in the STS CHSD, the one-star, two-star, and three-star designations specifically indicate higher than expected, same as expected, or lower than expected mortality, based on a model of risk adjustment that considers the operation performed and multiple patient-level factors. Similar to public reporting with STS ACSD, public reporting with STS CHSD provides data of varying levels of granularity, ranging from point estimates with CIs for statistically sophisticated users to star ratings. Coefficients for variables in the model are re-estimated every six months to ensure that the model remains well calibrated for its intended use in the STS-CHSD participant feedback report. The star-rating designations are determined using the 95% CIs of a center’s overall risk-adjusted O/E operative morality ratio for all cardiovascular surgical patients. The final appearance of these publicly reported data can be viewed at http://www.sts.org/quality-research-patient-safety/sts-public-reporting-online.

The Important Impact of Incomplete Coding

Because the 2014 STS CHSD Mortality Risk Model adjusts for procedural- and patient-level factors, it is critical that centers are aware of the important impact of incomplete entry of data in the fields for patient-level factors. These factors are used to estimate expected mortality, and incomplete coding of these factors can lead to inaccurate assessment of case mix and estimation of expected mortality.

In order to assure an accurate assessment of case mix and estimate of expected mortality, it is critical to assure accurate completion of the fields for patient factors, including preoperative factors. It is crucial to document variables such as whether the patient was preoperatively ventilated or had an important noncardiac congenital anatomic abnormality. The lack of entry of these variables will lead to an underestimation of expected mortality.

Strategies for the assessment of outcomes of pediatric and congenital cardiac surgery continues to evolve.^1,2 In the future, when models have been developed that encompass other outcomes in addition to mortality, pediatric and congenital cardiac surgical performance may be able to be assessed using a multidomain composite metric that incorporates both mortality and morbidity, adjusting for the operation performed and for patient-specific factors. It is our expectation that, in the future, this information will also be publicly reported. In this era of increased transparency, the complete and accurate coding of both patient-level and procedure-level factors is critical.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Abbreviations and Acronyms

References

Barach

Jacobs

Lipshultz

Laussen

. Pediatric and Congenital Cardiac Care - Volume 1: Outcomes Analysis. London, UK: Springer-Verlag London; 2014: 1–515.

Barach

Jacobs

Lipshultz

Laussen

. Pediatric and Congenital Cardiac Care - Volume 2: Quality Improvement and Patient Safety. London, UK: Springer-Verlag London; 2015: 1–456.

Jacobs

Mavroudis

Tchervenkov

Pasquali

SK.

Executive Summary: The Society of Thoracic Surgeons Congenital Heart Surgery Database—Twenty-second Harvest (January 1, 2011 – December 31, 2014). Durham, NC: The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical Center; Spring 2015 Harvest, 2015.

Franklin

RCG

Jacobs

Krogmann

. Nomenclature for congenital and paediatric cardiac disease: Historical perspectives and The International Pediatric and Congenital Cardiac Code. Cardiol Young. 2008;18(suppl 2):70–80.

The International Paediatric and Congenital Cardiac Code. Web site. http://www.ipccc.net/. Accessed September 5, 2015.

STS Congenital Heart Surgery Database Data Specifications. Web site. http://www.sts.org/sites/default/files/documents/CongenitalDataSpecsV3_22.pdf. Accessed September 5, 2015.

Clarke

Breen

Jacobs

. Verification of data in congenital cardiac surgery. Cardiol Young. 2008;18(suppl 2):177–187.

Jacobs

Lacour-Gayet

. Stratification of complexity improves the utility and accuracy of outcomes analysis in a Multi-Institutional Congenital Heart Surgery Database: Application of the Risk Adjustment in Congenital Heart Surgery (RACHS-1) and Aristotle Systems in the Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database. Pediatr Cardiol. 2009;30(8):1117–1130. doi:10.1007/s00246-009-9496-0.

O’Brien

Clarke

Jacobs

. An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg. 2009;138(5):1139–1153. PMID: 19837218.

10.

Jacobs

Maruszewski

. Initial application in the EACTS and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg. 2012;42(5):775–780. doi:10.1093/ejcts/ezs026. Epub 2012 Jun 14. PMID: 22700597.

11.

Jacobs

Mavroudis

Jacobs

. What is operative mortality? defining death in a surgical registry database: a report from the STS Congenital database task force and the joint EACTS-STS congenital database committee. Ann Thorac Surg. 2006;81(5):1937–1941.

12.

Overman

Jacobs

Prager

. Report from The Society of Thoracic Surgeons National Database Work Force: clarifying the definition of operative mortality. World J Pediatr Congenit Heart Surg. 2013;4(1):10–12. doi:10.1177/2150135112461924.

13.

Jacobs

Cerfolio

Sade

. The ethics of transparency: publication of cardiothoracic surgical outcomes in the lay press. Ann Thorac Surg. 2009;87(3):679–686. PMID: 19231369.

14.

Shahian

Edwards

Jacobs

. Public reporting of cardiac surgery performance: part 1-history, rationale, consequences. Ann Thorac Surg. 2011;92(3 suppl):s2–s11. PMID: 21867789.

15.

Shahian

Edwards

Jacobs

. Public reporting of cardiac surgery performance: part 2-implementation. Ann Thorac Surg. 2011;92(3 suppl):s12–s23. PMID: 21867788.

16.

Jacobs

O’Brien

Pasquali

. The importance of patient-specific preoperative factors: an analysis of the Society of Thoracic Surgeons Congenital Heart Surgery Database. Ann Thorac Surg. 2014;98(5):1653–1659. doi:10.1016/j.athoracsur.2014.07.029. PMID: 25262395.

17.

O’Brien

Jacobs

Pasquali

. The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Part 1-statistical methodology. Ann Thorac Surg. 2015;100(3):1054–1062. doi:10.1016/j.athoracsur.2015.07.014. PMID: 26245502.

18.

Jacobs

O’Brien

Pasquali

. The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Part 2-clinical application. Ann Thorac Surg. 2015;100(3):1063–1070. doi:10.1016/j.athoracsur.2015.07.011. PMID: 26245504.

19.

Pasquali

Jacobs

O’Brien

. impact of patient characteristics on hospital-level outcomes assessment in congenital heart surgery. Ann Thorac Surg. 2015;100(3):1071–1077. doi:10.1016/j.athoracsur.2015.05.101. PMID: 26245503.

20.

Hibbard

Peters

Slovic

Finucane

Tusler

. Making health care quality reports easier to use. Jt Comm J Qual Improv. 2001;27(11):591–604.