Relationship Between Risk-Adjustment Tools and the Pediatric Logistic Organ Dysfunction Score

Abstract

Background:

The Risk-Adjusted Classification for Congenital Heart Surgery (RACHS-1) method and Aristotle Basic Complexity (ABC) scores correlate with mortality. However, low mortality rates in congenital heart disease (CHD) make use of mortality as the primary outcome measure insufficient. Demonstrating correlation between risk-adjustment tools and the Pediatric Logistic Organ Dysfunction (PELOD) score might allow for risk-adjusted comparison of an outcome measure other than mortality.

Methods:

Data were obtained from the Virtual PICU Systems database. Patients with postoperative CHD between 2009 and 2010 were included. Correlation between RACHS-1 category and PELOD score and between ABC level and PELOD score was examined using Spearman rank correlation. Consistency of PELOD scores across institutions for given levels of case complexity was examined using Kruskal-Wallis nonparametric analysis of variance.

Results:

A total of 1,981 patient visits among 12 institutions met inclusion criteria. Positive correlations between PELOD score and RACHS-1 category (r _s = .353, P < .0001) as well as between PELOD score and ABC level (r _s = .328, P < .0001) were demonstrated. Variability in PELOD scores across individual centers for given levels of case complexity was observed (P < .04).

Conclusions:

Risk-Adjusted Classification for Congenital Heart Surgery categories and ABC levels correlate with postoperative organ dysfunction as measured by PELOD. However, the correlation was weak, potentially due to limitations of the PELOD score itself. Identification of a more accurate metric of morbidity for the congenital heart disease population is needed.

Keywords

congenital heart disease congenital heart surgery database outcomes

Introduction

Pediatric and congenital heart surgery has changed considerably since the first successful patent ductus arteriosus ligation in 1938. With advances in technology, surgical technique, and preoperative and postoperative care, mortality rates have improved dramatically. The desire to further improve outcomes has led to evaluation and comparison of surgeon and center performance. It has been recognized that risk adjustment for surgical case complexity is essential to this comparative process to avoid flawed interpretation of outcome data. This has led to the development of risk-adjustment tools such as the Risk-Adjusted Classification for Congenital Heart Surgery (RACHS-1) method and the Aristotle Complexity score.^1,2 The RACHS-1 method groups surgical procedures into six categories based on surgical complexity and a subjective estimate of the risk of mortality while in the Aristotle Basic Complexity (ABC) score, the scores are assigned to procedures based on potential morbidity, mortality, and anticipated technical difficulty and then divided into four levels of increasing complexity.² Studies have confirmed the association of RACHS-1 categories and ABC scores with mortality in a variety of clinical settings.^3

–7 However, with mortality rates for children undergoing congenital heart surgery hovering at 4%,⁸ it has become increasingly recognized that use of mortality alone as an outcome measure is no longer sufficient.^8
–10 Analyzing performance solely based on risk-adjusted mortality provides no means for evaluating performance and improving care for 96% of patients that survive resulting in a “gap” in the assessment of quality of care. Incorporation of a measure of postoperative morbidity, such as the Pediatric Logistic Organ Dysfunction (PELOD) score, into outcomes analysis could provide a way to assess quality of care for the majority of patients now surviving congenital heart surgery who fall into this “gap.”

The PELOD score takes into account organ dysfunction within six organ systems as well the severity of the dysfunction (Table 1).^11,12 It has been shown to correlate well with mortality rates as well as identify organ dysfunction even in patients in whom the mortality rate is low.¹² The PELOD score, used as an alternative to or in conjunction with mortality, might allow for outcome comparisons in the setting of limited sample sizes typical of the pediatric population. We hypothesize that there will be a positive correlation between postoperative organ dysfunction, as measured by PELOD score, and increasing levels of surgical case complexity, as defined by the RACHS-1 and ABC scores. Delineating the relationship between the PELOD score and these risk-adjustment tools developed for pediatric congenital heart surgery could set the stage for risk-adjusted comparison of postoperative morbidity, improving our ability to evaluate performance and improve care for patients surviving congenital heart surgery. As such, examining this relationship is an important step in enhancing our current approach to outcomes analysis and performance evaluation.

Table 1.

PELOD Scoring System.

Organ Dysfunction and Variable	Scoring System
Organ Dysfunction and Variable	0	1	10	20
Neurological^a
Glasgow coma score	12-15 and	7-11	4-6 or	3
Pupillary reactions	Both reactive	NA	Both fixed	NA
Cardiovascular^b
Heart rate, bpm
<12 years	≤195	NA	>195	NA
≥12 years	≤150	NA	>150	NA
	and		or
Systolic blood pressure, mm Hg
<1 month	>65	NA	35-65	<35
1 month-1 year	>75	NA	35-75	<35
1-12 years	>85	NA	45-85	<45
≥12 years	>95	NA	55-95	<55
Renal
Creatinine, µmol/L
<7 days	<140	NA	≥140	NA
7 days-1 year	<55	NA	≥55	NA
1-12 years	<100	NA	≥100	NA
≥12 years	<140	NA	≥140	NA
Respiratory
Pao₂, kPa/FiO₂ ratio	>9.3 and	NA	≤9.3 or	NA
Paco₂, kPa	≤11.7 and	NA	>11.7	NA
Mechanical ventilation	No ventilation	Ventilation	NA	NA
Hematological
White blood cell count, ×10⁹/L	≥4.5 and	1.5-4.4	<1.5	NA
Platelets, ×10⁹/L	≥35	<35	NA	NA
Hepatic
Aspartate transaminase, IU/L	<950 and	≥950 or	NA	NA
Prothrombin time (or INR)	<60 (<1.40)	≥60 (≥1.4)	NA	NA

PaO2: use arterial measurement only. PaO2/FiO2 ration cannot be assessed in patients with intracardiac shunts, is considered as normal in children with cyanotic heart disease. PaCO2 may be measured from arterial, capillary, or venous samples. Mechanical ventilation: use of mask ventilation is not counted as mechanical ventilation. ^aGlasgow coma score: use lowest value. If patient sedated, record estimated score before sedation. Pupillary reactions:non-reactive pupils must be > 3mm. ^bDo not assess heart rate and blood pressure during crying or iatrogenic agitation. Abrreviations: PaO2, arterial oxygen pressure; FiO2, fraction inspired oxygen; NA, not significant; PaCO2, arterial carbon dioxide pressure; PELOD, Pediatric Logistic Organ Dysfunction; IU, International Units; INR, international normalized ratio.

Materials and Methods

Study Population

All data were obtained from the Virtual PICU Systems (VPS) database following approval by the institutional review board and the VPS Research Committee. The VPS system is a clinical database dedicated to standardized data sharing and benchmarking among pediatric intensive care units (ICUs). Participating centers collect information on patient and hospital measures, diagnoses, interventions, discharge, organ donation, and pediatric severity of illness scores. As of January 1, 2012, the VPS database had over 470,000 patients, with a total of 117 ICUs from 99 participating sites. Patient visits entered into the VPS database between January 1, 2009, and September 30, 2010, were eligible for inclusion. Eligible patient visits were then filtered using selected criteria to capture the desired patient population. Specifically, visits to be included were limited to cardiac patients admitted to an ICU following a cardiac surgical procedure, and patients had to have RACHS-1 category, ABC score, and admission PELOD score recorded for inclusion. Risk-Adjusted Classification for Congenital Heart Surgery-1 categories and ABC scores are mapped directly to the index surgical procedure, which is assigned by VPS site data collectors who have been trained in proper designation of the index procedure, as defined by the Society of Thoracic Surgeons (STS). Nomenclature for cardiac procedures and diagnoses within the VPS database is consistent with the International Paediatric and Congenital Cardiac Code.

Data Analysis

Aristotle Basic Complexity scores were converted to ABC levels per published definitions.² RACHS-1 categories, ABC levels, and PELOD scores were treated as ordinal variables. Using Spearman rank correlation, data were examined for correlation between RACHS-1 category and admission PELOD score and between ABC level and admission PELOD score before and after stratifying for institutions’ estimated annual case volume. Projected annual surgical case volumes were calculated using the number of patients entered by an individual center and the number of quarters for which the center submitted data. Institutions were placed into groups based on the projected annual case volume with the following cutoffs: less than 100 cases per year, 100 to 250 cases per year, and greater than 250 cases per year. Using Kruskal-Wallis nonparametric analysis of variance, data were also examined to determine whether postoperative PELOD scores differed across individual centers for given levels of surgical complexity. P values <.05 were considered statistically significant.

Results

Over a 20-month period, 1,981 patient visits met inclusion criteria. These represented postoperative admissions to an ICU following congenital heart surgery at 12 institutions. The distribution of surgical patients among RACHS-1 categories and ABC levels are provided in Figure 1. Positive correlations were observed between RACHS-1 category and admission PELOD score (r = .353, P < .001) and between ABC level and admission PELOD score (r = .328, P < .001; Figure 2). The strength of the correlation between RACHS-1 category and admission PELOD score and between ABC level and admission PELOD score did not change after stratifying for centers’ projected annual surgical case volume (Table 2). Centers’ projected annual surgical case volumes are summarized in Figure 3.

Table 2.

Correlation of RACHS-1 Score and Aristotle Basic Complexity (ABC) Level With Pediatric Logistic Organ Dysfunction (PELOD) Score by Center Volume.^a

Projected Annual Center Volume	RACHS-1 and PELOD	ABC Level and PELOD
<100 cases per year	.319^b	.357^b
100-250 cases per year	.382^b	.357^b
>250 cases per year	.335^b	.279^b

Abbreviation: RACHS-1, Risk-Adjusted Classification for Congenital Heart Surgery.

^a P < .001 for all correlation coefficients.

^bSpearman rank correlation coefficients.

For given degrees of surgical case complexity, comparison of admission PELOD scores among centers demonstrated significant variability. There was consistency in PELOD score across centers for less complex procedures falling into ABC level 1 and RACHS-1 category 1. However, for more complex procedures, variability was observed. The relationship between PELOD scores and Aristotle levels at individual centers is shown in Figure 4. For ABC level 1, postoperative PELOD scores were consistent across centers. For all other ABC levels, organ dysfunction, as measured by postoperative PELOD score, varied across centers.

Although PELOD scores were noted to be higher in high-volume centers for RACHS-1 categories 2 and 3 (r = .141 and r = .116, respectively, and P values <.01) and for ABC levels 1 and 2 (r = .149 and r = .169, respectively, and P values <.03), the strength of these correlations was low with correlation coefficients <.2.

Discussion

Evaluation of quality of care and the comparison of outcomes are fundamental processes in providing optimal patient care and in evaluating performance. The RACHS-1 and ABC tools allow for risk-adjusted comparison of mortality. However, low mortality rates for children with pediatric and congenital heart disease (CHD) make the use of mortality as the sole outcome measure insufficient. Although RACHS-1 and ABC tools have been shown to be associated with mortality, their association with postoperative organ dysfunction has not been studied. Examining the relationship between established risk-adjustment tools and a measure of morbidity is important because demonstrating a positive correlation sets the stage for the comparison of outcomes for the majority of patients undergoing congenital heart surgery.

We studied the relationship between organ dysfunction, as measured by the PELOD score, and risk-adjustment tools in postoperative pediatric and CHD patients and demonstrated positive correlations. As expected, as surgical case complexity increased, so did the degree of postoperative organ dysfunction. We also found little variability in the strength of these correlations after adjusting for institutions’ estimated annual surgical case volume, allowing for risk-adjusted comparison of morbidity as measured by PELOD score among both high- and low-volume institutions. Although we did note higher PELOD scores within the group of higher volume centers for RACHS-1 categories 2 and 3 and for ABC levels 1 and 2, correlation coefficients were <.2, indicating that very little of the observed variation is explained by the chosen variable (case volume) and suggesting the presence and contribution of other confounding variables.

We also observed variability in the degree of postoperative organ dysfunction among individual centers after adjusting for surgical case complexity. These findings likely represent institutional practice pattern variation, and this variation may occur at a number of levels. For example, the degree of postoperative organ dysfunction could be a reflection of quality of the surgical repair and its effect on hemodynamics and systemic organ perfusion. It could also be related to intraoperative management strategies such as myocardial protection strategies and perfusion techniques or to postoperative care including management of fluids, ventilation, and inotropic medications. With just 12 institutions contributing data at the time of this study, there are insufficient data on center characteristics to allow for identification of the institutional practices that may be responsible for centers’ performance. Lack of adjustment for patient-related factors such as prematurity, low-birth weight, and noncardiac comorbidities that may impact postoperative organ dysfunction, and variability in scores from centers with small sample sizes limit the accuracy of center comparisons. With a larger sample size, potential variables associated with performance could be identified and hypotheses could be tested prospectively with adjustment for confounding variables in studies aimed at establishing causality between practice patterns and outcomes.

At the time of data collection for this study, the STS-European Association for Cardio-Thoracic Surgery Congenital Heart Surgery (EACTS) Mortality Scores and Categories (STAT Mortality Scores and Categories)¹³ were not recorded in the VPS database. Comparing the strength of correlation between the PELOD and the STS-EACTS Mortality Scores (STAT Mortality Scores) to those of RACHS-1 and Aristotle might be informative, as the STS-EACTS scores (STAT Mortality Scores) have certain advantages over RACHS-1 and Aristotle including development based on objective rather than subjective data, ability to capture a greater number of surgical procedures, and better discrimination of mortality.¹⁴ However, it is conceivable that the correlation between the PELOD score and the STAT Mortality Scores and Categories would still have been weak due to limitations of the PELOD score itself. Concerns about the validity of the PELOD score have been raised previously, including poor calibration over the range of risk¹⁵ and the lack of a continuous scale of risk assignment with gaps in data for patients with risk of mortality from 5% to 15% and from 40% to 80%.¹⁶ In addition, there are no reports of the score’s performance in a cardiac-specific population. Recently, however, STS morbidity scores and categories were developed using empirical data for the purpose of classifying congenital heart surgery procedures on the basis of their potential for morbidity.¹⁷ The score’s incorporation of complications likely to have considerable and lasting effects on patient health such as postoperative neurologic deficit persisting at discharge may make it a more accurate and relevant metric of morbidity than the potentially transient alterations in organ dysfunction measured by the PELOD score. Future evaluation of the ability of STS morbidity scores and categories to quantify morbidity and categorize case mix is needed as they may help fill the “gap” in quality of care assessment for patients surviving congenital heart surgery.

Finally, certain limitations are inherently introduced with use of a database. For example, the number of centers contributing data and the frequency with which data is submitted by participating centers cannot be controlled. Specifically, projected center volumes in this study may be underestimated. Also, while VPS site data collectors have been trained in proper assignment of the index surgical procedure per STS definition, it is possible that a procedure could be designated incorrectly.

Conclusions

Risk-Adjusted Classification for Congenital Heart Surgery-1 categories and ABC levels correlate with postoperative organ dysfunction as measured by PELOD. The severity of organ dysfunction among centers is variable and may represent institutional practice pattern variation. Before attempts are made to distinguish variables associated with postoperative morbidity, identification of a more accurate metric of morbidity for the CHD population than the PELOD score is needed. Preliminary work suggests that the STS morbidity score holds promise to serve as a meaningful measure of morbidity, and further assessment of the score is needed.

Footnotes

Acronyms and Abbreviations

Presented at the Pediatric Critical Care Colloquium in Santa Monica, USA; September 6-9, 2012.

Authors’ Note

All work on the project was performed at Children’s Hospital of Wisconsin/Medical College of Wisconsin in Milwaukee, WI.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the Laura P. and Leland K. Whittier Virtual PICU Fellowship.

References

Jenkins

Gauvreau

Newburger

Spray

Moller

Iezzoni

. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg. 2002;123(1):110–118.

Lacour-Gayet

Clarke

Jacobs

The Aristotle score: a complexity-adjusted method to evaluate surgical results. Eur J Cardiothorac Surg. 2004;25(6):911–924.

Larsen

Pedersen

Jacobsen

Johnsen

Hansen

Hjortdal

. The RACHS-1-1 risk categories reflect mortality and length of stay in a Danish population of children operated for congenital heart disease. Eur J Cardiothorac Surg. 2005;28(6):877–881.

Al-Radi

Harrell

Jr Caldarone

Case complexity scores in congenital heart surgery: a comparative study of the Aristotle basic complexity score and the risk adjustment in congenital heart surgery (RACHS-1-1) system. J Thorac Cardiovasc Surg. 2007;133(4):865–875.

Boethig

Jenkins

Hecker

Thies

Breymann

. The RACHS-1-1 risk categories reflect mortality and length of hospital stay in a large German pediatric cardiac surgery population. Eur J Cardiothorac Surg. 2004;26(1):12–17.

O'Brien

Jacobs

Clarke

Accuracy of the Aristotle basic complexity score for classifying the mortality and morbidity potential of congenital heart surgery operations. Ann Thorac Surg. 2007;84(6):2027–2037.

Kang

Tsang

Elliott

de Leval

Cole

. Does the Aristotle score predict outcome in congenital heart surgery? Eur J Cardiothorac Surg. 2006;29(6):986–988.

Jacobs

Mavroudis

Jacobs

What is operative mortality? Defining death in a surgical registry database: a report of the STS congenital database taskforce and the Joint EACTS-STS congenital database committee. Ann Thorac Surg. 2006;81(5):1937–1941.

Jacobs

Wernovsky

Elliott

. Analysis of outcomes for congenital cardiac disease: can we do better? Cardiol Young. 2007;17(suppl 2):145–158.

10.

Clarke

Lacour-Gayet

Jacobs

The assessment of complexity in congenital cardiac surgery based on objective data. Cardiol Young. 2008;18(suppl 2):169–176.

11.

Leteurtre

Martinot

Duhamel

Development of a pediatric multiple organ dysfunction score: use of two strategies. Med Decis Making. 1999;19(4):399–410.

12.

Leteurtre

Martinot

Duhamel

Validation of the paediatric logistic organ dysfunction (PELOD) score: prospective, observational, multicentre study. Lancet. 2003;362(9379):192–197.

13.

O'Brien

Clarke

Jacobs

. An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg. 2009;138:1139–53.

14.

Jacobs

Maruszewski

Initial application in the EACTS and STS congenital heart surgery databases of an empirically derived methodology of complexity adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg. 2012;42(5):775–780.

15.

Garcia

Eulmesekian

Sffogia

Limitation in paediatric logistic organ dysfunction score. Lancet. 2006;368(9542):1151.

16.

Tibby

. Does PELOD measure organ dysfunction…and is organ function a valid surrogate for death? Intensive Care Med. 2010;36(1):4–7.

17.

Jacobs

O’Brien

Jacobs

An empirically based tool for analyzing morbidity associated with operations for congenital heart disease. J Thorac Cardiovasc Surg. 2013;145(4):1046–1057.