Acute Physiology and Chronic Health Evaluation II Score Is a Better Predictor of Mortality than IBMP-10 in Patients with Ventilator-Associated Pneumonia

Abstract

Background:

The (Immunodeficiency, Blood pressure [<90 mm Hg], Multilobular intiltrates [chest x-ray], Platelets [<100×10⁹/L], hospitalization [<10 days] before the onset of ventilator-associated pneumonia [VAP]) IBMP-10 is a new scoring system proposed as an easy-to-use alternative to the Acute Physiology and Chronic Health Evaluation II (APACHE II) score for predicting mortality in patients with ventilator-associated pneumonia (VAP). The objective of this study was to determine the validity of the IBMP-10 score compared with APACHE II in predicting mortality for an independent population consisting predominantly of surgical and neurotrauma patients.

Methods:

The IBMP-10 and APACHE II scores on the day of VAP diagnosis were calculated, and areas under the receiver-operating characteristic curves (AUROCs) were compared to determine the tests' abilities to predict 14- and 28-day mortality.

Results:

A total of 168 patients meeting the radiologic and clinical criteria for VAP for a single hospitalization between 2004 and 2007 were included; 80% of these were from the surgical or neurotrauma intensive care unit. Overall mortality rates were 15% and 23% at 14 and 28 days, respectively. The AUROC for the IMBP-10 score for predicting 14-day mortality was 0.609 (p=0.084) compared with 0.648 (p=0.017) for the APACHE II score. Both IBMP-10 and APACHE II AUROCs for predicting 14-day mortality were lower than observed in the original score validation (0.808 and 0.743, respectively). The AUROCs for predicting 28-day mortality were 0.602 (p=0.056) and 0.705 (p<0.001) for IBMP10 and APACHE II, respectively.

Conclusions:

The IBMP-10 score was less reliable than the APACHE II score in predicting 14-day mortality in this independent population of VAP patients. This finding highlights the need for additional validation of new disease severity scoring systems in a study population independent of the population used to derive score criteria, as well as in more specific populations of critically ill patients.

Ventilator-associated pneumonia (VAP) is still a common infection among mechanically ventilated patients, occurring in as many as 27% of all intubated patients, even in the face of continued research efforts and improvements in both prevention and management of the infection [1,2]. Patients who develop VAP have a higher risk of mortality, a longer stay in the intensive care unit (ICU), and greater hospitalization cost [2]. A reliable scoring system that is convenient for clinical practice would be a valuable tool for predicting prognosis and improving clinical decisions regarding the course of therapy in patients who develop VAP, as well as for improving clinical trials [3]. The prognostic capabilities of a number of severity-of-illness scoring systems have been evaluated in VAP patients with little success, and a simplified severity classification system has yet to be developed specifically for VAP [4]. The Acute Physiology and Chronic Health Evaluation II (APACHE II), originally developed for predicting mortality in patients on admission to the ICU, historically has been used to predict mortality associated with VAP when calculated on the day of diagnosis [3 –7]. Even though this technique has been validated, it is difficult and cumbersome to use routinely in clinical practice.

The (Immunodeficiency, Blood pressure [<90 mm Hg], Multilobular intiltrates [chest x-ray], Platelets [<100×10⁹/L], hospitalization [<10 days] before the onset of ventilator-associated pneumonia [VAP]) IBMP-10 score is a simple five-point system developed recently for predicting mortality in VAP patients. One point is assigned for each of the following: immunosuppression (I), systolic blood pressure <90 mm Hg or mean arterial pressure ≤70 mm Hg (B), multilobar infiltration on chest radiograph obtained the day of VAP diagnosis (M), platelet count <100,000/mm³ (P), and >10 days of hospitalization prior to VAP diagnosis (10) [8]. Immunosuppression is defined as steroid use >10 mg of prednisolone or an equivalent for more than five days, use of an immunomodulator, active malignant disease at the time of VAP diagnosis or cancer diagnosed within one year preceding the current VAP episode, acquired immunodeficiency syndrome, or chemotherapy or radiotherapy within the 30 days prior to the diagnosis of VAP [8].

The IBMP-10 scoring system has been validated only in the population of patients used to derive the five criteria that best predicted 14-day mortality in developing the scoring system. Additionally, demographics regarding the type of ICU where the patient developed VAP were not reported. Therefore, the ability of this system to predict 14-day mortality in a population independent of those patients used initially or within special populations of critically ill patients has not been determined. The objective of this study was to validate the utility of this new scoring system in an independent population of critically ill patients with VAP, consisting primarily of surgical and neurotrauma patients.

Patients and Methods

Study Design

We conducted a secondary analysis of data gathered from a previous evaluation of patients with VAP at a single institution, Hartford Hospital in Hartford, CT. The original study was a prospective, observational evaluation of patients with VAP who were managed using an antibiotic clinical pathway compared with patients in a historical control group [9]. The study was approved by the Institutional Review Committee at Hartford Hospital, and the need for informed consent was waived. Baseline characteristics, all variables needed to calculate IBMP-10 and APACHE II scores on the day of VAP diagnosis, and patient outcomes at 14 days, 28 days, and hospital discharge were available from an electronic database and review of patient medical records.

The IBMP-10 scores were calculated according to the criteria outlined in the original study by Mirsaeidi et al., and APACHE II scores were calculated as previously described [6,8]. Scores were calculated using the worst value within 24 h before or after VAP diagnosis. The ability of the IBMP-10 score to predict mortality was compared with that of the APACHE II score. For comparison with IBMP-10 scores, APACHE II scores were divided into categorical ranges, as was done by Mirsaeidi et al.: 0=score <15 points; 1=score 15–19 points, 2=score 20–24 points; 3=score 25–29 points; 4=score ≥30 points [8]. In addition, we compared our results with those of the original IMBP-10 study, which included patients from the Improving Medicine through Pathway Assessment of Critical Therapy of Hospital-Acquired Pneumonia (IMPACT-HAP) database [8].

Patient Population

A total of 168 patients from three ICUs (medical, surgical, and neurotrauma) at Hartford Hospital, an 867-bed tertiary-care center, were included. All patients were hospitalized adults >17 years of age who acquired VAP between July 2004 and September 2007 [9]. Ventilator-associated pneumonia was diagnosed using clinical criteria similar to those employed in the original score development and validation [8]. Specifically, VAP was defined as a new or progressing infiltrate on chest radiograph in a patient who had been ventilated mechanically for more than 48 h or weaned from mechanical ventilation within the previous 48 h, and who met at least two of the following clinical criteria: Body temperature >38°C or <36°C without another recognized cause, white blood cell count >10,000 cells/mm³ or <5,000 cells/mm³, or a macroscopically purulent tracheal aspirate [1,9].

Statistical Analysis

Categorical variables were compared using either the chi-square test or the Fisher exact test where appropriate, and continuous variables were compared using either the Student t-test or the Mann-Whitney rank-sum test. All statistical tests were completed using SigmaStat Statistical Software Version 2 (SPSS Inc., Chicago, IL). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for 14-day, 28-day, all-cause in-hospital mortality were determined for both the APACHE II and the IMBP-10 scores; receiver operating characteristic (ROC) curves were constructed; and the area under the ROC curves was calculated and analyzed using MedCalc Version 11 (MedCalc Software, Mariakerke, Belgium). A p value <0.05 was considered statistically significant.

Results

A total of 168 patients were included; the sample size was similar to that of the initial score validation set (n=178) [7]. The mean age (± standard deviation) was 56.7±20.6 years, and 109 patients (65%) were male. There were 26 nonsurvivors at 14 days (15%), and 39 non-survivors at 28 days (23%). A 14-day mortality rate of 15% was observed during the initial score validation [8]. No significant differences in patient demographics or co-morbidities were observed between survivors and non-survivors at 14 or 28 days, with the exception that the non-survivors had a significantly higher APACHE II score on the day of VAP diagnosis (Table 1). Immunosuppression was uncommon in our patient population (n=34; 20%) and was slightly less frequent than in the population used in the initial IBMP-10 validation (32%). Eighty percent of the patients were from surgical and neurotrauma ICUs (Table 1). A large percentage of the current study population also had cerebrovascular disease (n=106; 63%). At least one causative organism was isolated from 144 patients (88%); a total of 182 baseline organisms were identified in these patients (Table 2).

Table 1.

Baseline Characteristics of Patients with Ventilator-Associated Pneumonia, Stratified by Mortality at 14 and 28 Days

		Survival at 14 days			Survival at 28 days
Characteristic	Total (n=168)	No (n=26)	Yes (n=142)	p value	No (n=39)	Yes (n=129)	p value
Age
Mean years±SD	56.7±20.6	60.7±18.1	56.0±21.0	0.389	63.6±17.0	54.7±21.2	0.032
Male (%)	109 (65)	19 (73)	90 (63)	0.466	28 (72)	81 (63)	0.400
Co-morbidities (%)
Diabetes mellitus	50 (30)	10 (38)	40 (28)	0.411	15 (38)	35 (27)	0.248
Cerebrovascular disease	106 (63)	17 (65)	89 (63)	0.966	28 (72)	78 (60)	0.273
Immunosuppression	31 (18)	4 (15)	27 (19)	0.870	9 (23)	22 (17)	0.539
Liver disease	15 (9)	5 (19)	10 (7)	0.103	7 (18)	8 (6)	0.053
Chronic renal disease	21 (13)	4 (15)	17 (12)	0.872	6 (15)	15 (12)	0.730
Respiratory disease	39 (23)	6 (23)	33 (23)	0.815	9 (23)	30 (23)	0.847
Malignant disease	21 (13)	4 (15)	17 (12)	0.872	6 (15)	15 (12)	0.730
Traumatic brain injury	37 (22)	10 (38)	27 (19)	0.052	13 (33)	24 (19)	0.085
Type of ICU (%)
MICU	34 (20)	6 (23)	28 (20)	0.899	15 (38)	19 (15)	0.003
SICU	70 (42)	11 (42)	59 (42)	0.885	14 (36)	56 (43)	0.517
NTICU	64 (38)	9 (35)	55 (38)	0.859	10 (26)	54 (42)	0.421
APACHE II, mean±SD
At ICU admission	19.2±8.6	22.3±10.0	18.6±8.2	0.113	22.9±10.0	18.1±7.8	0.011
At VAP diagnosis	25.5±6.3	28.7±7.6	25.0±5.8	0.004	29.1±6.7	24.5±5.7	<0.001
Late-onset VAP (%)	109 (65)	11 (42)	98 (69)	0.016	22 (56)	87 (67)	0.283
ICU days prior to VAP, mean±SD	7.9±6.5	5.6±3.9	8.3±6.8	0.025	6.1±3.8	8.5±7.1	0.072
Ventilator days prior to VAP, mean±SD	7.5±6.3	5.4±3.9	7.9±6.6	0.027	5.6±3.8	8.1±6.8	0.024

APACHE II=Acute Physiology and Chronic Health Evaluation II; Late-onset VAP=pneumonia developing greater than four days after initiation of mechanical ventilation; MICU=medical intensive care unit; NTICU=neurotrauma intensive care unit; SICU=surgical intensive care unit; SD=standard deviation.

Table 2.

Isolated Causative Organism in 168 Patients at Baseline, Stratified by 14-Day Mortality

Pathogen (% of total)	Total (%)
Methicillin-sensitive Staphylococcus aureus	27 (16)
Methicillin-resistant S. aureus	15 (9)
Pseudomonas aeruginosa	41 (24)
Acinetobacter baumannii	6 (4)
Stenotrophomonas maltophilia	4 (2)
Enterobacter spp.	20 (12)
Klebsiella spp.	24 (14)
Escherichia coli	11 (7)
Other	34 (20)

Mortality rates by IBMP-10 score are depicted in Figure 1. Overall, the mortality rates for the current study population were similar to those from the IMPACT-HAP database when the IBMP-10 and APACHE-II scores were ≤2 points but lower in the current study when severity scores were >2 points.

FIG. 1.

Mortality rates at 14 days for Acute Physiology and Chronic Health Evaluation II (APACHE II) and IBMP-10 (see text for definition) scores in 168 patients with ventilator-associated pneumonia. Categories for APACHE II Scores: 0=<15; 1=15–19, 2=20–24; 3=25–29; 4=≥30. Abbreviations: HH=Hartford Hospital; IMPACT-HAP=Improving Medicine through Pathway Assessment of Critical Therapy of Hospital-Acquired Pneumonia.

Patient numbers, sensitivity, specificity, PPV, and NPV for both scores and at various cut-off points are listed in Table 3. The NPV was relatively high, whereas the PPV was low for both scoring systems. The areas under the ROC curves (AUROCs) for predicting 14-day mortality with the IBMP-10 score (0.609; 95% confidence interval [CI] 0.5301–0.683; p=0.0844) as well as the APACHE II score (0.648; 95% CI, 0.571–0.720; p=0.0174) were lower than those observed in the initial score validation, 0.808 and 0.743, respectively (Table 4) [8].

Table 3.

IBMP-10 and (APACHE II) Score Characteristics for Predicting 14-day Mortality in Patients with Ventilator-Associated Pneumonia

Score	N	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
IBMP-10
≥0	168	100.0	0	15.5	–
≥1	161	96.2	4.2	15.5	85.7
≥2	124	84.6	28.2	17.7	90.9
≥3	53	46.2	71.1	22.6	87.8
≥4	11	11.5	94.4	27.3	85.4
≥5	1	0	99.3	0	84.4
APACHE II
≥5	168	100	0	16.7	–
≥10	167	100	0.7	15.6	100
≥15	165	100	2.1	15.8	100
≥20	140	88.5	17.6	16.4	89.3
≥25	92	69.2	47.9	19.6	89.5
≥30	41	46.2	79.6	29.3	89

APACHE=Acute Physiology and Chronic Health Evaluation II; NPV=negative predictive value; PPV=positive predictive value.

Table 4.

Area under the Receiver Operating Characteristic Curves (AUROC) for IBMP-10 and Acute Physiology and Chronic Health Evaluation II (APACHE II) Scores

	IBMP-10 score		APACHE II score
	AUROC (95% CI)	p value	AUROC (95% CI)	p value
All Patients
IMPACT-HAP database (n=178)
14-day mortality	0.808 (0.721–0.895)	<0.001	0.743 (0.628–0.857)	<0.001
HH VAP patients (n=168)
14-day mortality	0.609 (0.530–0.683)	0.084	0.648 (0.571–0.720)	0.017
28-day mortality	0.602 (0.524–0.677)	0.056	0.705 (0.630–0.773)	<0.001
All cause, in-hospital mortality	0.621 (0.543–0.695)	0.005	0.728 (0.654–0.794)	<0.001
Subgroups
Excluding traumatic brain injuries (n=131)
14-day mortality	0.636 (0.547–0.718)	0.085	0.634 (0.546–0.717)	0.089
28-day mortality	0.610 (0.521–0.694)	0.086	0.708 (0.622–0.784)	<0.001
All cause, in-hospital mortality	0.641 (0.552–0.723)	0.004	0.735 (0.651–0.808)	<0.001
Pre-pathway (n=74)
14-day mortality	0.678 (0.559–0.782)	0.145	0.459 (0.342–0.579)	0.691
28-day mortality	0.662 (0.543–0.768)	0.043	0.631 (0.511–0.741)	0.104
All cause, in hospital mortality	0.728 (0.612–0.825)	<0.001	0.689 (0.570–0.791)	0.004
Post-pathway (n=94)
14-day mortality	0.532 (0.426–0.635)	0.633	0.773 (0.676–0.853)	<0.001
28-day mortality	0.536 (0.430–0.639)	0.540	0.773 (0.675–0.853)	<0.001
All cause, in-hospital mortality	0.517 (0.412–0.621)	0.764	0.788 (0.692–0.866)	<0.001

CI=confidence interval; HH=Hartford Hospital; IMPACT-HAP=Improving Medicine through Pathway Assessment of Critical Therapy of Hospital-Acquired Pneumonia; VAP=ventilator-associated pneumonia.

ROC curves also were constructed and analyzed for 28-day mortality as well as several other subpopulations (Table 4). The results were similar when controlling for traumatic brain injury, which could carry an additional mortality risk. Because utilization of the antibiotic clinical pathway in our original study was shown to have an impact on the infection-related mortality rate, we also analyzed ROC curves for both scoring systems in patients prior to and after VAP clinical pathway implementation. The AUROCs for 14-day mortality were not statistically significant for either score in patients prior to pathway implementation, whereas the AUROC for the APACHE II score was statistically significant after pathway implementation. However, when looking at discrimination for predicting 28-day and all-cause mortality, the AUROCs for the IBMP-10 score were not only statistically significant, but also higher than those for the APACHE II score in the pre-pathway group only.

Lastly, rates of occurrence for each of the five components of the IBMP-10 score were compared for survivors and nonsurvivors at 14 and 28 days. Although all five score criteria were significantly more prevalent in non-survivors in the initial score validation, thrombocytopenia was the only criterion that was significantly more prevalent in non-survivors at 14 days in the current study (Table 5). Multi-lobar pneumonia was also observed at the same rate in non-survivors than survivors at 14 days (the difference was not statistically significant. The prevalences of immunosuppression and hypotension were similar in non-survivors and survivors at 14 days. Hospitalization for >10 days before the onset of VAP was not more frequent in survivors than in non-survivors (the difference was not statistically significant).

Table 5.

Differences in Occurrence Rates of IBMP-10 Score Criteria in Nonsurvivors and Survivors at 14 Days

IBMP-10 Criteria	Nonsurvivors (n=26)	Survivors (n=142)	p value
Immunosuppression (%)	4 (15)	27 (19)	0.870
Hypotension (SBP ≤90 mm Hg or MAP ≤70) mmHg), n (%)	21 (81)	117 (82)	0.936
Multilobar infiltration (%)	22 (85)	91 (64)	0.068
Platelet count
Mean/mm³±SD	162±135	229±132	0.018
<100,000/mm³ (%)	11 (42)	16 (11)	<0.001
Duration of hospitalization prior to VAP diagnosis
Mean days±SD	6.0±4.3	9.3±7.1	0.023
Duration >10 days(%)	4 (15)	37 (26)	0.357
Mean IBMP-10 score±SD	2.4±1.0	2.0±1.0	0.083

MAP=mean arterial blood pressure; SBP=systolic blood pressure; SD=standard deviation; VAP=ventilator-associated pneumonia; IMBP-10=see text for definition.

Discussion

The IBMP-10 score was recently proposed as an easier-to-use alternative to the APACHE II score for predicting mortality in patients with VAP, but before this study had been validated only in the study population used to derive the score criteria. The IBMP-10 score was not able to predict accurately 14-day mortality in our independent series of critically ill patients. Conversely, the APACHE II was a valid predictor of both 14- and 28-day mortality when calculated on the day of VAP diagnosis, similar to what has been demonstrated in previous studies [7].

In the current study, we attempted to reproduce the validation of the IBMP-10 as originally described by Mirsaeidi et al. [8] using an independent population of patients with VAP. Hence, we used the same clinical diagnostic criteria for VAP employed in the original study. We recognize that using clinical criteria alone is not the most accurate for diagnosing VAP, as alternative diagnoses could be responsible for patients' clinical findings. Despite using the same methodologies and a population of similar size (n=168 vs. 178) with the same 14-day mortality rate (15%), the IBMP-10 score could not be validated. Differences in patient characteristics between our population and the population used in the initial validation could account for the discordant results. Our population consisted predominately of patients from surgical and neurotrauma ICUs. However, the original validation study did not differentiate patients by ICU, nor were patient demographics described in detail, making it difficult to make a comparison. Another factor that could have contributed to our conflicting results is variations in the number of patients in each score category. However, without knowing the distributions in the initial validation populations, the role this played is difficult to assess.

It also is important to recognize the potential bias introduced when the IBMP-10 score was validated initially, as this procedure was conducted in the same population used to derive the five score criteria found to be predictive of mortality. In contrast, there was no difference in occurrence rates for four of these five criteria between survivors and nonsurvivors at 14 days in the current study. This observation stresses the need for the current validation, conducted in a population that did not contribute to the construction of the scoring system. Although validation in a large, multi-center population certainly is useful, single-center observations such as this one also are important in validating new scoring systems, as this is how the score likely will be used.

In our single-center validation, the IBMP-10 was not as reliable as APACHE II in predicting mortality in a population consisting largely of surgical and neurotrauma patients. Perhaps the score would have performed better in a population of medical ICU patients, where the mortality rate attributable to VAP could be lower. However, the current study did not have a sufficient population of medical ICU patients to allow this comparison.

The AUROCs from the current analysis of the IBMP-10 score's ability to predict 14- and 28-day mortality associated with VAP (0.609; p=0.084 and 0.602; p=0.056, respectively) were lower than those observed for other prognostic indices in previous studies [6,9]. In a study evaluating the prognostic capability of APACHE II, the Sequential Organ Failure Assessment (SOFA), and the Clinical Pulmonary Infection Score (CPIS) for VAP in 63 patients, the APACHE II and SOFA scores had excellent discriminatory ability, with AUROCs of 0.81 (p=0.001) and 0.71 (p=0.005), respectively, compared with 0.63 (p=0.069) for the CPIS [7]. The PIRO (predisposition, insult, response, organ dysfunction) score is another new prognostic system for patients with VAP [10]. In the initial validation of this score, PIRO was a better discriminator of ICU mortality than the APACHE II score in 441 patients, with AUROCs of 0.81 and 0.53, respectively. The inclusion of additional variables associated with ICU mortality in this scoring system compared with the IBMP-10 score may make this a better discriminator of disease severity. This scoring system also requires further validation prior to use in clinical practice.

In conclusion, the prognostic capability of the IBMP-10 score for determining the 14-day mortality risk in patients with VAP could not be validated in the current study population, comprised of predominantly surgical and neurotrauma patients. In this independent population, the IBMP-10 score was less reliable than the APACHE II score in predicting 14-day mortality, although its performance improved for predicting 28-day mortality. This finding highlights the need for additional validation of new prognostic scoring systems in a study population independent of the population used to derive the score criteria, in addition to validations in more specific populations of critically ill patients (i.e., medical or surgical) prior to implementation in clinical practice.

Footnotes

Acknowledgments and Disclosure

We thank Ilene Staff, Ph.D., for her assistance with statistical analyses. This study was undertaken with funds from the Center for Anti-Infective Research and Development, Hartford Hospital.

The authors have no financial conflicts to disclose.

References

American Thoracic Society Board of Directors and Infectious Diseases Society of America Guideline Committee. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-acquired pneumonia. Am J Resp Crit Care Med, 2005; 288–416.

Safdar

, Dezfulian

, Collard

et al. Clinical and economic consequences of ventilator-associated pneumonia: A systematic review. Crit Care Med, 2005; 33:2184–2193.

Napolitano

. Use of severity scoring and stratification factors in clinical trials of hospital-acquired and ventilator-associated pneumonia. Clin Infect Dis, 2010; 51:S67–S80.

Wiskirchen

, Kuti

, Nicolau

. The use of scoring systems to predict prognosis in patients with ventilator associated pneumonia. Clin Pulm Med, 2011; 18:181–185.

Knaus

, Zimmerman

, Wagner

et al. APACHE—Acute Physiology and Chronic Health Evaluation: A physiologically based classification system. Crit Care Med, 1981; 9:591–597.

Knaus

, Draper

, Wagner

et al. APACHE II: A severity of disease classification system. Crit Care Med, 1985; 13:818–829.

Gursel

, Demirtas

. Value of APACHE II, SOFA and CPIS scores in predicting prognosis in patients with ventilator associated pneumonia. Respiration, 2006; 73:503–508.

Mirsaeidi

, Peyrani

, Ramirez

et al. Predicting mortality in patients with ventilator-associated pneumonia: The APACHE II score versus the new IBMP-10 score. Clin Infect Dis, 2009; 49:72–77.

Nicasio

, Eagye

, Nicolau

et al. Pharmacodynamic-based clinical pathway for empiric antibiotic choice in patients with ventilator-associated pneumonia. J Crit Care, 2010; 25:69–77.

10.

Lisboa

, Diaz

, Sa-Borges

et al. The ventilator-associated pneumonia PIRO score: A tool for predicting ICU mortality and health-care resources use in ventilator-associated pneumonia. Chest, 2008; 134:1208–1216.