Abstract
Background and purpose
The nomogram is an important component of modern medical decision-making, which calculates the probability of an event entirely based on individual characteristics. We aimed to develop and validate a nomogram for individualized prediction of the probability of unfavorable outcome in intravenous thrombolysis-treated stroke patients included in the large multicenter Safe Implementation of Thrombolysis in Stroke-International Stroke Thrombolysis Register.
Methods
All patients registered in the Safe Implementation of Thrombolysis in Stroke-International Stroke Thrombolysis Register by 179 Italian centers between May 2001 and March 2016 were originally included. The main outcome measure was three-month unfavorable outcome (modified Rankin Scale 3–6). Four non-categorical predictors of unfavorable outcome (baseline National Institutes of Health (NIH) Stroke Scale score: 0–25, age ≥18 years, pre-stroke modified Rankin Scale score: 0–2, and onset-to-treatment time: 0–270 min) were identified a-priori by three neurologists with expertise in the management of stroke. To generate the NIHSS STroke Scale score, Age, pre-stroke mRS score, onset-to-treatment Time (START), the pre-established predictors were entered into a logistic regression model. The discriminative performance of the model was assessed using the area under the receiver operating characteristic curve.
Results
A total of 15,862 patients with complete data for generating the START was randomly dichotomized into training (2/3, n = 10,574) and test (1/3, n = 5288) sets. The area under the receiver operating characteristic curve of START was 0.800 (95% confidence interval: 0.792–0.809) in the training set and 0.815 (95% confidence interval: 0.804–0.822) in the test set.
Conclusions
By using a limited number of non-categorical predictors, the START is the first nomogram developed and validated in a large Safe Implementation of Thrombolysis in Stroke-International Stroke Thrombolysis Register cohort, which reliably calculates the probability of unfavorable outcome in intravenous thrombolysis-treated stroke patients.
Introduction
Intravenous thrombolysis (IVT) is the most important therapeutic achievement over the past 20 years in the field of ischemic stroke management. Accurate estimate of the likelihood of treatment success or failure for individual patients is essential to provide a reasonable approach to patient management and help the patient and the family understanding the course of the disease.
Several prognostic models have been designed in the last few years with the aim of predicting outcome after IVT for acute ischemic stroke.1–6 The DRAGON (hyperDense cerebral artery sign or early infarct signs on admission computed tomography (CT) head scan, pre-stroke modified Rankin Scale (mRS) score, Age, Glucose level, Onset-to-treatment (OTT) time, National Institutes of Health (NIH) Stroke Scale (NIHSS) score) 1 and the Acute Stroke Registry and Analysis of Lausanne (ASTRAL), 4 after a first external validation in large multicenter cohorts, are two scores that have recently been validated to predict three-month unfavorable functional outcome (i.e. mRS score: 3–6) in the large multicenter Safe Implementation of Treatments in Stroke-International Stroke Thrombolysis Register (SITS-ISTR). 7 However, the performance of these scores for individualized prediction of outcome after IVT is limited by using dichotomization/categorization of the strongest predictors such as age, OTT time, NIHSS score, and pre-stroke mRS score.8,9
Several studies described a better performance of nomograms compared to risk grouping.10–14 By using a continuous score, a nomogram is a graphical statistical instrument that calculates the continuous probability of a particular outcome for an individual patient. In contrast to risk groups, a nomogram generates an individualized estimate of the predicted probability of an event of interest, which is entirely based on individual characteristics. The nomogram is an important component of modern medical decision-making. To date, nomogram has been used in an extensive array of applications including cancer, surgery, and other specialties.15–17
The present study aimed to develop and validate a nomogram by using a limited number of easily available non-categorical variables to predict three-month unfavorable functional outcome after IVT for individual stroke patients included in the large multicenter SITS-ISTR.
Material and methods
Study design, participants, and procedures
All patients registered in the SITS-ISTR by Italian centers between May 2001 and March 2016 were originally included. All centers registering patients are required to accept the rules of participating in SITS-ISTR, including consecutive registration of all stroke patients receiving IVT, irrespective of whether treatment was on label or off label. The scientific use of the data registered in the SITS-ISTR was approved by the local ethics committees.
In contrast to the model of prognostic scores including several categorized variables, we chosen a priori to develop the nomogram model including a limited number of non-categorical clinical variables that are readily available when IVT starts. Four putative strong predictors of unfavorable outcome (baseline NIHSS score, age, pre-stroke mRS score, and OTT time for IVT) were identified by three neurologists with clinical expertise in the management of stroke to compose the NIHSS STroke Scale score, Age, pre-stroke mRS score, onset-to-treatment Time (START) nomogram. We did not incorporate the baseline levels of glucose and systolic blood pressure (SBP) in our model because it is still questionable as to whether hyperglycemia and hypertension may be an epiphenomenon of clinical parameters, for example, severity of stroke, or glycemia and SBP lowering interventions, or pre-existing medication for diabetes mellitus or hypertension.18,19
We selected the range 0–25 points for NIHSS score because patients with very severe stroke as assessed clinically (e.g. NIHSS > 25) should not be treated with IVT according to the current summary of product characteristics (SPC) of Alteplase. 20 Age ≥ 18 years and OTT time ≤ 270 min were selected in agreement with the current guidelines and SPC of Alteplase.20,21 In addition, we selected the range 0–2 points for pre-stroke mRS score because all patients with pre-stroke mRS > 2 have unfavorable outcome at three months according to the classical definition for IVT (mRS score: 3–6). 7
We included all patients with complete data of age, pre-stroke mRS score, baseline NIHSS score, OTT time, and three-month mRS score. Patients treated with endovascular procedure after IVT were excluded from the analysis.
Outcome
The outcome measure was unfavorable functional outcome defined as mRS score 3–6 at three months. 7
Statistical analysis
The Italian SITS-ISTR cohort was randomly dichotomized into training and test sets by the statistical software STATA 13.0.1 (StataCorp, College Station, TX): 2/3 of the cohort was used to develop the prediction model, while the remaining 1/3 to perform the validation of the model.
Differences between the cohorts were explored using the Mann–Whitney U-test for continuous variables. Differences between proportions were assessed by Fisher’s exact test or the χ 2 test, when appropriate. Continuous variables were reported as median and interquartile range values. Proportions were calculated for categorical variables, dividing the number of events by the total number excluding missing/unknown cases.
To generate the START nomogram, the pre-established predictors were entered into a logistic regression model. Regression coefficients and odds ratios with two-sided 95% confidence intervals (CIs) for each of the variable included in the model were finally calculated. Collinearity of combinations of variables in the training set was evaluated by the variation inflation factors (<2 being considered non-significant) and condition index (<30 being considered non-significant).
Discrimination, i.e. the degree to which the START score enables to discriminate patients with favorable from those with unfavorable outcome, was assessed by calculation of the area under the receiver operating characteristic curve (AUC-ROC). Calibration, i.e. the agreement between predicted and actual unfavorable outcome, was assessed visually in the test cohort by using a calibration plot, in which the predicted probabilities were plotted against the frequency of the observed unfavorable outcome. A 45° line indicates perfect calibration when the predictive value of the model perfectly matches the patient’s actual risk. Spearman correlation coefficient was used to assess the correlation between the probability of unfavorable outcome according to START nomogram and mRS score in the test cohort. The correlation is weak, moderate, or strong for an absolute value of rs ≤ 0.39, 0.40–0.59, or ≥0.60, respectively.
To compare performance of START nomogram, we used DRAGON 1 and ASTRAL 4 scores (online-only Data Supplement) because these are the only two scores validated to predict three-month unfavorable functional outcome even in the SITS-ISTR database.
Results
Among the 25,808 patients registered in the SITS-ISTR cohort by 179 Italian centers, 550 (2.1%) patients were excluded for being treated with endovascular procedure after IVT. Additional patients were excluded for pre-stroke mRS score > 2 or unknown (n = 731; 2.8%), NIHSS score > 25 or unknown (n = 184; 0.7%), age < 18 years or unknown (n = 27; 0.1%), OTT time >270 min or unknown (n = 558; 2.2%), lack of three-month mRS score (n = 5761; 22.3%), and a combination of the previous criteria (n = 2135; 8.3%) (Supplemental Table 1).
Therefore, a final number of 15,862 patients with complete data for generating the nomogram were included in the study. The clinical characteristics of the patients in the training (n = 10,574) and test (n = 5288) cohorts are provided in the online-only Data Supplement (Supplemental Table 2). The proportion of patients with three-month unfavorable outcome was 39.7% in the training cohort and 38% in the test cohort.
The pre-established non-categorical predictors were entered into a logistic regression model (Supplemental Table 3) to generate the START nomogram for prediction of the probability of unfavorable functional outcome in the training cohort. No significant statistical collinearity was observed for any of the four pre-established variables. The nomogram was created by assigning a graphic preliminary score to each of the four predictors with a point range from 0 to 10, which was then summed to generate a total score, finally converted into an individual risk of unfavorable outcome after IVT. Details of the model are provided in the online-only Data Supplement. The START nomogram is shown in Figure 1 taking into account the approximation of all the variables that are graphed without decimal. An example of using the START nomogram is provided in the online-only Data Supplement (Supplemental Figure 1).
START nomogram for predicting the probability of three-month unfavorable outcome in patients receiving intravenous thrombolysis. NIHSS: NIH Stroke Scale; OTT: onset-to-treatment.
The AUC-ROC value of the START nomogram was 0.800 (95% CI: 0.792–0.809) in the training cohort (Supplemental Figure 2). The model was internally validated using 20,000 bootstrap samples to calculate the discrimination with accuracy of 0.808 (95% CI: 0.798–0.815). The model was validated in the test cohort with AUC-ROC value of 0.815 (95% CI: 0.804–0.822) (Supplemental Figure 3). The calibration plot for the nomogram model showed the adequate agreement between predictors calculated with the START nomogram and actual unfavorable outcomes in the test cohort (Supplemental Figure 4). To describe the better performance of START nomogram, two paradigmatic examples are provided in the online-only Data Supplement (Supplemental Table 4).
In the entire test set, 4784 (90.5%) patients had available data to calculate the DRAGON score and 4830 (91.3%) patients the ASTRAL. The AUC values of the DRAGON and ASTRAL scores were lower than the START nomogram (DRAGON: 0.776, 95% CI: 0.762–0.789, p < 0.001; ASTRAL: 0.804, 95% CI: 0.792–0.817, p < 0.001) (Supplemental Figure 5). The correlation between the probability of unfavorable outcome according to START nomogram and mRS score was strong (rs = 0.62) (Supplemental Figure 6) whereas was moderate for DRAGON and ASTRAL (rs = 0.52 and 0.55, respectively).
Additional analyses were performed in the subgroups of test cohort including patients with hyperdense cerebral artery sign (HCAS) on admission CT scan and intra-cranial occlusive thrombus on CT or magnetic resonance imaging (MRI) angiography. Among 1257 patients with HCAS, the proportion of patients with unfavorable outcome was 55.6%. The AUC-ROC value of the START nomogram was 0.828 (95% CI: 0.805–0.851); it was higher than the DRAGON and ASTRAL scores (DRAGON: 0.767, 95% CI: 0.741–0.794, p < 0.001; ASTRAL: 0.816, 95% CI: 0.792–0.839, p = 0.003). Among 878 patients with intra-cranial occlusive thrombus, the proportion patients with unfavorable outcome was 60%. The AUC-ROC value of the START nomogram was 0.835 (95% CI: 0.808–0.862); it was higher than the DRAGON and ASTRAL scores (DRAGON: 0.798, 95% CI: 0.767–0.828, p < 0.001; ASTRAL: 0.813, 95% CI: 0.782–-0.843, p < 0.001).
Among the patients who were excluded from database generating the nomogram for being treated with endovascular procedure after IVT, we selected 242 patients with complete data of three-month mRS score and available data to calculate the START nomogram, DRAGON, and ASTRAL scores. The proportion of patients with unfavorable outcome was 55.8%. The AUC value of the START nomogram was 0.766 (95% CI: 0.707–0.826); it was higher than the DRAGON score (0.685, 95% CI: 0.619–0.751, p = 0.002) and similar to ASTRAL score (0.744, 95% CI: 0.682–0.807; p = 0.122).
Discussion
We present here the START nomogram, which predicts the probability of three-month unfavorable outcome for individual stroke patients receiving IVT. The discriminative performance of the START model was good in the training and test cohorts.
The START nomogram may be a new and reliable mean for estimating the risk of three-month mRS score 3–6 in an individual patient. Differently from prognostic scores, our nomogram assigns a probability (from 0% to 100%) of having an unfavorable outcome. The prediction of high probability of unfavorable outcome should not exclude from IVT, the patients who are legitimately eligible for treatment. If anything, for patients at high risk of long-term disability, clinicians may proceed with caution when facing stroke individuals in the presence of exclusion conditions listed in the current SPC of Actilyse 20 or relative exclusion criteria according to current guidelines 21 in relation to possible increase of bleeding risk. The START may provide important information to clinicians when discussing prognosis with patients and their families. From the organizational and resource allocation perspective, the START may also facilitate the early identification of patients who are candidates for intensive rehabilitation or support rational decision-making for institutionalization of patients with very high probability of unfavorable outcome following IVT. In addition, the START may also be useful in stratifying patients in randomized controlled trials (RCTs) of new thrombolytic or neuroprotective drugs or new rehabilitation programs to increase the likelihood of balance between the different treatment groups.
The START nomogram is a graphical calculation instrument including only four non-categorical predictors easily available before IVT bypassing the use of the statistical expedient of the artificial cut-off to separate the patients into two or more different clusters. The DRAGON, 1 the iScore, 2 and the Totaled Health Risks in Vascular Events 3 scores are points-based risk scores based on cut-off values of discrete and continuous predictors, but risk categorizations are different. An important disadvantage of dichotomization is that it does not make use of within-category information. Everyone above or below the cut-point is treated as equal, yet the prognosis may vary considerably. Individuals close to but on opposite sides of the cut-point are characterized as being very different rather than very similar.
The ASTRAL 4 and the Stroke Prognostication using age and NIHSS-1005 are not scores based on cut-off values of non-categorical predictors. Instead, both scores are integer-based point scoring systems for each covariate; the overall score was calculated as the sum of the covariate weighted scores. Differently, the total score of the START is converted into a continuum of individual probability through a logarithmic formula. Another prognostic model developed to predict the functional outcome after IVT is the Stroke-Thrombolytic Predictive Instrument (TPI). 6 The Stroke-TPI was developed using a logistic regression equations on data from five major RCTs testing Alteplase in the 0 - to 6-h window in patients with NIHSS score > 4 and without pre-existing disability. Instead, the START nomogram was developed using a database of IVT-treated patients selected by criteria of age, NIHSS score, and OTT time for treatment in agreement with the current guidelines and SPC of Alteplase. Therefore, predictions may be more reliable for patients treated in routine practice.
Despite bridging of thrombectomy with IVT is a therapeutic option for a limited portion of stroke patients and rapid drip and ship transfer to larger centers is not currently feasible in small population centers, the number of endovascular procedures will be likely to increase in the near future; therefore, an easy prognostic tool may be useful. In the subgroups of patients who were potentially eligible to bridging therapy because of HCAS on admission CT scan or intra-cranial large vessel occlusion on CT or MRI angiography, the discriminative performance of the START model was good. The performance of the START nomogram in the subgroups of patients potentially eligible to bridging therapy could be a strategic tool for improving organizational models to increase access to bridging therapy for the growing number of patients, especially for those who received a high chance of having unfavorable outcome after IVT alone according to the START nomogram.
Finally, we tested the START nomogram in the Italian SITS-ISTR cohort of patients who received bridging of thrombectomy with IVT; its discriminative performance was discrete. Our model consisting of few strong clinical predictors for any stroke could also be a basic prognostic system for the population of patients eligible to bridging therapy. Unfortunately, known neurodiological non-categorical predictors such as the Alberta Stroke Program Early CT Score for continuous degree of extent of early infarct signs, core infarct volume, perfusion lesion volume, mismatch ratio, collateral circulation grades, or times of endovascular procedure are not available in the SITS-ISTR. Despite the combination of fulfilled imaging selection criteria for recent trials may greatly reduce the rate of candidates for thrombectomy, 22 future prospective studies will have to assess whether the extent of the START nomogram model with the combination of neuroradiological predictors may increase its discriminative performance in patients treated with endovascular procedure after IVT.
Our study has some limitations. First, it is based on a retrospective analysis of an ongoing database, so bearing the limitations of such study design. Despite our belief that the patient data in the entire Italian SITS-ISTR cohort are representative for clinical practice across a variety of demographics and stroke center types, for the risk score to be suitable for routine clinical practice, an external validation in a completely different cohort is warranted. Second, the number of missing data for three-month follow-up of the mRS score may have influenced the final outcome. This is a limitation of all studies that have used a large stroke thrombolysis database such as the SITS-ISTR. Third, the START nomogram cannot be applied to patients eligible for direct thrombectomy. However, primary thrombectomy is a therapeutic option within 6 h of stroke onset for only a few patients with large vessel occlusion and contraindications to IVT (i.e. approximately 15% of patients enrolled in the five randomized control trials) 23 and beyond 6 h for strictly selected patients according to the DAWN or DEFUSE 3 eligibility criteria. 24 Fourth, the exclusion of patients who received bridging therapy may limit the use of the START nomogram in the current practice. Despite our model was developed by using a cohort of patients treated with IVT alone, the predictive accuracy of the START nomogram resulted better than DRAGON and ASTRAL scores also in a small group of patients undergoing bridging of thrombectomy with IVT. Finally, the lack of advanced neuroimaging could limit the application of the nomogram to patients who received thrombectomy after IVT. However, to date, consensus does not exist on the best neuroimaging technique for the assessment of patient eligibility. 25 In the near future, the contribution of neuroradiological markers to predict the probability of three-month unfavorable outcome may be integrated into our model for patients eligible to bridging of thrombectomy with intravenous Alteplase or Tenecteplase. 25
Conclusions
By using a limited number of non-categorical clinical predictors, the START is the first nomogram developed and validated in a large SITS-ISTR cohort, which reliably calculates the probability of three-month unfavorable outcome in stroke patients who received IVT alone. The discriminative performance of the model was discrete in a small group of patients who underwent bridging therapy.
Supplemental Material
Supplemental material for The START nomogram for individualized prediction of the probability of unfavorable outcome after intravenous thrombolysis for stroke
Supplemental material for The START nomogram for individualized prediction of the probability of unfavorable outcome after intravenous thrombolysis for stroke by Manuel Cappellari, Gianni Turcato, Stefano Forlivesi, Fabio Bagante, Gianfranco Cervellin, Giuseppe Lippi, Bruno Bonetti, Paolo Bovi and Danilo Toni in International Journal of Stroke
Footnotes
Acknowledgments
We thank all the Italian centers registering patients in the Stroke-International Stroke Thrombolysis Register and their investigators for their participation. We also thank all patients who participated in Stroke-International Stroke Thrombolysis Register.
Authors’ contribution
MC, GT, SF, and PB contributed to concept and design. All authors contributed to acquisition, analysis, or interpretation of data. MC, GT, SF, FB, and DT drafted the manuscript. All authors critically revised the manuscript for important intellectual content. MC, GT, and FB conducted statistical analysis. GC, GL, BB, PB, and DT supervised the study.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
