Prognostic accuracy of inflammatory markers in predicting risk of ICU admission for COVID-19: application of time-dependent receiver operating characteristic curves

Abstract

Objective

Intensive care unit (ICU) admission occurs at different times during hospitalization among patients with COVID-19. We aimed to evaluate the time-dependent receive operating characteristic (ROC) curve and area under the ROC curve, AUC(t), and accuracy of baseline levels of inflammatory markers C-reactive protein (CRP) and neutrophil-to-lymphocyte ratio (NLR) in predicting time to an ICU admission event in patients with severe COVID-19 infection.

Methods

In this observational study, we evaluated 724 patients with confirmed severe COVID-19 referred to Ayatollah Rohani Hospital, affiliated with Babol University of Medical Sciences, Iran.

Results

The AUC(t) of CRP and NLR reached 0.741 (95% confidence interval [CI]: 0.661–0.820) and 0.690 (95% CI: 0.607–0.772), respectively, in the first 3 days after hospital admission. The optimal cutoff values of CRP and NLR for stratification of ICU admission outcomes in patients with severe COVID-19 were 78 mg/L and 5.13, respectively. The risk of ICU admission was significantly greater for patients with these cutoff values (CRP hazard ratio = 2.98; 95% CI: 1.58–5.62; NLR hazard ratio = 2.90; 95% CI: 1.45–5.77).

Conclusions

Using time-dependent ROC curves, CRP and NLR values at hospital admission were important predictors of ICU admission. This approach is more efficient than using standard ROC curves.

Keywords

Neutrophil-to-lymphocyte ratio C-reactive protein time-dependent area under the receiver operating characteristic curve COVID-19 intensive care unit admission time-dependent ROC curve

Introduction

In December 2019, an outbreak of the novel coronavirus disease COVID-19 occurred for the first time in Wuhan, China and spread rapidly to all countries worldwide, leading to the World Health Organization (WHO) declaring COVID-19 to be a pandemic. Globally, as of 5 April 2021, more than 131,000,000 confirmed cases of COVID-19 and 2,850,521 deaths have been reported to the WHO.¹

Although previous studies have shown that most patients with COVID-19 (nearly 80%) have initially mild or moderate disease and a favorable clinical outcome, others experience severe or critical illness.² Early diagnosis of these patients is crucial to decrease admissions to the intensive care unit (ICU) as well as the length of hospitalization, mortality rate, and hospital costs.^3,4 Cytotoxicity caused by the causative agent, SARS-CoV-2, in pulmonary endothelial cells and the consequences of severe inflammatory outcomes can cause serious and even life-threatening illness.⁵ Recent studies have shown that higher levels of inflammatory markers in the blood at admission, such as C-reactive protein (CRP) and the neutrophil-to-lymphocyte ratio (NLR), are associated with disease severity and death among hospitalized patients with COVID-19.^6–8 The receiver operating characteristic (ROC) curve is frequently used to evaluate the diagnostic accuracy of baseline values of CRP and NLR in determining disease status by depicting the sensitivity (true positive rate) versus 1-specificity (true negative rate). The area under the ROC curve (AUC) is considered to indicate the probability of correct classification of disease versus healthy individuals.^8–11 In past studies, CRP and disease status have been measured at the same time whereas other indicators of disease status, such as admission to the ICU or death, are assessed at different times during follow-up. Thus, sensitivity, specificity, and ROC curves are time-dependent, and it is preferable to compute these at different times.^12–17 An important point to consider when selecting time-dependent ROC curves, or ROC(t), for predicting the accuracy of a biomarker is that the endpoint (i.e., the time to occurrence of an event, given the value of the biomarker at baseline), the risk of disease progression, and the event of interest often change over time. Hence, using a standard ROC curve is unsuitable and may provide misleading findings.¹⁸ We designed this study to evaluate prediction of patient classification in terms of the risk of ICU admission among those with severe COVID-19 infection using ROC(t) curve analysis.

Methods

Study design and participants

This was a historical observational cohort study among patients with COVID-19 who were referred to Ayatollah Rohani Hospital in Babol, Iran between 22 October 2020 and 5 March 2021. The data were extracted from patients’ electronic medical records. The study protocol was approved by the Institutional Ethical Board of Babol University of Medical Sciences (Ethics code: IR.MUBABOL.REC.1400.204). Consent from patients/subjects participating in this study was not required because this study was retrospective and the data were extracted from patients’ electronic medical records.

We excluded patients under age 18 years who were initially admitted to the ICU, as well as patients who died or were discharged within 24 hours of hospitalization, and those with missing values for CRP or NLR at baseline or for up to 2 days after hospital admission.

As shown in Figure 1, the sample included in this analysis included patients with severe illness, defined according to the WHO guidelines for COVID-19.¹⁹ In defining severe cases, the following criteria were used: (1) respiratory rate >30 counts per minute, (2) oxygen saturation ≤93%, and (3) clinical signs of severe respiratory distress.

Figure 1.

Flow chart depicting inclusion and exclusion of patients in the study.

The reporting of this study conforms to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement guidelines.²⁰

Data collection and definitions

Demographic, laboratory, and clinical outcome data were collected from patients’ electronic medical records. Demographic characteristics included age and sex. Clinical data comprised comorbidities and symptoms as well as laboratory parameters measured at admission, such as neutrophils, lymphocytes, ferritin, D-dimer, total bilirubin, erythrocyte sedimentation rate (ESR), and procalcitonin (PCT). Data of the inflammatory biomarker CRP and the NLR were also used. CRP and NLR were measured at baseline or up to 2 days after admission. Values for blood urea nitrogen (BUN), creatinine, and liver enzymes such as alanine aminotransferase, aspartate aminotransferase (AST), and alkaline phosphatase (ALP) were collected. Clinical outcomes of the study were ICU admission as the event of interest, and time to this event for patients with severe COVID-19, which was the time (in days) from hospital admission to ICU admission.

Statistical analysis

Descriptive statistics are reported as frequency and percentage for categorical data and as median and interquartile range (IQR) for quantitative laboratory findings according to sex. In bivariate analysis, we used the chi-square test for categorical data and the Mann–Whitney test for quantitative laboratory findings to compare non-survivors and survivors. We performed univariate and multivariable Cox regression models to explore the predictors of survival (after ICU admission) by estimating the hazard ratio (HR) and 95% confidence interval (CI). We used time-dependent sensitivity and specificity, and ROC(t) curves created according to the cumulative/dynamic definitions of Heagerty and Zheng,²¹ to evaluate the predictive ability of CRP and NLR at different times from days 3 to 12 after hospital admission and to select a suitable threshold for the largest area under the ROC(t) curve, or AUC(t). For a threshold c and a given time t, the cumulative sensitivity Se (c, t), patients who experience the event of interest at time t and have a biomarker value greater than c are considered true positives (i.e., admission to the ICU). For dynamic specificity Sp (c, t), patients who survive beyond time t and have a biomarker value less than or equal to c are considered true negatives (no ICU admission).

Under this definition, the sensitivity and specificity are defined as:

Se (c, t) = P (X > c | T \leq t) andSp (c, t) = P (X \leq c | T > t)

The ROC(t) is defined as the plot of Se (c, t) versus 1-Sp (c, t) for all values of c; this curve suggests the best threshold for a marker. The AUC(t) is therefore equal to the probability of (Xi > Xj|Ti ≤ t,Tj > t), with i and j representing the indexes for two independent patients.¹⁵ We selected an optimal cut-point for each AUC(t) according to different times (days). There are several methods to determine the optimal cutoff values, one being the Youden index that selects the maximum sensitivity + specificity−1.²² Finally, the performance of CRP and NLR was estimated according to the value of HRs obtained from the desired optimal cut-point. All statistical analyses were performed with R software version 3.6.3 (The R Project for Statistical Computing, Vienna, Austria).

Results

A total of 724 patients with severe COVID-19 were included in this analysis, among which 49% were female and 51% were male patients. The mean (±standard deviation) ages of the non-survivor group (71.42±13.91 years) and the survivor group (58.92 ± 15.85 years) were significantly different (p < 0.001); most patients were over 65 years old. Fifty-two (7.2%) patients were admitted to the ICU during hospitalization. The proportion of non-survivors admitted to the ICU was larger than the proportion among survivors (46.7% vs. 1.5%; p < 0.001). The results of comparisons of demographic and laboratory findings between the non-survivor and survivor groups are shown in Tables 1 and 2.

Table 1.

Demographic and clinical outcomes of non-survivors and survivors with severe COVID-19 infection.

Demographic andclinical characteristics	Total(n = 724)	Non-survivors(n = 75)	Survivors(n = 649)	p-value^†
Age (years), n (%)				<0.001
18–45	126 (17.5)	4 (3.2)	122 (96.8)
45–65	290 (40.3)	17 (5.9)	273 (94.1)
≥65	303 (42.1)	54 (17.8)	249 (82.2)
Sex
Male		27 (7.3)	343 (92.7)	0.006
Female		48 (13.6)	306 (86.4)
Comorbidities
Diabetes mellitus, n (%)	179 (24.7)	22 (29.3)	157 (24.2)	0.328
Hypertension, n (%)	164 (22.7)	25 (33.3)	139 (21.4)	0.020
Chronic kidney diseases, n (%)	9 (1.2)	3 (4.0)	6 (0.9)	0.023
Cardiovascular diseases, n (%)	111 (15.3)	19 (25.3)	92 (14.2)	0.011
Nervous system diseases, n (%)	12 (1.7)	4 (5.3)	8 (1.2)	0.008
Pulmonary diseases, n (%)	7 (1)	2 (2.7)	5 (0.8)	0.112
Cancer, n (%)	13 (1.8)	2 (2.7)	11 (1.7)	0.548
Signs and symptoms
Fever, n (%)	365 (50.4)	35 (46.7)	330 (50.8)	0.493
Cough, n (%)	326 (45.0)	32 (42.7)	294 (45.3)	0.664
Dyspnea, n (%)	331 (45.7)	38 (50.7)	293 (45.1)	0.364
Headache, n (%)	57 (7.9)	5 (6.7)	52 (8.0)	0.682
Anorexia, n (%)	138 (19.1)	13 (17.3)	125 (19.3)	0.687
Chest pain, n (%)	16 (2.2)	2 (2.7)	14 (2.2)	0.776
Abdominal pain, n (%)	15 (2.1)	1 (1.3)	14 (2.2)	0.635
Dizziness, n (%)	22 (3)	1 (1.3)	21 (3.2)	0.363
Myalgia or fatigue, n (%)	374 (51.7)	36 (48.0)	338 (52.1)	0.503
Diarrhea, n (%)	30 (4.1)	2 (2.7)	28 (4.3)	0.498
Nausea and vomiting, n (%)	84 (11.6)	5 (6.7)	79 (12.2)	0.159
Loss of smell, n (%)	8 (1.1)	1 (1.3)	7 (1.1)	0.842
Loss of taste, n (%)	9 (1.2)	0 (0)	9 (1.4)	0.305
Oxygen saturation, median (IQR)	90 (90–92)	90 (85–90)	90 (90–92)	0.001
Length of illness onset tohospitalization (days), median (IQR)	3 (1–7)	3 (1–6)	3 (1–7)	0.837
Length of hospital stay (days),median (IQR)	7 (5–10)	8 (5–14)	7 (5–9)	0.021
Time from hospitalization to ICU (days),median (IQR)	6 (4–8)	3 (2–7)	6 (4–8)	<0.001
Clinical outcomes
Admission to ICU, n (%)	52 (7.2)	35 (46.7)	10 (1.5)	<0.001

^†p-values calculated with chi-square test and Mann–Whitney U-test.

IQR, interquartile range; ICU, intensive care unit.

Table 2.

Laboratory values among non-survivors and survivors with severe COVID-19 infection.

Parameters	Total(n=724)	Non-survivors(n=75)	Survivors(n=649)	p-value^†
Neutrophil count (×10⁹/L), median (IQR)	80 (70–85)	86 (80–90)	78 (70–85)	<0.001
Lymphocyte count (×10⁹/L), median (IQR)	19 (13–26)	12 (9.2–18.0)	20 (15–27)	<0.001
NLR, median (IQR)	4.16 (2.69–6.17)	7.3 (4.5–9.58)	4 (2.5–5.6)	<0.001
Creatinine (mmol/L), median (IQR)	1 (0.8–1.2)	1.1 (0.9–1.5)	1 (0.8–1.1)	0.001
Blood urea nitrogen (mmol/L), median (IQR)	19 (15–25)	27 (19–44)	19 (14–24)	<0.001
Ferritin (g/L), median (IQR)	437 (260–843.75)	652 (345–1400)	432 (256–800)	0.183
D-dimer (mg/L), median (IQR)	408 (291–827)	802 (263–2018)	406 (293–783)	0.214
D-dimer (mg/L), n (%)
<1000	100 (84.7)	5 (55.6)	95 (87.2)	0.011
≥1000	18 (15.3)	4 (12.8)	14 (44.4)
Total bilirubin (mmol/L), median (IQR)	0.5 (0.4–0.6)	0.5 (0.4–0.73)	0.5 (0.4–0.6)	0.106
Alanine transferase (U/L), median (IQR)	29 (20–42)	29 (20–42)	29 (20–42)	0.998
Aspartate transferase (U/L), median (IQR)	40 (32–55)	54 (40–90)	39 (31–53)	<0.001
Alkaline phosphatase (U/L), median (IQR)	161 (128–207)	183.5 (137.5–270.7)	159 (127–203)	0.004
Erythrocyte sedimentation rate (mm/h), median (IQR)	38 (23–65)	50 (30–78)	35 (22–60)	0.010
C-reactive protein (mg/L), median (IQR)	69 (32–97)	94 (68–110)	65 (30–94)	<0.001
C-reactive protein (mg/L), n (%)
<64	326 (45.0)	16 (21.3)	310 (47.8)	<0.001
≥64	398 (55.0)	59 (78.7)	339 (52.2)
Procalcitonin (ng/mL), median (IQR)	0.07 (0.07–0.30)	0.3 (0.07–2.63)	0.07 (0.07–0.16)	<0.001

^†p-values calculated with chi-square test and Mann–Whitney U-test.

IQR, interquartile range; NLR, neutrophil-to-lymphocyte ratio.

Table 1 shows that non-survivors comprised a significantly larger proportion of older patients than survivors (72% vs. 38.7%). The prevalence of comorbid diseases was also higher among non-survivors than survivors for hypertension (33.3% vs. 21.4%), chronic kidney disease (4% vs. 0.9%), cardiovascular diseases (25.3% vs. 14.2%), and nervous system diseases (5.3% vs. 1.2%), and admission to an ICU (46.7% vs 1.5%). The median time from hospitalization to ICU admission was 3 (IQR 2–7) days versus 6 (IQR 4–8) days in the two groups, respectively. The values of laboratory parameters are reported in Table 2. Levels of BUN, AST, ALP, ESR, CRP, PCT, NLR, neutrophil count, and creatinine were all significantly elevated in non-survivors, as compared with survivors. Non-survivors also had significantly lower lymphocyte counts (Table 2).

Based on univariate analysis in the Cox regression model, NLR, PCT, CRP, and incidence of comorbidities such as chronic kidney disease had significant associations with ICU admission, with HR (95% CI) values 1.13 (1.06–1.20), 1.61 (1.17–2.21), 4.14 (1.95–8.82), and 4.26 (1.31–13.85), respectively. After including indicators with p < 0.10 in the multivariate Cox regression model, we found that CRP was associated with an ICU admission event. Compared with patients who had CRP < 64 mg/L, those with CRP ≥64 mg/L were more likely to be admitted to the ICU (HR: 3.08, 95% CI 1.11–8.55; p = 0.030). For NLR, the HR was 1.09 (95% CI 1.01–1.19; p = 0.023). The time from symptom onset to hospitalization and PCT were not associated with ICU admission (Table 3). The AUC(t) values of CRP and NLR are shown in Figure 2. The figure shows that these two markers had higher AUC(t)s in the first 3 days, suggesting that CRP and NLR are the best predictors of an ICU admission event before the third day after hospital admission. Table 4 shows the results of ROC(t) analysis, which revealed the cutoff value for CRP was 78 mg/L and its AUC (t = 3 days) was 0.741 (95% CI 0.661–0.820; sensitivity 0.817, specificity 0.607). The optimal cutoff value for NLR was 5.13 and the AUC (t = 3 days) value was 0.690 (95% CI 0.607–0.772; sensitivity 0.715, specificity 0.625). We also compared the results of the ROC(t) curve with those of a standard ROC curve, which showed that the AUC of the standard ROC curve was lower than that of the ROC(t) curve (Table 4). Stratification of patients' risks based on the desired optimal cut-point was illustrated using Kaplan–Meier curves (Figure 2), with the log-rank test (p = 0.001 for CRP and p = 0.001 for NLR). The HR of ICU admission among patients with CRP above the estimated optimal cut-point (78 mg/L) versus below this cut-point was 2.98 (95% CI 1.58–5.62; p = 0.001) and the HR for NLR above the estimated cut-point (5.13) compared with below this cutoff was 2.90 (95% CI 1.45–5.77; p = 0.002). Ultimately, the AUC remained constant after a time of 15 days because a few patients experienced admission to the ICU beyond 15 days of follow-up (Figure 3).

Table 3.

Univariate and multivariable Cox regression models to explore the independent associations of CRP, NLR, and risk of ICU admission among patients with severe COVID-19 infection.

Variables	Unadjusted Adjusted
Variables	HR (95% CI)	p-value	HR (95% CI)	p-value^††
Female sex	1.34 (0.77–3.37)	0.298	–	–
Age (years)			–	–
18–44	0.55 (0.23–1.33)	0.183	–	–
45–64	0.62 (0.34–1.14)	0.124	–	–
≥65	1 (ref)		–	–
Hypertension (yes)	1.05 (0.53–2.08)	0.879	–	–
Diabetes (yes)	1.05 (0.54–2.04)	0.879	–	–
Chronic kidney disease (yes)	4.26 (1.31–13.85)	0.016	7.07 (1.56–30.90)	0.011
Cardiovascular diseases (yes)	1.17 (0.56–2.44)	0.674	–	–
Nervous system diseases (yes)	1.51 (0.21–10.94)	0.686	–	–
Pulmonary diseases (yes)	3.47 (0.84–14.38)	0.087	5.66 (1.32–24.23)	0.020
Cancer (yes)	1.11 (0.15–8.09)	0.915	–	–
Neutrophil count (×10⁹/L)^†	1.08 (1.03–1.13)	0.001	–	–
Lymphocyte count (×10⁹/L)^†	0.93 (0.89–0.97)	0.002	–	–
NLR^†	1.13 (1.06–1.20)	<0.001	1.09 (1.01–1.19)	0.023
C-reactive protein (mg/L), <64 vs. ≥64	4.14 (1.95–8.82)	<0.001	3.08 (1.11–8.55)	0.030
D-dimer (<1000 vs. ≥1000)	1.95 (0.37–10.11)	0.426	–	–
Creatinine (mmol/L)^†	1.24 (0.967–1.59)	0.090	–	–
Procalcitonin (ng/mL)^†	1.61 (1.17–2.21)	0.003	–	–
Erythrocyte sedimentation rate (mm/h)^†	1.01 (1.00–1.02)	0.004	–	–
Alanine transferase (U/L)^†	1.00 (0.99–1.01)	0.736	–	–
Aspartate transferase (U/L)^†	1.01 (1.00–1.02)	<0.001	–	–
Alkaline phosphatase (U/L)^†	1.001 (1.000–1.003)	0.093	–	–
Total bilirubin (mmol/L)^†	1.16 (0.880–1.53)	0.279	–	–
Blood urea nitrogen (mmol/L)^†	1.02 (1.00–1.03)	0.022	–	–
Ferritin (g/L)^†	0.79 (0.21–2.96)	0.729	–	–
Time from symptom onset to hospitalization (days)^†	1.03 (0.96–1.08)	0.373	–	–

NLR, neutrophil-to-lymphocyte ratio HR, hazard ratio; CI, confidence interval; ICU, intensive care unit.

^†HRs estimated for an additional increase for markers in the corresponding scale.

^††p-value calculated using Wald statistic.

Figure 2.

(a) and (c) Time-dependent ROC curves at 3, 6, 9, and 12 days prior to ICU admission for CRP and NLR. (b) and (d) Kaplan–Meier curves of CRP and NLR based on the cutoff for patients with severe COVID-19 infection.

Table 4.

Estimated AUC, sensitivity, specificity, and the optimal cut-points of CRP and NLR.

		CRP				NLR
Method	Time	AUC (95% CI)	Se	Sp	Cutoff	AUC (95% CI)	Se	Sp	Cutoff

St.	–	0.682 (0.619–0.744)	0.756	0.426	73.5	0.666 (0.569–0.764)	0.625	0.329	5.6
	3	0.741 (0.661–0.820)	0.817	0.607	78	0.690 (0.607–0.772)	0.715	0.625	5.13
C/D	6	0.718 (0.637–0.798)	0.763	0.628	80	0.667 (0.584–0.750)	0.665	0.618	5.00
	9	0.668 (0.585–0.750)	0.804	0.455	61	0.655 (0.572–0.738)	0.615	0.574	4.50
	12	0.669 (0.586–0.752)	0.829	0.565	62	0.581 (0.497–0.665)	0.730	0.406	3.67

Sp, specificity; Se, sensitivity; AUC, area under the receiver operating characteristic curve; CRP, C-reactive protein; NLR, neutrophil-to-lymphocyte ratio; C/D, cumulative/dynamic time-dependent receiver operating characteristic curve; St., standard receiver operating characteristic curve; CI, confidence interval.

Figure 3.

AUC(t) based on (a) CRP and (b) NLR markers for 30 days.

Discussion

Our findings showed the AUC(t) at 3 days for CRP and NLR were more efficient predictors than the standard AUC regarding the risk of ICU admission. Our study showed that the corresponding HRs above the estimated optimal cut-point at time of the highest AUC(t) were significant for both CRP and NLR. Thus, these two inflammatory markers are important predictors of ICU admission in severe cases of COVID-19. The relationship between inflammatory markers (NLR and CRP) and disease progression has been demonstrated in several clinical trials,^8,23–25 in accordance with our findings. CRP and NLR are valuable indicators, and high values are associated with a worse patient outcome. A study by Prasetya et al. revealed that patients with severe COVID-19 had a higher risk of ICU admission with CRP > 47 mg/L and NLR > 6. However, those authors used a traditional/standard ROC curve to determine the associations between these markers and ICU admission during hospitalization. The optimal cutoff was specified by choosing the highest AUC value.⁸

Because patients with severe COVID-19 may require admission to the ICU at different time points after hospitalization, time-dependent sensitivity, specificity, and ROC curves are expected to vary with time. In the current study, ROC(t) curves were used instead of the standard ROC curve approach.¹⁵ In such situations, classifying patients while ignoring the time-dependence of an ICU event is inappropriate^15,18,26 For this purpose, the ROC(t) is estimated as the AUC(t) at different time points t, and the optimal cutoff is determined by selecting the highest AUC value indicating that the marker has the ability to predict ICU admission.^15,27 Huiqing et al. researched the performance of cumulative oxygen deficit (COD) in predicting death among patients with COVID-19 and acute respiratory hypoxemia who were hospitalized in Jingmen, Wuhan. The predictive ability of COD was estimated using the AUC(t). The COD was found to be a good predictor after day 14, and the authors determined that patients with COD >30 mmHg/day were more likely to die during hospitalization.²⁸ Using a time-dependent, cumulative/dynamic ROC/AUC approach, we estimated AUC(t) values between 0.741 and 0.669 for CRP and between 0.690 and 0.581 for NLR during the first 12 days after hospital admission. When we considered the AUC(t), the best predictive ability occurred at approximately 3 days, compared with the standard ROC curve. Our results suggest that these markers have good predictive ability for the progression from admission to ICU admission, with the highest AUC = 0.741 for CRP and AUC = 0.690 for NLR. The AUC values seemed to decrease from AUC (t = 3 days) to AUC (t = 12 days), indicating that the discrimination potential of these markers declined during study follow-up. Using an ROC(t) curve approach, the NLR cutoff value of ICU admission was 5.13 in our study; another study suggested a cutoff of 3.71 using a standard ROC curve.²⁹ In yet another study of patients with COVID-19, a CRP above 41.8 mg/L was accepted as indicating the possibility of developing severe disease,³⁰ and Asghar et al. found that the cutoff for ICU admission was above 103.6.²⁹ In our study, the cutoff value of CRP was found to be 78 mg/L. The reason for the differences in cut-point values among studies may be because only patients who had a CRP value at admission or up to 2 days after hospitalization were included in our study. Moreover, the prediction value during hospitalization owing to medication and changes in the disease process may be distorted from the actual value of the optimal cut-point.

Considering CRP as a binary variable specified by the cutoff determined using this method, we showed that higher values of the estimated threshold increased the risk of ICU admission 2.98 times and 2.25 times according to the cutoff determined for NLR. The clinical application of these markers is clearly observed in COVID-19. Other findings of our study showed that the AUC(t) of CRP was considerably higher than that of NLR (0.741 vs. 0.690), indicating that CRP has better predictive performance than NLR in most cases. Thus, clinicians should consider the magnitude of CRP in the clinical management of patients with COVID-19. Our results are in accordance with those reported by Akan et al.³⁰ but do not agree with other previous findings.⁹ Further, Yufei et al. showed that the AUC of combined NLR and CRP improved diagnostic accuracy in predicting COVID-19.¹⁰

No previous studies have used ROC(t) analysis to examine the association of CRP and NLR, with ICU outcome during hospitalization. This approach has been proven to be more efficient when estimating the ROC.^15,16 It is also suggested that this method be used with other biomarkers and the corresponding time-process events.

The performance of these markers in predicting patient outcomes was somewhat limited owing to the small number of patients admitted to the ICU at our hospital. A prospective multicenter cohort study with a large sample size can provide better evidence in future research.

Conclusion

Analysis of the AUC(t) using baseline value of markers such as CRP and NLR can provide more efficient results in predicting short-term outcomes of patients with COVID-19 infection, such as ICU admission. This approach should be adapted to diagnostic studies in clinical research with time-dependent prognostic factors in ROC analysis.

Footnotes

Acknowledgement

The authors would like to acknowledge the deputy of Research and Technology of Babol University of Medical Sciences for the support that was provided.

Declaration of conflicting interest

The authors declare that they have no conflicts of interest.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

ORCID iDs

Zahra Geraili

Karimollah Hajian-Tilaki

References

World Health Organization (2020) Director-General's remarks at the media briefing on 2019-nCoV on 5 April 2021. https://covid19.who.int/?gclid.

Bal

Dogan

Cabalak

, et al. Lymphocyte-to-C-reactive protein ratio may serve as an effective biomarker to determine COVID-19 disease severity. Turkish Journal of Biochemistry 2021; 46: 21–26.

Guan

, et al. Clinical characteristics of coronavirus disease 2019 in China. New England Journal of Medicine 2020; 382: 1708–1720.

Zhang

Luo

, et al. Clinical features and short-term outcomes of 221 patients with COVID-19 in Wuhan, China. J Clin Virol 2020; 127: 104364.

Rowaiye

Okpalefe

Adejoke

, et al. Attenuating the Effects of Novel COVID-19 (SARS-CoV-2) Infection-Induced Cytokine Storm and the Implications. J Inflamm Res 2021; 14: 1487–1423.

Karagoz

Yoldas

Platelet to lymphocyte and neutrophil to lymphocyte ratios as strong predictors of mortality in intensive care population. Rev Assoc Med Bras 2019; 65: 633–636.

Merad

Martin

JC.

Pathological inflammation in patients with COVID-19: a key role for monocytes and macrophages.

Nat Rev Immunol 2020; 20: 355–362.

Prasetya

Lorens

Sungono

, et al. Prognostic value of inflammatory markers in patients with COVID-19 in Indonesia. Clin Epidemiol Glob Health 2021: 11: 100803.

Hashem

Khedr

Daef

, et al. Prognostic biomarkers in COVID-19 infection: value of anemia, neutrophil-to-lymphocyte ratio, platelet-to-lymphocyte ratio, and D-dimer. The Egyptian Journal of Bronchology 2021; 15: 1–9.

10.

Yufei

Mingli

Xuejiao

, et al. Utility of the neutrophil-to-lymphocyte ratio and C-reactive protein level for coronavirus disease 2019 (COVID-19). Scand J Clin Lab Invest 2020; 80: 536–540.

11.

Hajian-Tilaki

Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med 2013; 4: 627–635.

12.

Beyene

El Ghouch

Smoothed time-dependent ROC curve for right censored survival data. 2019; 39: 3373–3396.

13.

Blanche

Dartigues

JacqminGadda

Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J 2013; 55: 687–704.

14.

Heagerty

Lumley

Pepe

MS.

Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics 2000; 56: 337–344.

15.

Kamarudin

Cox

Kolamunnage-Dona

Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol 2017; 17: 1–19.

16.

Lambert

Chevret

Summary measure of discrimination in survival models based on cumulative/dynamic time-dependent ROC curves.

Stat Methods Med Res 2016; 25: 2088–2102.

17.

Greene

A simple method to estimate the time-dependent receiver operating characteristic curve and the area under the curve with right censored data

Stat Methods Med Res 2018; 27: 2264–2214.

18.

Shen

Ning

Yuan

A direct method to evaluate the time-dependent predictive accuracy for biomarkers.

Biometrics 2015; 71: 439–449.

19.

World Health Organization. Clinical management of severe acute respiratory infection when Novel coronavirus (nCoV) infection is suspected: interim guidance. March 13, 2020. https://apps.who.int/iris/handle/10665/331446.

20.

Von Elm

Altman

Egger

, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 2007; 147: 573–577.

21.

Heagerty

Zheng

Survival model predictive accuracy and ROC curves.

Biometrics 2005; 61: 92–13.

22.

HajianTilaki

The choice of methods in determining the optimal cut-off value for quantitative diagnostic test evaluation. Stat Methods Med Res 2018; 27: 2374–2379.

23.

Abakay

Sen

, et al. The relationship between inflammatory marker levels and pulmonary tuberculosis severity. Inflammation 2015; 38: 691–696.

24.

Mousavi-Nasab

Mardani

Nasr Azadani

, et al. Neutrophil to lymphocyte ratio and C-reactive protein level as prognostic markers in mild versus severe COVID-19 patients. Gastroenterol Hepatol Bed Bench 2020; 13: 361–365.

25.

Ponti

Maccaferri

Ruini

, et al. Biomarkers associated with COVID-19 disease progression. Crit Rev Clin Lab Sci 2020; 57: 389–399.

26.

Pepe

MS.

The statistical evaluation of medical tests for classification and prediction . New York, NY: Oxford University Press, 2003.

27.

Cai

Pepe

Zheng

, et al. The sensitivity and specificity of markers for event times. Biostatistics 2006; 7: 182–115.

28.

Zhou

, et al. Cumulative oxygen deficit is a novel biomarker for the timing of invasive mechanical ventilation in COVID-19 patients with respiratory distress: a time-dependent propensity score analysis. PeerJ 2020; 8: e10497.

29.

Asghar

Kazmi

SJH

Khan

, et al. Clinical profiles, characteristics, and outcomes of the first 100 admitted COVID-19 patients in Pakistan: a single-center retrospective study in a tertiary care hospital of Karachi. Cureus 2020; 12: e8712.

30.

Akan

Bilgir

Effects of Neutrophil/Monocyte, Neutrophil/Lymphocyte, Neutrophil/Platelet Ratios and C-Reactive Protein Levels on the Mortality and Intensive Care Need of the Patients Diagnosed with Covid-19. Eurasian Journal of Medicine and Investigation. 2021; 5: 21–25.