Restricted ROC curves are useful tools to evaluate the performance of tumour markers

Abstract

In Clinical Epidemiology, receiver operating characteristic (ROC) analysis is a standard approach for the evaluation of the performance of diagnostic tests for binary classification based on a tumour marker distribution. The area under a ROC curve is a popular indicator of test accuracy, but its use has been questioned when the curve is asymmetric. This situation often happens when the marker concentrations overlap in the two groups under study in the range of low specificity, corresponding to a subset of values useless for classification purposes (non-informative values). The partial area under the curve at a high specificity threshold has been proposed as an alternative, but a method to identify an optimal cut-off that separates informative from non-informative values is not yet available. In this study, a new statistical approach is proposed to perform this task. Furthermore, a statistical test associated with the area under a ROC curve corresponding to informative values only (restricted ROC curve) is provided and its properties are explored by extensive simulations. Finally, the proposed method is applied to a real data set containing peripheral blood levels of six tumour markers proposed for the diagnosis of neuroblastoma. A new approach to combine couples of markers for classification purposes is also illustrated.

Keywords

Tumour markers receiver operating characteristic analysis diagnostic tests restricted receiver operating characteristic curve

1 Introduction

Receiver operating characteristic (ROC) curves are statistical tools largely applied in Clinical Epidemiology to evaluate the performance of tumour markers (TMs) in the context of binary classification problems.^1–3 Because the early steps for a marker validation, in general, require a case-control approach,^2,4 in this article the class of diseased subjects will be referred to as the ‘cases’ and the referent group, which typically includes healthy individuals or subjects affected by less severe diseases, as the ‘controls’. In general, to be useful for diagnostic purposes a TM should present, on average, higher values in the class of the cases than in that of the controls. A binary test may be obtained by selecting a specific value of a continuous TM and by defining as positive any test with value exceeding such a threshold, and as negative the remaining other tests. The proportion of positive cases provides an estimate of the test sensitivity and the proportion of negative controls an estimate of its specificity. A ROC curve is obtained by plotting the true positive fraction (sensitivity) vs the false positive fraction (1-specificity) using all the available values of a TM concentration. Figure 1(a) shows a theoretical ROC curve obtained from infinite values of a hypothetical TM and an empirical ROC curve obtained from a finite sample of 200 units (100 cases and 100 controls) randomly selected from the same binormal distribution (displayed in panel (b)).

The area under a ROC curve (AUC) is a popular measure of diagnostic accuracy. It is equivalent to the Mann-Whitney U statistic, thus representing the probability that a subject, randomly selected among the case class, shows a marker value higher than a subject randomly extracted from the controls.^5,6 For completely non-informative TMs the ROC curve will approach the rising diagonal (called ‘chance diagonal’ or ‘chance line’, Figure 1(a)) and AUC will tend to 0.5, i.e. the expected probability for a classification due to chance alone. On the contrary, in the case of a perfect classification the ROC curve will reach the point of the highest theoretical accuracy (sensitivity and specificity both 100%) and AUC will tend to one, i.e. the highest probability value. A statistical test to verify that AUC is different from its expected value under the null hypothesis is easily performed exploiting the normal asymptotic distribution of the U statistic^7–9

zAUC = \frac{\overset{\land}{AUC} - 0.5}{\sqrt{\overset{\land}{Var} (\overset{\land}{AUC})}}

(1)

where Var(AUC) represents the variance of AUC that may be estimated by some different formulas.^5,8,9 In this study, the following equation was adopted, which was derived from the variance of the U statistic under the null hypothesis¹⁰

\overset{\land}{Var} (\overset{\land}{AUC}) = \frac{n_{0} + n_{1} + 1}{12 n_{0} n_{1}}

(2)

An optimal cut-off for a binary classification test may be identified on a ROC curve as the value corresponding to the highest vertical distance J from the chance line (Figure 1, panel (a)). J also corresponds to the highest value of the Youden’s index, a popular measure of pure accuracy.^2,11,12

Proper ROC curves are concave and symmetric and never cross each other, thus allowing an easy and reliable comparison between the corresponding markers. In fact, for proper ROC curves, the highest AUC ever corresponds to the highest sensitivity at any specificity value. Furthermore, the selection of an optimal cut-off on proper curves is easily performed, because they include a unique point that corresponds to the optimal threshold based on the Youden’s index.^2,11,12 However, in the analysis of TM, asymmetric (not proper) ROC curves are often encountered, due to the presence in the two classes under study of overlapping marker values at an extreme of their distributions. In many situations, a lack of sensitivity is observed in the correspondence of the lowest range of specificity values,¹³ causing the corresponding ROC curve to be unimodal and right asymmetric with respect to the chance line. In this article, curves of this kind will be called ‘positive skewed ROC’. Four examples of such curves are illustrated in Figure 2(a) to (d), while Figure 3(a) to (d) shows four corresponding density distributions of four hypothetical TM concentrations. In more details, the ROC curve in Figure 2(a) is positive skewed and concave, and may correspond to a unimodal distribution of marker values among the controls and a bimodal ones among the cases (‘Normal–Binormal model’, Figure 3(a)). In the real world, this situation may occur when a subgroup of cases does not differ from the controls for the expression of the TM under study.¹³ Another situation is that illustrated in Figure 3(b), corresponding to the ROC curve in Figure 2(b), in which cases and controls have a similar distribution density (both normal), but among the cases the marker have both higher mean and variance values (‘Binormal model’ with different variances). This behaviour may be considered as a variant of the previous one, when the two subgroups of cases are less distinguishable. In that case, the corresponding ROC tends to lose its concavity at the lowest scale of specificity and it may even cross the chance line^2,3,13,14 (Figure 2(b)). Moreover, Figure 3(c) shows a bimodal distribution both in the cases and in the controls, with a subgroup within any class showing a similar TM distribution in the range of low marker values (‘Binormal–Binormal model’). This behaviour may be attributable to a detection threshold of the device used to measure the TM concentrations, which automatically replaces null values with a white noise. The corresponding ROC curve will be similar to that in Figure 2(c), when the curve approaches the chance line at low specificity values. Finally, a quite common variant of this situation is that of a TM with mass at zero, illustrated in Figure 3(d), in which the range of non-informative values is made by some zero values (‘Zero-inflated Binormal’ model). The corresponding ROC curve will be still positive skewed and it will also show a jump discontinuity of the derivative,¹⁵ like the curve in Figure 2(d).

In the case of not proper ROC curves, some Authors have proposed the use of the partial area under the curve (pAUC) at an a priori selected cut-off of high specificity, or based on some utility function.^13,16,17 All these approaches, however, do not provide a method to identify the range of non-informative values, if any, within the TM distribution. In general, subjects whose values fall in such a range are defined as ‘test negative’, albeit this range may include different proportions of cases and controls.

In this article, a new simple method of ROC analysis is illustrated, which allows to identify an optimal threshold to separate the range of non-informative values of a TM from those potentially useful for classification purposes. A statistical test associated with the ROC curve corresponding to the informative range only (restricted ROC, rROC) is also provided. Finally, a method to combine restricted and standard ROC analysis is illustrated and applied to a real data set of six TMs for the diagnosis of localized neuroblastoma.

This article is organized as follows: in Section 2 a new statistical test for rROC is provided, based on the standard normal distribution; in Section 3 its statistical power is investigated by extensive simulations and compared with that of the standard test on AUC; in Section 4 a simple rule for the choice between standard and rROC curves is illustrated; in Section 5 a method to combine information from standard and rROC curves is analysed; in Section 6 the new proposed method is applied to a real data set of concentrations of six putative TMs proposed for the diagnosis of neuroblastoma; in Section 7 a discussion of the method and the obtained results is provided.

2 Definition of a new statistical test for rROC

Let X be a TM whose concentrations are expressed on a continuous scale in two classes (controls and cases). Let Y be a random sample of X-values sorted in an ascending order in an array {y_i} (i = 1, … , N) and let n₀ be the sample size of the controls and n₁ that of the cases, with n₀ + n₁ = N. A left rROC curve (simply called rROC, because right restriction will not be considered in this article) is defined as the ROC curve obtained after the elimination from Y of the first j ordered values. For any value of j ( j = 0, … , N), a rROC _j may be identified and the corresponding area under the curve (rAUC_j) estimated. Similarly to equation (1) a test statistic may be associated to rAUC_j as follows

{rzAUC}_{j} = \frac{\overset{\land}{{rAUC}_{j}} - 0.5}{\sqrt{\overset{\land}{Var} (\overset{\land}{{rAUC}_{j}})}}

(3)

where, similarly to equation (2)

\overset{\land}{Var} (\overset{\land}{{rAUC}_{j}}) = \frac{{rn}_{0} + {rn}_{1} + 1}{12 {rn}_{0} {rn}_{1}}

(4)

and rn₀ and rn₁ represent the sample size of the two classes (controls and cases, respectively) after the restriction to N – j samples.

The new statistic test proposed in this article is defined as follows

rzAUC = Max ({rzAUC}_{j})

(5)

Accordingly to equations (3) and (5), rzAUC represents the highest ‘standardized’ AUC, i.e. the value that identifies the rROC with the highest accuracy, estimated taking into account both the probability of a correct classification (i.e. rAUC) and a measure of its variability. Accordingly, the corresponding j allows the identification of a cut-off C_j (if any) which separates informative TM values from non-informative ones. j may be zero in the case of non-restriction, i.e. when the standard AUC has a higher performance than any other rAUC_j. Finally, to allow rzAUC to be unique for any ROC curve, in the case of more than one value from equation (5), only that corresponding to the lowest j is retained, corresponding to the rROC based on the largest sample size.

The probability density of the rzAUC statistic under the null hypothesis H0 of an equal distribution of TM values among the two classes was investigated by extensive random simulations. N samples from a standard normal distribution were generated by letting N vary from 10 to 500 in each class. Each simulation was repeated 10⁵ times in order to obtain stable estimates of the expected value and of the variance of rzAUC. The analysis was performed by a software ad hoc developed by the freely available package Microsoft Visual Basic Express 2010. Random values from an uniform probability density were obtained by the RAN1 algorithm,¹⁸ while normal distributions were obtained by the GASDEV algorithm.¹⁸ Furthermore, most routines for both standard and rROC analysis were implemented in an open source statistical program (called ‘rROC’) freely available at: http://www.ge.ieiit.cnr.it/∼muselli/supplement/smmr-2012.html. Finally, histograms and the quantile–quantile inverse normal plots (qqplots), for the evaluation of the normality of rzAUC distribution, were obtained by STATA for Windows statistical package (release 11.0, Stata Corporation, College Station, TX, USA).

Figure 1S(a) to (f) in Supplemental Material shows six histograms of the simulated probability density of rzAUC under H0 in the comparison of two balanced classes. The density of rzAUC distribution seems to approach a normal probability function when the sample size is increased. However, in the presence of a high number of samples the distribution tends to be slightly leptokurtic and negative skewed (Figure 1S, panel (f)).

The following statistical test based on a standard normal distribution applied to rzAUC is proposed to test the null hypothesis of a non-informative rROC

z \approx \frac{\overset{\land}{rzAUC} - \overset{\land}{E} (\overset{\land}{rzAUC})}{\sqrt{\overset{\land}{Var} (\overset{\land}{rzAUC})}}

(6)

where the estimates of the expected value

\overset{\land}{E} (\overset{\land}{rzAUC})

and variance

\overset{\land}{Var} (\overset{\land}{rzAUC})

of rzAUC are obtained from the mean values and the variances of the simulated distributions, which are listed in Table 1 and in Table 2, respectively. To assess the degree of agreement of the proposed test to a standard normal distribution, simulated data were standardized using equation (6) and their cumulative distribution plotted vs the corresponding expected values from an inverse normal function (qqplot). Results are shown in Figure 2S(a) to (f) in the Supplemental Material, where the vertical lines correspond to the critical values for one and two sided tests at the conventional 0.05 threshold for the type I (α) error. rzAUC shows a quite good agreement with a Gaussian distribution, but it does not asymptotically converge to it. However, in each plot rzAUC strictly approaches the normal distribution in the correspondence of critical α-values, indicating that a small bias is expected to occur in the application of the test in equation (6) using conventional thresholds for statistical significance (Figure 2S). The same analysis was carried out for unbalanced groups (Figures 3S to 12S). Strong departures for normality was observed for very unbalanced groups when the smaller sample size referred to controls (see for example, Figure 3S, panel (f), which included 10 controls and 500 cases). Conversely, rzAUC showed a quite good approximation to the Gaussian distribution in the presence of a small number of samples among cases, even for very unbalanced classes (see for example Figure 11S, panel (a), which included 500 controls and 10 cases). The cumulative distribution of the standardized rzAUC showed a quite satisfactory agreement with the corresponding value of the inverse normal in the range of α = 0.05 also in the presence of a strong departure from normality of the corresponding density (see for example the qqplot in Figure 4S, panel (f)).

Table 1.

Expected value of rzAUC as a function of the sample size under the null hypothesis

n ₀	n ₁
n ₀	10	15	20	25	30	35	40	45	50	60	70	80	90	100	150	200	250	300	400	500
10	1.15	1.28	1.36	1.42	1.46	1.50	1.53	1.55	1.58	1.61	1.64	1.67	1.69	1.71	1.76	1.80	1.83	1.85	1.87	1.89
15	1.15	1.28	1.36	1.42	1.46	1.50	1.53	1.55	1.57	1.62	1.64	1.67	1.69	1.70	1.77	1.81	1.83	1.85	1.88	1.90
20	1.15	1.29	1.36	1.42	1.47	1.50	1.53	1.56	1.58	1.62	1.65	1.67	1.69	1.71	1.77	1.81	1.84	1.86	1.89	1.91
25	1.15	1.28	1.36	1.42	1.47	1.51	1.53	1.56	1.58	1.62	1.65	1.67	1.69	1.71	1.77	1.81	1.84	1.86	1.89	1.92
30	1.15	1.28	1.37	1.42	1.47	1.51	1.54	1.57	1.58	1.62	1.65	1.67	1.69	1.72	1.77	1.82	1.84	1.87	1.90	1.92
35	1.14	1.28	1.37	1.42	1.47	1.51	1.53	1.56	1.59	1.62	1.65	1.67	1.70	1.72	1.77	1.81	1.85	1.86	1.90	1.92
40	1.15	1.28	1.36	1.43	1.47	1.51	1.54	1.56	1.59	1.62	1.65	1.68	1.70	1.71	1.78	1.82	1.85	1.87	1.91	1.93
45	1.15	1.28	1.37	1.42	1.47	1.51	1.54	1.57	1.59	1.63	1.65	1.67	1.70	1.72	1.78	1.82	1.85	1.87	1.90	1.93
50	1.14	1.28	1.37	1.43	1.47	1.51	1.54	1.56	1.59	1.62	1.66	1.68	1.70	1.71	1.78	1.82	1.85	1.88	1.91	1.93
60	1.14	1.28	1.37	1.43	1.47	1.51	1.54	1.56	1.58	1.63	1.66	1.68	1.70	1.71	1.79	1.82	1.85	1.87	1.91	1.93
70	1.15	1.28	1.37	1.43	1.48	1.51	1.54	1.57	1.59	1.62	1.66	1.68	1.70	1.72	1.78	1.83	1.86	1.88	1.92	1.94
80	1.15	1.28	1.37	1.43	1.47	1.51	1.54	1.57	1.59	1.62	1.66	1.68	1.71	1.72	1.78	1.83	1.86	1.88	1.91	1.94
90	1.15	1.28	1.37	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.68	1.71	1.72	1.79	1.83	1.86	1.88	1.91	1.94
100	1.15	1.28	1.36	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.68	1.71	1.72	1.79	1.83	1.86	1.89	1.92	1.94
150	1.15	1.28	1.36	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.69	1.71	1.72	1.79	1.83	1.87	1.89	1.92	1.95
200	1.14	1.28	1.36	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.68	1.71	1.73	1.79	1.84	1.86	1.89	1.92	1.95
250	1.14	1.28	1.37	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.68	1.71	1.73	1.79	1.83	1.87	1.90	1.93	1.95
300	1.15	1.28	1.37	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.69	1.71	1.72	1.80	1.84	1.87	1.89	1.93	1.95
400	1.15	1.28	1.36	1.43	1.47	1.51	1.54	1.57	1.59	1.63	1.66	1.69	1.71	1.73	1.80	1.84	1.87	1.90	1.94	1.96
500	1.15	1.28	1.37	1.43	1.48	1.51	1.55	1.57	1.59	1.63	1.66	1.68	1.70	1.73	1.79	1.84	1.88	1.90	1.94	1.96

n₀ = number of controls; n₁ = number of cases.

Estimates obtained by 10⁵ random samples.

Table 2.

Variance of rzAUC as a function of the sample size under the null hypothesis

n ₀	n ₁
n ₀	10	15	20	25	30	35	40	45	50	60	70	80	90	100	150	200	250	300	400	500
10	0.59	0.52	0.49	0.46	0.45	0.43	0.41	0.40	0.39	0.37	0.36	0.34	0.33	0.32	0.28	0.26	0.24	0.23	0.21	0.20
15	0.60	0.55	0.51	0.49	0.47	0.45	0.44	0.43	0.41	0.40	0.39	0.38	0.37	0.35	0.32	0.30	0.28	0.26	0.24	0.23
20	0.60	0.55	0.52	0.50	0.48	0.46	0.45	0.44	0.43	0.41	0.40	0.39	0.38	0.38	0.35	0.31	0.30	0.29	0.27	0.26
25	0.60	0.55	0.52	0.50	0.48	0.47	0.45	0.45	0.44	0.42	0.41	0.41	0.39	0.39	0.36	0.34	0.32	0.31	0.29	0.28
30	0.61	0.56	0.53	0.50	0.49	0.48	0.47	0.45	0.45	0.43	0.42	0.41	0.40	0.40	0.37	0.35	0.34	0.32	0.30	0.29
35	0.60	0.56	0.53	0.51	0.49	0.48	0.47	0.46	0.45	0.44	0.43	0.42	0.41	0.40	0.37	0.35	0.34	0.33	0.31	0.30
40	0.61	0.56	0.53	0.51	0.49	0.48	0.47	0.46	0.45	0.44	0.43	0.42	0.41	0.40	0.39	0.36	0.35	0.34	0.32	0.31
45	0.61	0.56	0.53	0.51	0.49	0.48	0.47	0.46	0.45	0.44	0.43	0.43	0.41	0.42	0.39	0.37	0.36	0.35	0.33	0.32
50	0.61	0.56	0.53	0.51	0.50	0.48	0.47	0.47	0.46	0.44	0.44	0.42	0.42	0.41	0.39	0.37	0.36	0.35	0.34	0.32
60	0.60	0.56	0.53	0.51	0.50	0.49	0.48	0.47	0.46	0.45	0.44	0.43	0.42	0.42	0.40	0.38	0.37	0.36	0.35	0.33
70	0.61	0.56	0.53	0.52	0.50	0.49	0.47	0.47	0.46	0.45	0.44	0.44	0.43	0.42	0.40	0.38	0.37	0.36	0.35	0.34
80	0.61	0.56	0.54	0.52	0.50	0.49	0.48	0.47	0.47	0.45	0.44	0.44	0.43	0.42	0.40	0.39	0.38	0.37	0.36	0.35
90	0.61	0.56	0.53	0.52	0.50	0.49	0.48	0.47	0.46	0.46	0.44	0.44	0.43	0.43	0.41	0.39	0.38	0.37	0.36	0.35
100	0.61	0.56	0.54	0.52	0.50	0.49	0.48	0.48	0.47	0.45	0.45	0.44	0.43	0.43	0.41	0.40	0.38	0.38	0.36	0.35
150	0.61	0.56	0.54	0.52	0.50	0.49	0.49	0.48	0.47	0.46	0.45	0.45	0.44	0.44	0.41	0.40	0.39	0.38	0.37	0.37
200	0.61	0.56	0.54	0.52	0.50	0.49	0.49	0.48	0.47	0.46	0.46	0.44	0.45	0.44	0.42	0.41	0.40	0.39	0.38	0.37
250	0.61	0.57	0.53	0.52	0.51	0.49	0.48	0.48	0.48	0.46	0.46	0.45	0.45	0.44	0.42	0.41	0.40	0.39	0.38	0.37
300	0.61	0.56	0.53	0.52	0.51	0.49	0.49	0.48	0.48	0.46	0.46	0.45	0.44	0.44	0.42	0.41	0.40	0.40	0.39	0.38
400	0.61	0.56	0.54	0.52	0.51	0.49	0.49	0.48	0.47	0.46	0.46	0.45	0.44	0.44	0.43	0.41	0.41	0.40	0.39	0.38
500	0.61	0.56	0.53	0.52	0.50	0.49	0.48	0.48	0.47	0.46	0.45	0.45	0.45	0.44	0.43	0.41	0.41	0.40	0.39	0.38

n₀ = number of controls; n₁ = number of cases.

Estimates obtained by 10⁵ random samples.

3 Estimates of statistical power of rzAUC

Statistical power of the approximately normal test for rzAUC, shown in equation (6), was estimated under some simulated distributions and compared with the standard asymptotic normal test for AUC in equation (1). The selected distributions corresponded both to the binormal model with equal variances (Figure 1(b)) and to the distributions illustrated in Figure 3(a) to (d), namely: (a) normal–binormal, (b) binormal with different variances, (c) binormal–binormal and (d) zero-inflated binormal. In each analysis, statistical power was estimated as the proportion of positive results (i.e. tests called statistically significant) on a total of 1000 simulated distributions. Moreover, any test was also repeated using 2000 random permutations of the samples in the two classes in order to assess the impact of the non-perfect fit of rzAUC to the normal distribution. Results are resumed in Table 3.

Table 3.

Proportion of positive results (%) for test on rzAUC and AUC under different TMs distribution assumption

TM distribution	Δμ = 0.0			Δμ = 0.5			Δμ = 1.0			Δμ = 1.5			Δμ = 2.0
	AUC	rzAUC		AUC	rzAUC		AUC	rzAUC		AUC	rzAUC		AUC	rzAUC
	Asy	Asy	Per	Asy	Asy	Per	Asy	Asy	Per	Asy	Asy	Per	Asy	Asy	Per
n₀ = 10, n₁ = 10
Binormal – equal variance	5.0*	3.8*	5.2*	25.9	13.2	17.1	67.1	41.4	48.8	94.0	76.6	80.5	99.4	93.6	95.3
Binormal – unequal variance	5.4*	8.7	12.3	23.3	21.0	27.5	50.9	41.9	50.3	79.9	67.1	74.1	95.4	89.0	91.6
Normal–binormal	5.1*	3.7*	5.8*	14.5	7.5	11.7	25.7	15.5	21.7	39.2	26.1	33.2	51.8	45.8	56.5
Binormal–binormal	4.2*	2.6*	3.6*	8.1	6.6	11.1	13.8	13.5	21.5	21.6	29.0	38.6	25.4	43.4	57.0
Zero-inflated binormal	0.0*	0.4*	3.0*	0.1	2.0	10.1	0.0	5.4	22.8	0.2	14.0	36.4	0.2	29.4	59.1
n₀ = 15, n₁ = 15
Binormal – equal variance	5.0*	4.0*	5.7*	39.3	20.8	24.5	81.8	59.8	63.5	98.8	92.2	93.5	100	99.8	100
Binormal – unequal variance	5.4*	16.0	19.2	27.6	31.8	38.9	65.0	61.0	66.4	93.7	88.4	90.8	99.2	98.1	98.7
Normal–binormal	3.5*	2.9	4.4	12.0	8.4	10.7	27.0	19.1	23.3	46.8	36.4	41.5	60.2	61.0	67.1
Binormal–binormal	3.1*	3.9*	5.4*	7.2	9.7	13.1	15.8	22.1	29.7	27.3	43.1	49.9	36.6	65.7	73.7
Zero-inflated binormal	0.0*	0.8*	3.7*	0.1	3.8	11.8	0.2	14.0	30.4	0.4	31.8	54.7	0.6	57.8	79.9
n₀ = 20, n₁ = 20
Binormal – equal variance	4.3*	3.4*	4.5*	44.0	23.7	26.8	92.5	75.0	78.2	99.9	98.1	98.4	100	100	100
Binormal – unequal variance	5.7*	16.2	19.5	34.1	39.8	45.0	78.9	75.4	77.6	97.3	94.8	96.3	100	99.6	99.6
Normal–binormal	2.7*	4.3*	4.1*	20.0	12.8	15.0	41.3	31.1	34.7	65.8	55.9	60.5	80.7	80.7	84.2
Binormal–binormal	3.4*	3.4*	4.3*	13.3	11.6	13.2	25.2	33.5	38.4	42.1	65.0	69.0	54.1	86.1	88.2
Zero-inflated binormal	0.1*	1.6*	4.7*	0.0	7.4	15.9	2.4	25.5	41.3	6.1	54.4	71.3	10.3	82.5	92.2
n₀ = 30, n₁ = 30
Binormal – equal variance	5.3*	4.6*	4.6*	58.9	38.6	38.4	97.8	91.4	90.8	100	99.9	99.8	100	100	100
Binormal – unequal variance	4.7*	26.8	26.6	47.1	63.4	63.0	90.8	91.7	91.8	99.8	99.5	99.5	100	100	100
Normal–binormal	2.7*	4.3*	4.1*	25.4	19.1	18.6	51.9	45.4	44.7	80.7	77.9	77.2	94.0	95.6	95.2
Binormal–binormal	2.8*	4.9*	4.5*	15.8	19.2	18.6	34.6	51.9	51.6	64.3	86.2	86.0	77.3	99.1	98.9
Zero-inflated binormal	0.0*	2.7	4.6*	0.7	12.4	18.9	6.3	42.5	53.2	26.0	80.4	86.3	52.9	95.9	97.5
n₀ = 50, n₁ = 50
Binormal – equal variance	4.8*	4.7*	4.9*	77.2	49.7	50.4	100	98.8	98.8	100	100	100	100	100	100
Binormal – unequal variance	6.9*	40.9	40.9	63.9	78.6	79.5	98.6	98.8	98.8	100	100	100	100	100	100
Normal–binormal	5.4*	4.3*	4.9*	35.2	23.1	24.2	76.7	61.6	62.7	95.1	92.0	92.5	99.4	99.7	99.7
Binormal–binormal	2.8*	3.9*	3.9*	23.9	27.8	28.9	61.0	73.6	74.4	84.0	97.4	97.6	96.8	100	100
Zero-inflated binormal	0.3*	2.1*	5.8*	3.1	18.7	26.2	27.8	62.6	72.2	75.9	96.8	98.6	97.1	100	100

TMs: tumour markers.

Asy = asymptotic normal test; Per = permutation test (based on 2000 random permutations); n₀ = number of controls; n₁ = number of cases; and *H0 is true.

Δμ: expected difference in TM means between cases and controls.

3.1 Binormal model with equal variances

In the case of binormal model with equal variances, when the means in the two groups were equal (difference between the two means, Δμ = 0) the proportion of positive results represented an estimate of the test bias under H0, whereas in the presence of a positive Δμ value, such a proportion provided an estimate of the corresponding statistical power. Both asymptotic and permutation tests on rzAUC showed a rather good agreement with the expected α-value (0.05) under H0 and with the corresponding results from the standard analysis on AUC, even if the asymptotic test on rzAUC was slightly biased in the presence of small sample size (Table 3). Under H1 (Δμ from 0.5 to 2.0) a good agreement between the asymptotic and the permutation analysis was observed for both statistics at any sample size, except for a small loss of power for the asymptotic test for rzAUC, which tended to decrease when increasing Δμ. The test for AUC showed the highest statistical power, but the difference between the two tests tended to disappear with increasing both the sample size and Δμ.

3.2 Binormal model with different variances

The estimate of statistical power of the two tests under a binormal model with different variances was obtained setting to 1.0 the variance among controls and to 2.0 that among the cases. Results from permutation and asymptotic methods were in a rather good agreement for both tests in any analysis, except for a small loss of statistical power in the permutation test for rzAUC either at sample size ≤20 or when the difference between means (Δμ) was lower than 1.5 (Table 3). The test for rzAUC showed a higher statistical power than that for AUC at Δμ = 0.5, while for larger Δμ values the two tests had a comparable performance. At Δμ = 0, the proportion of positive results was higher for the test for rzAUC (both from permutation than asymptotic analysis) than that for AUC. However, it should be noted that in this situation results for the new proposed test do not estimate the test bias under H0, because the corresponding ROC curve crosses the chance line.

3.3 Normal-binormal model

Simulations for the normal–binormal model (corresponding to ROC curves like that in Figure 3(c)) were carried out setting to 1.0 the variance for any distribution (i.e. two normal distributions among the cases and one normal among the controls). Mean values were set to zero for both the first subgroup of cases and the class of controls, while it ranged from 0.0 to 2.0 for the second subgroup of cases. A sampling ratio of 2:1 was adopted for the two subgroups of cases (the largest one was that with the same distribution of the controls). In this analysis, Δμ represents the difference between the means between the two groups of cases or (equivalently) between the second group of cases and the controls.

Permutation test showed a higher statistical power than the asymptotic test, but this difference tended to disappear with increasing the sample size (Table 3).

3.4 Binormal–binormal model

Under the binormal–binormal model, in the two subgroup of cases and controls with the same mean, variance was 1.0, the mean value was 0.0 and the sample size was the same. In the remaining two subgroups of controls and cases, mean values were 1.0 for the controls and were left to vary from 1.0 to 3.0 in the subgroup of cases (in Table 3 such differences were expressed as distances Δμ between such subgroups, which accordingly varied from 0.0 to 2.0). Variance and sample size were equal to those of the former two subgroups.

In the presence of small sample size, the normal test for rzAUC showed a loss of statistical power compared with the permutation approach. Tests for rzAUC outperformed that for AUC, with differences becoming more pronounced when increasing Δμ values.

3.5 Zero-inflated binormal model

For the zero-inflated binormal model the same sample size was adopted for the two subgroups of cases and controls with mass at zero and for the remaining two subgroups. For these latter, a binormal model with equal variances (σ² = 1.0) was used, with a mean value of 1.0 in the subgroup of controls and a mean value varying from 1.0 to 3.0 in the subgroup of cases (corresponding to Δμ from 0.0 to 2.0 in Table 3). Negative values extracted from the normal distributions were replaced with zero. At Δμ = 0.0, the test for AUC showed a proportion of false positive tests close to 0. The asymptotic test for rzAUC was also biased, especially for low sample size, while for the permutation test the false positive proportion was quite similar to the selected α-value (0.05).

The permutation approach outperformed the asymptotic test for rzAUC in any analysis. Both tests for rzAUC showed a higher statistical power than that for AUC.

In summary, results of power analysis reported in Table 3 suggest that, in most cases the asymptotic test for rzAUC seems to be appropriated when at least 20 samples are present in each class, whereas for smaller sample sizes the permutation test should be preferred.

4 Choosing between complete and rROC curves

In order to find a convenient rule to choose between restricted and standard ROC analysis we analysed the following three situations: (a) only a rROC analysis was performed and the test associated with the rAUC (equation (3)) was statistically significant (first column in Table 4); (b) the p-value associated to a rROC (equation (3)) was statistically significant and also lower than the corresponding p-value associated to the entire curve (equation (1), second column in Table 4); (c) the previous rule was respected and the mean difference percent (MDP) between rAUC and AUC was higher of some selected thresholds (third column in Table 4), where MDP was defined as

\overset{\land}{MDP} = \frac{\overset{\land}{rAUC} - \overset{\land}{AUC}}{\overset{\land}{AUC}} \cdot 100

(7)

Table 4.

Distribution of the number of samples j excluded by rROC analysis at the conventional 0.05 α level for not informative (Δμ = 0) and for proper ROC curves (Δμ > 0)

Samples		(a) rROC analysis only						(b) Standard and rROC analysis (any MDP)						(c) Standard and rROC analysis (MDP > 20%)
Samples		Percentiles					%	Percentiles					%	Percentiles					%
n ₀	n ₁	10	25	50	75	90	%	10	25	50	75	90	%	10	25	50	75	90	%
Δμ = 0.0
10	10	0	2	5	8	11	5.2	3	4	6	9	11	3.3	2	3	5	8	9	2.8
15	15	1	5	12	15	20	5.7	4	9	13	17	20	4.7	7	11	14	18	20	4.2
20	20	5	9	18	23	27	3.4	7	16	20	25	28	2.6	13	16	20	25	28	2.5
30	30	3	11	25	38	43	4.6	9	18	29	39	44	3.6	17	23	31	40	44	3.1
50	50	7	26	55	71	85	4.7	26	45	63	73	87	3.8	32	52	64	75	87	3.5
Δμ = 0.5
10	10	0	0	2	6	9	17.1	1	2	5	7	10	9.3	3	4	7	9	10	6.4
15	15	0	0	3	10	16	24.5	2	4	10	15	18	10.1	9	11	14	17	19	6.2
20	20	0	0	3	12	20	23.7	5	9	14	19	25	6.3	9	12	17	23	25	4.7
30	30	0	0	6	21	35	38.6	5	11	20	34	41	12.0	16	22	29	39	43	7.2
50	50	0	0	5	22	55	49.7	6	13	31	55	73	10.2	41	48	57	73	79	4.7
Δμ = 1.0
10	10	0	0	0	3	8	48.8	1	2	4	8	10	12.8	5	6	8	9	11	5.8
15	15	0	0	0	3	8	63.5	1	3	5	9	16	11.7	6	8	11	16	17	4.5
20	20	0	0	0	3	11	75.0	1	5	11	17	21	6.7	9	12	18	20	25	3.2
30	30	0	0	0	3	13	91.4	1	3	8	15	31	10.1	13	22	33	37	47	1.7
50	50	0	0	0	1	6	98.8	1	2	4	9	21	14.0	43	50	54	63	70	0.5
Δμ = 1.5
10	10	0	0	0	0	3	80.5	1	2	3	5	7	9.1	3	6	7	7	9	1.8
15	15	0	0	0	0	4	93.5	1	2	4	7	11	9.2	11	11	13	13	14	1.1
20	20	0	0	0	3	5	98.1	1	1	4	6	9	6.0	23	23	23	23	23	0.1
30	30	0	0	0	0	2	99.9	1	2	4	6	11	6.8	n.e.	n.e.	n.e.	n.e.	n.e.	0.0
50	50	0	0	0	0	1	100	1	2	3	4	7	9.9	n.e.	n.e.	n.e.	n.e.	n.e.	0.0
Δμ = 2.0
10	10	0	0	0	0	0	95.3	1	2	3	4	5	5.5	3	4	8	9	11	0.6
15	15	0	0	0	0	0	100	1	2	3	4	7	5.7	n.e.	n.e.	n.e.	n.e.	n.e.	0.0
20	20	0	0	0	0	0	100	2	2	4	5	8	2.5	n.e.	n.e.	n.e.	n.e.	n.e.	0.0
30	30	0	0	0	0	0	100	2	2	4	4	7	3.1	n.e.	n.e.	n.e.	n.e.	n.e.	0.0
50	50	0	0	0	0	0	100	3	3	4	5	7	2.4	n.e.	n.e.	n.e.	n.e.	n.e.	0.0

rROC: restricted receiver operating characteristic.

n₀ = number of controls; n₁ = number of cases; % = percentages of rROC curves selected; MDP = mean difference percent between rAUC and AUC; and n.e. = not evaluable.

Δμ = expected difference in TM means between cases and controls.

Simulations included only proper ROC curves, because the standard ROC analysis approach should be preferred in the case of a proper ROC curve, due to the highest statistical power of the test for AUC. Moreover, the case of non-informative ROC curve was also considered. The distribution of j-values was obtained from 1000 simulated TMs. The corresponding statistical significance of rzAUC was estimated by the test based on the normal approximation (equation (6)), except when the sample size was lower than 20 to prevent the loss of statistical power. In that case, a test based on 2000 random permutations was employed.

Table 4 reports the percentages of rROC curves that would have been selected using the above cited three criteria and some corresponding percentiles of the j statistic, which corresponds to the number of samples excluded from the curve. The first column corresponds to the results of rROC analysis only (case a). The number of excluded samples was about 5% under H0 and tended to increase with increasing Δμ, while the corresponding median value of j tended to decrease. In particular, for Δμ ≥ 1.0 the median value of j was 0 in any comparison and for Δμ ≥ 1.5 also the 75° percentile equals 0 in all comparisons except one, indicating that the rROC analysis tends to provide results very similar to those from the standard ROC approach when the underline TM distributions are well separated.

The second column in Table 4 (case b) shows the results of the combination of restricted and standard ROC analysis based on the comparison between the corresponding p-values. The proportion of selected rROC was slightly lower than those observed by the rROC only (first column) at Δμ = 0.0, while the median value of j was slightly higher. Proportion of errors tended to increase with increasing Δμ up to Δμ = 1.0 and to decrease subsequently. Finally, when only rROC curves corresponding to DMP >20% were selected (case c, third column in Table 4) the proportion of errors clearly decreased ranging between 4% and 7% for Δμ ≤ 1.0 and dropping to 0–2% for Δμ > 1.5, while the corresponding j-values decreased accordingly.

Taken together, these results suggest that in the presence of a statistically significant test for the rROC curve and a MDP > 20% standard ROC analysis may be replaced by rROC ones with an acceptable proportion of errors (about 5% or less).

5 Combining information from restricted and standard ROC curves

The main limit of the rROC analysis is the exclusion from the ROC curve of the j subjects whose TM values are lower than C_j, the cut-off that separates informative from non-informative values (as illustrated in Section 2). To recover this drawback, information from two or more TMs can be combined. A new approach will be illustrated, which extends the traditional method of applying a logistic regression model to two or more marker values in order to obtain a risk score (RS) for classification purposes.¹⁹ RS is usually employed as a new TM in a standard ROC analysis and a binary test is identified from a corresponding suitable cut-off.

In a standard approach, a logistic regression model with main effects can be employed, and RS estimated as a weighted sum of the TMs value (TM_i), by adopting as weights the corresponding regression coefficients β_k

{\overset{\land}{RS}}_{i} = \sum_{k = 1}^{p} {\overset{\land}{β}}_{k} {TM}_{ki}

(8)

where i represents the statistical units (i.e. the individuals) and p the number of predictors (i.e. the TMs).

In the presence of rROC curves, the corresponding markers can be introduced in the regression model using a nested approach

{\overset{\land}{RS}}_{i} = \sum_{k = 1}^{p} {\overset{\land}{β}}_{1, k} D_{ki} + {\overset{\land}{β}}_{2, k} N_{ki}

(9)

where D_ki is a dummy variable, which takes the value 0 when TM_i < C_j and the value 1 otherwise, while N_ki is the TM nested variable, obtained by multiplying the TM_i values by D_ki. Using this approach each non-informative value will do not contribute to RS_i, because both D_ki and N_ki will take the value 0. Conversely, in the presence of a proper ROC curve all TM values will be considered as potentially informative. In this case, equation (9) will reduce to equation (8). As a consequence, adopting the above described nested modelling approach, information from TMs corresponding to both proper and rROC curves can be easily combined.

In the further section an application of the proposed method will be illustrated using a real data set (Section 6.3).

6 Application of the new method to a real data set

6.1 Description of the data set

The new method of ROC curve analysis based on rROC was applied to a real data set that collected data of peripheral blood concentrations of seven putative TMs from 15 patients affected by neuroblastoma and from 20 healthy controls, namely: tyrosine hydroxylase (TH), beta-1,4-N-acetyl-galactosaminyl transferase 1 (GD2), dopamine-decarboxylase (DDC), doublecortin (DCX), embryonic lethal, abnormal vision-4 (ELAV-4), sialyltranferase ST8SiaII (STX) and paired-like homeobox 2b (Phox2b).²⁰ Neuroblastoma is the most frequent extra cranial solid tumour of childhood, presenting in localized (stages 1, 2 and 3) and metastatic (stages 4 and 4S) forms.²¹ Since neuroblastoma originates from neuronal cell precursors, the presence of neuronal specific RNAs in peripheral blood samples have been proposed as a measure of tumour cell contamination. However, low expression of neuronal RNAs from peripheral blood cells have been documented also in samples from healthy subjects.²² Patients with localized neuroblastoma are supposed to have little, if any, tumour cells in blood, but their presence may have important prognostic value.²¹ Therefore, it is of the outmost importance to identify the TMs able to discriminate patients with tumour infiltration in the blood from those without.

The TM concentration values were obtained by means of RT–qPCR analysis of total RNA extracted from 2 mL of peripheral blood samples using the same procedure for both cases and controls.²⁰ Samples were considered negative if the three quantification cycle (Cq) values (i.e. the cycle at which amplification overcomes the limit of detection) were equal to 40. Samples were considered positive if the three Cq values were lower than 40, and positive results were expressed as relative values (ΔΔCt method²³), using β2-microglobulin as endogenous reference RNA, and a neuroblastoma cell line as the exogenous reference sample. Table 5 reports the concentration values measured in the study subjects together with patient and control characteristics. Phox2B data were excluded due to the presence of eight missing values (four in the class of controls and four among the cases).

Table 5.

Concentration values^a of six putative neuroblastoma TMs measured in the study subjects

ID	Sex	Stage	TH	GD2	DDC	STX	DCX	Elav
Neuroblastoma patients (cases)
T01	F	3	3.3 × 10⁻⁶	3.6 × 10⁻⁶	2.5 × 10⁻⁵	1.7 × 10⁻⁷	4.6 × 10⁻⁷	8.6 × 10⁻⁶
T02	F	1	3.7 × 10⁻⁵	1.7 × 10⁻⁵	5.0 × 10⁻⁶	0	3.7 × 10⁻⁶	2.3 × 10⁻⁶
T03	F	1	4.3 × 10⁻⁵	1.6 × 10⁻⁴	2.4 × 10⁻⁶	1.4 × 10⁻⁷	0	1.6 × 10⁻⁶
T04	F	1	0	2.0 × 10⁻⁶	5.2 × 10⁻⁶	0	3.7 × 10⁻⁷	2.9 × 10⁻⁶
T05	F	3	1.3 × 10⁻²	0	2.9 × 10⁻⁴	9.7 × 10⁻⁷	2.9 × 10⁻⁶	2.2 × 10⁻⁵
T06	M	1	0	2.9 × 10⁻⁶	0	0	0	1.0 × 10⁻⁵
T07	M	3	3.0 × 10⁻²	3.9 × 10⁻⁷	1.9 × 10⁻⁶	0	0	6.6 × 10⁻⁶
T08	M	3	5.2 × 10⁻⁵	6.5 × 10⁻⁶	0.45	1.3 × 10⁻⁵	0.41	1.7 × 10⁻²
T09	F	1	4.4 × 10⁻⁴	1.5 × 10⁻⁵	0	5.1 × 10⁻⁷	0	2.4 × 10⁻⁵
T10	M	1	0	6.4 × 10⁻⁴	3.8 × 10⁻⁵	0	7.5 × 10⁻⁶	3.0 × 10⁻⁵
T11	F	3	0	8.2 × 10⁻²	0	3.1 × 10⁻⁷	0	8.4 × 10⁻⁷
T12	M	1	0	1.6 × 10⁻⁵	0.14	0	0	3.5 × 10⁻⁴
T13	F	3	1.6 × 10⁻⁴	3.9 × 10⁻⁷	2.7 × 10⁻⁵	1.9 × 10⁻⁶	3.2 × 10⁻⁷	9.0 × 10⁻⁶
T14	M	1	3.1 × 10⁻⁵	3.2 × 10⁻³	1.7 × 10⁻⁶	2.1 × 10⁻⁷	0	1.2 × 10⁻⁵
T15	F	3	4.3 × 10⁻⁴	1.1 × 10⁻⁴	6.5 × 10⁻³	8.3 × 10⁻⁵	4.4 × 10⁻⁴	1.3 × 10⁻⁴
Healthy individuals (controls)
C01	F	n.a.	1.4 × 10⁻⁶	3.7 × 10⁻⁶	0	0	2.2 × 10⁻⁷	3.6 × 10⁻⁶
C02	M	n.a.	0	2.2 × 10⁻⁶	1.8 × 10⁻⁶	1.1 × 10⁻⁷	0	4.2 × 10⁻⁶
C03	M	n.a.	3.7 × 10⁻⁶	4.9 × 10⁻⁶	0	7.0 × 10⁻⁸	1.3 × 10⁻⁷	4.7 × 10⁻⁶
C04	M	n.a.	4.0 × 10⁻⁷	2.3 × 10⁻⁶	0	0	1.9 × 10⁻⁷	4.8 × 10⁻⁶
C05	M	n.a.	1.3 × 10⁻⁵	0	0	1.0 × 10⁻⁷	1.7 × 10⁻⁷	5.8 × 10⁻⁶
C06	M	n.a.	2.8 × 10⁻⁵	1.5 × 10⁻⁶	1.6 × 10⁻⁶	0	0	1.5 × 10⁻⁶
C07	M	n.a.	4.5 × 10⁻⁷	1.9 × 10⁻⁶	0	0	0	7.4 × 10⁻⁶
C08	F	n.a.	4.0 × 10⁻⁷	2.3 × 10⁻⁶	0	0	2.0 × 10⁻⁷	3.1 × 10⁻⁶
C09	M	n.a.	0	6.5 × 10⁻⁶	1.4 × 10⁻⁶	1.4 × 10⁻⁶	1.5 × 10⁻⁷	4.7 × 10⁻⁶
C10	M	n.a.	3.9 × 10⁻⁶	7.6 × 10⁻⁷	6.8 × 10⁻⁶	1.3 × 10⁻⁷	1.6 × 10⁻⁷	4.1 × 10⁻⁶
C11	F	n.a.	9.4 × 10⁻⁷	10.8 × 10⁻⁶	1.5 × 10⁻⁶	4.0 × 10⁻⁸	1.5 × 10⁻⁷	4.3 × 10⁻⁶
C12	M	n.a.	4.0 × 10⁻⁷	3.1 × 10⁻⁶	0	8.0 × 10⁻⁸	1.6 × 10⁻⁷	4.3 × 10⁻⁶
C13	M	n.a.	1.0 × 10⁻⁶	2.1 × 10⁻⁶	0	0	0	8.4 × 10⁻⁶
C14	F	n.a.	4.7 × 10⁻⁶	1.1 × 10⁻⁶	2.2 × 10⁻⁶	0	2.6 × 10⁻⁷	2.4 × 10⁻⁶
C15	M	n.a.	6.3 × 10⁻⁷	3.0 × 10⁻⁷	0	6.04 × 10⁻⁸	0	4.5 × 10⁻⁶
C16	F	n.a.	0	1.8 × 10⁻⁶	0	8.24 × 10⁻⁷	0	3.4 × 10⁻⁶
C17	F	n.a.	0	1.8 × 10⁻⁶	0	9.04 × 10⁻⁸	1.3 × 10⁻⁷	7.1 × 10⁻⁶
C18	M	n.a.	5.3 × 10⁻⁷	6.9 × 10⁻⁶	0	0	0	3.3 × 10⁻⁶
C19	M	n.a.	2.8 × 10⁻⁷	1.4 × 10⁻⁶	4.2 × 10⁻⁶	7.54 × 10⁻⁷	0	7.5 × 10⁻⁶
C20	F	n.a.	5.0 × 10⁻⁷	6.0 × 10⁻⁷	1.6 × 10⁻⁶	0	1.8 × 10⁻⁷	3.1 × 10⁻⁶

TH: tyrosine hydroxylase; GD2: beta-1,4-N-acetyl-galactosaminyl transferase 1; DDC: dopamine-decarboxylase; STX: sialyltranferase ST8SiaII; DCX: doublecortin; and Elav: embryonic lethal, abnormal vision.

Values were obtained by applying the ΔΔCq formula to the readout of real-time PCR equipment (see text for details).

n.a. = non-applicable.

6.2 Comparison between standard ROC and rROC analysis

Figure 4 (panels (a) to (f)) shows the ROC curves corresponding to the TM values in Table 5. In each panel, results of both standard ROC analysis (AUC, zAUC) and rROC analysis (rAUC and rzAUC) are reported with the corresponding p-values obtained by the normal tests (pNorm) and by 2000 random permutations (pPermut). Moreover, in each plot the cut-off of highest accuracy J and the cut-off C, which identifies the rROC according to equation (5), are also reported with the corresponding 95% confidence intervals (in brackets) estimated by the percentiles of the bootstrapped distribution obtained from 5000 bootstrapped samples.²⁴

All the considered markers corresponded to a non-proper positive skewed ROC curve. In particular, TH, DDC, STX and DCX (Figure 4, panels (a), (c), (d) and (e)) showed a behaviour similar to that of the theoretical ROC curve in Figure 2(d), due to the presence of numerous null values in the TM distribution (Table 5). The shape of the curve corresponding to Elav (panel (f)) is due to the presence of a range of small positive values of marker concentration, similarly distributed among the two classes under study and it is consistent with a behaviour like those described by the ROC curves in Figure 2, panels (b) and (c). Finally, the ROC curve for GD2 (panel (b)) showed an intermediate behaviour, due to the presence of a range of small non-informative TM concentrations that also included some zero values. Results from permutation tests were consistent with those from normal tests for both standard ROC and rROC analysis. Using a conventional 0.05 level of statistical significance (one sided test), STX and DCX would have been rejected as TM by the standard ROC analysis (AUC = 0.64, pNorm = 0.086 for STX and AUC = 0.63, pNorm = 0.103 for DCX, respectively), whereas the test for rROC was statistically significant in both cases (pNorm = 0.045 and pNorm < 0.001, respectively). For the remaining curves both approaches provided evidence of a possible application of the corresponding TM for the diagnosis of localized neuroblastoma. Interestingly, the two approaches provided the same result (AUC = rAUC) for DDC, whose ROC curve (Figure 11S, panel (c)) was rather similar to a proper ROC curve (Figure 1, panel (a)). All TMs, except DDC, showed a MDP > 20%, suggesting that the application of the rROC analysis may be appropriated (Section 4).

Table 6 resumes the results of both standard ROC and rROC analysis reported in Figure 4; TMs were sorted in an ascending order on the basis of rzAUC values. The first two markers identified by rROC analysis (Elav and TH) ranked third and fourth, respectively, when the standard approach of analysis was adopted, while the TM with the highest AUC (DDC) ranked only fifth when analysed by the rROC method.

Table 6.

Results of standard ROC and rROC analysis applied to data in Table 5 and ROC curves in Figure 4

TM	rAUC	rzAUC	Rank	N_excl (N_cases, N_contr)	AUC	Rank
Elav	0.98	4.27	1	6 (4,2)	0.73	3
TH	0.97	3.95	2	9 (5,4)	0.68	4
GD2	0.91	3.77	3	5 (3,2)	0.75	2
DCX	1.00	3.70	4	15 (7,8)	0.63	6
DDC	0.82	3.23	5	0 (0,0)	0.82	1
STX	0.84	2.55	6	15 (6,9)	0.64	5

rROC: restricted receiver operating characteristic; TM: tumour marker; TH: tyrosine hydroxylase; GD2: beta-1,4-N-acetyl-galactosaminyl transferase 1; DDC: dopamine-decarboxylase; STX: sialyltranferase ST8SiaII; DCX: doublecortin; and Elav: embryonic lethal, abnormal vision.

N_excl = total number of values excluded from rROC; N_cases = number of values among cases excluded from the rROC; and N_contr = number of values among controls excluded from the rROC.

6.3 Identification of a classification rule based on the RS method

A classification rule was identified for the combination of couple of TMs using the approach described in the previous section (equation (9)). Results of this analysis are reported in Table 7 that also reports results of leave-one-out cross-validation performed on each couple of TMs. A global validation was also carried out, selecting at each step the best combination of markers based on the highest AUC.¹⁹

Table 7.

Results of the classification using a RS obtained by the combination of couple of the tumour markers in Table 3 by either a standard approach or a nested model based on the rROC analysis

TMs	Standard approach (main effect model)			rROC approach (nested model)
TMs	Se%	Sp%	Acc%	Se%	Sp%	Acc%
Whole data set
TH, GD2	80.0	100	91.4	93.3	90.0	91.4
TH, DDC	80.0	100	91.4	86.7	100	94.3
TH, STX	60.0	100	83.9	86.7	95.0	91.4
TH, DCX	73.3	95.0	85.7	86.7	95.0	91.4
TH, Elav	86.7	90.0	88.6	100	100	100
GD2, DDC	86.7	95.0	91.4	93.3	95.0	94.3
GD2, STX	66.7	95.0	82.9	80.0	100	91.4
GD2, DCX	80.0	100	91.4	100	90	94.3*
GD2, Elav	86.7	95.0	91.4	100	100	100
DDC, STX	73.3	85.0	80.0	73.3	85.0	80.0
DDC, DCX	60.0	95.0	80.0	80.0	90.0	85.7
DDC, Elav	86.7	90.0	88.6	86.7	100	94.3
STX, DCX	53.3	90.0	74.3	100	70.0	82.9
STX, Elav	66.7	100	85.7	93.3	90.0	91.4
DCX, Elav	73.3	100	88.6	93.3	95.0	94.3*
Verification set in leave-one-out cross-validation
TH, GD2	80.0	100	91.4	80.0	85.0	82.9
TH, DDC	73.3	90.0	82.9	73.3	95.0	85.7
TH, STX	46.7	95.0	74.3	46.7	80.0	65.7
TH, DCX	66.7	85.0	77.1	66.7	85.0	77.1
TH, Elav	73.3	85.0	80.0	86.7	90.0	88.6
GD2, DDC	80.0	90.0	85.7	80.0	90.0	85.7
GD2, STX	60.0	95.0	80.0	60.0	90.0	77.1
GD2, DCX	40.0	95.0	71.4	40.0	95.0	71.4
GD2, Elav	73.3	95.0	85.7	73.3	95.0	85.7
DDC, STX	60.0	80.0	71.4	60.0	75.0	68.6
DDC, DCX	46.7	85.0	68.6	60.0	85.0	74.3
DDC, Elav	53.3	95.0	77.1	73.3	80.0	77.1
STX, DCX	33.3	85.0	62.9	66.7	70.0	68.6
STX, Elav	53.3	85.0	71.4	73.3	80.0	77.1
DCX, Elav	66.7	100	85.7	86.7	90.0	88.6
Global validation	73.3	95.0	85.7	86.7	90.0	88.6

RS: risk score; rROC: restricted receiver operating characteristic; TMs: tumour markers; TH: tyrosine hydroxylase; GD2: beta-1,4-N-acetyl-galactosaminyl transferase 1; DDC: dopamine-decarboxylase; STX: sialyltranferase ST8SiaII; DCX: doublecortin; and Elav: embryonic lethal, abnormal vision.

Legend: Se = sensitivity; Sp = specificity; Acc = diagnostic accuracy; Global validation: the best couple of markers (based on the highest AUC for the RS) selected at each step from the whole data set.

Convergence was not achieved.

The new method based on rROC analysis and regression models with nested variables clearly outperformed the standard one showing an equal to higher accuracy in each comparison. However, in two cases (corresponding to the inclusion in the model of GD2 and DCX, and DCX and Elav, respectively) the convergence was not achieved.

In cross-validation the new method outperformed the standard one in five cases, had a lower accuracy in four and an equal accuracy in the remaining six comparisons. In the global cross-validation, the accuracy of the new method barely exceeded that of the standard one (88.6% vs 85.7%).

7 Discussion

rROC curves represent new simple tools for the analysis of TMs for diagnostic purposes. In this article a new statistical test, associated to the area under a rROC curve, has been proposed. In the presence of positive skewed ROC curves, the new test outperformed the standard approach both on simulated distributions and on a set of real TMs values. In particular, in the real data set two TMs, namely STX and DCX, would have been treated as non-informative by standard ROC analysis.

Standard measures of summary accuracy like AUC or the Youden’s index were demonstrated to be a quite simple, powerful and useful tools for diagnostic purposes in many medical fields.^2,3,8 However, when applied to not proper ROC curves, standard ROC analysis may be unable to extract most of the relevant information or it may even provide unreliable results.²⁵ In Clinical Epidemiology not proper ROC curves are often encountered, especially in the analysis of putative TMs. In particular, positive skewed ROCs may results from the presence of similar values at low TM concentrations in the two classes under study. Many reasons may be advocated for this behaviour, including the presence of a detection threshold in the measure device, the presence of a subgroup of cases with a TM expression similar to that of controls, and the presence among controls of individuals with a rather high expression of TM in the absence of neoplastic cells, as briefly illustrated in the Introduction and shown by some examples in Figures 2 and 3. With respect to the real data set analysed in this article,²⁰ the shape of the corresponding ROC curves reported in Figure 4 indicates that a mix of these different situations have occurred. In particular, zero-inflated distributions may have arisen from the detection threshold of the RT-qPCR method that is unable to perform more than 40 amplification cycles, while bimodal distributions among cases may be related to the fact that all the considered TMs can be expressed in some proportion also by normal cells as a consequence of the so called illegitimate transcription.²³ Furthermore, some patients with localized neuroblastoma, which have little tendency to spread to distant tissues, including blood, may have TMs concentration values similar to those of healthy individuals.

Information from the shape of a ROC curve seems to be at least as important as information from summary indices of accuracy like AUC. In recent years, modern ROC analysis has tried to develop new approaches to extract and combine all available information from a TM, exploring the ROC regions with informative values.²⁶ Probably the most important and reliable method is based on pAUC, which was demonstrated to be a useful tool for diagnostic purposes in the presence of asymmetric ROC plots.^13,17,26 The method based on rROC, described in this article, may be considered as an extension of the methods based on pAUC. In fact, the area under a rROC curve (rAUC) is equivalent to the pAUC identified by the cut-off C, divided by the corresponding 1-specificity value. However, there are two major differences with the pAUC based approaches, namely: (a) the cut-off that identifies rROC is selected by a recursive procedure coupled with an index of accuracy (rzAUCj, equation (3)) and (b) samples corresponding to the non-informative TM concentrations are considered as missing values and not as negative test. For this reason at least another diagnostic marker, if available, is needed in order to complete the classification analysis, thus allowing those subjects, whose TM levels lie in the range of ‘not informative values’ (TM concentrations <C), to be allocated in either one class under study. Using the real data set of neuroblastoma markers,²⁰ a combination of information from both standard and rROC analysis via a logistic regression model provided a classification rule based on the corresponding RS whose accuracy was higher than that of any other combination of couples of TMs. However, a more reliable estimate should have been obtained from an external validation set, which unfortunately was not available. The small sample size of the data set (20 controls and 15 cases) just allowed the application of the leave-one out cross-validation. Most results from this analysis indicated a better performance of the new proposed method, but accuracy of the global validation was only marginally higher than that of the standard analysis. Furthermore, the use of regression models with nested variables in some instances provided unstable estimates due to a lack of convergence even in the analysis of the whole data set, probably as a consequence of the small sample size. Finally, nested models may be more prone to overfitting than the corresponding main effect models due to the inclusion of a higher number of predictors. Further analyses, based on both extensive simulations and real databases of TMs with different sample sizes, are needed to estimate the accuracy and the precision of the new proposed method and to assess the advantage of using rROC analysis in classification based on RS from a regression model.

Another limitation of the proposed method is that the statistical properties of the new proposed test were explored only via extensive simulations, whereas an analytical formula for rzAUC distribution under H0 is not available. This limit might make difficult its application to very large data sets, such as multicentre follow up study, where the analysis by random permutations is rather time-consuming. Furthermore, only left restriction has been considered in this article in the presence of positive skewed ROC curves, whereas in clinical setting many other types of not proper curves may be encountered. In particular, applying a right restriction, the proposed method should be extended to negative skewed ROC, that may originate by a ceiling detection threshold in a diagnostic device or by the presence of some spiked values among the controls. However, the extension of the rROC method to include either type of restriction will require a very large number of simulations and the use of new databases of real data, and is behind the scope of this investigation. Finally, the method should also be extended to allow the inclusion of an utility function¹⁶ in the evaluation of optimal cut-offs, since false positive and false negative errors may have different costs in different situations. Nonetheless, results from this investigation indicate that rROC curves may be a new simple and useful tool to explore the diagnostic utility of putative TMs. In the coming future, new statistical tests should be developed to extract information from different kinds of TM distributions, possibly exploiting the properties of the corresponding rROC plots. The development of a comprehensive approach to combine any relevant information from both standard and modern ROC analysis will provide an optimal framework for the evaluation of the diagnostic potential of new TMs.

Footnotes

Acknowledgements

The authors thank Dr Paolo Bruzzi (National Cancer Research Institute of Genoa) for providing precious advice.

Funding

This study was partly supported by the Ligurian Region and by the Italian Neuroblastoma Foundation (Fondazione Italiana per la Lotta al Neuroblastoma). B.C. is a recipient of a grant from the Italian Neuroblastoma Foundation.

Declaration of conflicting interest

None declared.

References

Erdreich

. Use of relative operating characteristic analysis in epidemiology. Am J Epidemiol 1981; 114: 649–662.

Pepe

. The statistical evaluation of medical tests for classification and prediction, Oxford, UK: Oxford University Press, 2003.

Krzanowski

Hand

. ROC curves for continuous data, Boca Raton, FL: CRC Press, 2009.

Baker

. Improving the biomarker pipeline to develop and evaluate cancer screening tests. J Natl Cancer Inst 2009; 101: 1–4.

Bamber

. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychology 1975; 12: 387–415.

Mann

Whitney

. On a test whether one of two random variables is stochastically larger than another. Ann Math Stat 1947; 18: 50–60.

Hoeffding

. A class of statistics with asymptotically normal distribution. Ann Math Stat 1948; 19: 293–325.

Hanley

McNeil

. A method of comparing the area under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148: 839–843.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

10.

Rosner

Glynn

. Power and sample size estimation for the Wilcoxon rank sum test with application to comparisons of c statistics from alternative prediction models. Biometrics 2009; 65: 188–197.

11.

Baker

Kramer

. Peirce, Youden, and receiver operating characteristic curves. Am Statistician 2007; 61: 343–346.

12.

Perkins

Schisterman

. The inconsistency of “oprimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol 2006; 163: 670–675.

13.

Pepe

Longton

Anderson

. Selecting differentially expressed genes from microarray experiments. Biometrics 2003; 59: 133–142.

14.

Hanley

. The use of the ‘binormal’ model for parametric ROC analysis of quantitative diagnostic tests. Stat Med 1996; 15: 1575–1585.

15.

Schisterman

Reiser

Faraggi

. ROC analysis for markers with mass at zero. Stat Med 2006; 25: 623–638.

16.

Baker

. Identifying combinations of cancer markers for further study as triggers of early intervention. Biometrics 2000; 56: 1082–1087.

17.

Dodd

Pepe

. Partial AUC estimation and regression. Biometrics 2003; 59: 614–623.

18.

Sprott

. Numerical recipes software: numerical recipes: routine and examples in BASIC, New York, NY, USA: Cambridge University Press, 1998.

19.

McIntosh

Pepe

. Combining several screening tests: optimality of the risk score. Biometrics 2002; 58: 657–664.

20.

Corrias

Haupt

Carlini

. Multiple target molecular monitoring of bone marrow and peripheral blood samples from patients with localized neuroblastoma and healthy donors. Pediatr Blood Cancer 2012; 58(1): 43–49.

21.

Maris

. Recent advances in neuroblastoma. N Engl J Med 2010; 362: 2202–2211.

22.

Beiske

Burchill

Cheung

. Consensus criteria for sensitive detection of minimal neuroblastoma cells in bone marrow, blood and stem cell preparations by immunocytology and RT-QPCR: recommendations by the International Neuroblastoma Risk Group Task Force. Br J Cancer 2009; 100: 1627–1637.

23.

Livak

Schmittgen

. Analysis of relative gene expression data using real-time quantitative PCR and the 2(-Delta Delta C(T)) Method. Methods 2001; 25: 402–408.

24.

Davison

Hinkley

The basic bootstrap. In: Davison

Hinkley

(eds). Bootstrap methods and their application, New York: Cambridge University Press, 2006, pp. 11–69.

25.

Lee

Hsiao

. Alternative summary indices for the receiver operating characteristic curve. Epidemiology 1996; 7: 605–611.

26.

Kagaris D and Yiannoutsos CT. A multi-index ROC-based methodology for high throughput experiments in gene discovery. Int J Data Min Bioinform 2011; in press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.15 MB