Nonparametric inference for the localization receiver operating characteristic curve and its extension to free-response image localization tasks

Abstract

The localization receiver operating characteristic (LROC) and the free-response receiver operating characteristic (FROC) curves are popular methods for evaluating the performance of diagnostic tests concerning detecting and locating lesions. However, the existing LROC and FROC curves have some limitations in their definition and estimation methods. In this article, we extend the LROC curve to the free-response tasks where each case can be marked with multiple suspicious locations and propose a nonparametric estimation of the LROC curve to compensate for the limitations of existing LROC and FROC curves. We derive the asymptotic properties for our estimators and propose a bootstrap method for constructing confidence bands for the proposed LROC curve. Simulations are conducted to compare the proposed LROC curve with the existing AFROC curve and to examine the theoretical results. Finally, we apply our method to evaluate the performance of a diagnostic test investigating AI-assisted pulmonary nodule diagnosis.

Keywords

Localization tests ROC FROC LROC nonparametric methods

1. Introduction

For AI or human readers that both detect and localize abnormalities—such as pulmonary nodule detection software¹ and colon polyp computer-aided detection tools²—the receiver operating characteristic (ROC) curve may overestimate performance, as it fails to penalize mislocalization errors that undermine clinical utility. To address this limitation, two extended ROC frameworks have been developed for detection and localization tasks, the localization ROC (LROC) curve³ and the free-response ROC/alternative free-response ROC (FROC/AFROC) curve.⁴ These two curves exhibit distinct application scenarios: in imaging studies where observers (or AI tools) are restricted to marking and scoring only a single suspicious location per image, the LROC curve proves optimal; if annotating all suspicious locations with confidence scores is allowed (e.g. multi-lesion detection), the FROC/AFROC curves become the appropriate framework. Liu et al.¹ collectively refers to these two types of curves as “location-specific ROC curves.” Unlike the conventional ROC curve, which only distinguishes between diseased and normal cases, location-specific ROC curves integrate spatial localization accuracy into the performance metric, providing a more comprehensive and clinically relevant evaluation.⁵

The LROC curve has been widely used in radiology studies.^6–9 Compared to the ROC curve, the LROC curve adds the true localization of diseased cases into the gold standard, giving a more appropriate definition of true positive in diagnostic cases where localization matters. Swensson¹⁰ introduced a binormal parametric model to estimate the LROC curve using the maximum likelihood method, and Popescu¹¹ proposed a nonparametric model using kernel density estimation for LROC analysis. Wilcoxon statistic was later used to estimate the area under the curve (AUC) of the LROC curve by Tang and Balakrishnan.¹² However, the parametric model assumes the normal distribution of the test results, which may not be satisfied in practice Bandos et al.¹³ The nonparametric model needs fewer assumptions, but the existing research lacks proof of the consistency and asymptotic properties.

The FROC curve was first proposed by Bunch et al.⁴ and developed by Chakraborty.¹⁴ In an FROC study, readers are allowed to mark any number of suspicious lesions for each image and then assign confidence scores to them, respectively. Each marked location can be classified as either a true positive (TP) or false positive (FP) finding based on the presence or absence of an actual abnormal lesion. The data is summarized in an FROC plot, where the $y$ axis is the proportion of abnormalities that are detected and correctly located, and the $x$ axis is the mean number of false-positive findings generated per patient.⁵ The FROC curve has several limitations. The area under the FROC curve (AUC of FROC) can be infinite and does not have a probabilistic explanation like the AUC of ROC , so it cannot be used as a valid index of the observer performance study.^15,16 These problems mean that we can’t use the FROC curve or the AUC of the FROC curve to compare two different tests. To solve these problems, the AFROC curve⁴ has been proposed, and the empirical estimation is a popular method to estimate the AUC of the AFROC curve^17–19 since assuming that the test results have certain distributions may be impractical. However, there are no asymptotic properties of these empirical estimators.

Our motivation comes from a diagnostic test where researchers aim to investigate the accuracy of an AI software in diagnosing lung nodules on CT scans. For each scan, the software marks all suspicious locations that might be lung nodules and assigns confidence scores to them, where a higher score indicates a greater probability that the marked location is a pulmonary nodule. The gold standard is given by a group of experts. The data consists of two parts: the confidence scores and the localization information. Since some scans have more than one marked location, the existing LROC curve is not suitable. The AFROC curve can be applied, but existing methods mainly focus on its AUC, while other indices that are also important, such as the true positive rate (TPR) at a fixed false positive rate (FPR) and within a range of FPRs,⁵ are neglected. Another limitation of the AFROC curve is that it doesn’t consider the confidence scores of the diseased cases with wrong localization. This implies that the use of AFROC curves fails to reflect the software’s capability of avoiding incorrect markings during lung CT scans of cancer patients, which is crucial as it prevents misjudgment in patient treatment. In conclusion, although there exist location-specific ROC curves that can be applied to our dataset, they fail to comprehensively measure the diagnostic and localization accuracy of the software. Therefore, we extend the LROC curve to the case where there can be any marked locations in each case and propose nonparametric inferences for it to achieve the goal of the diagnostic test.

Our article has three main contributions. First, we propose the empirical LROC curve and prove its asymptotic properties on the process level. Our method needs no extra assumptions on the distributions of test results, has a convergence rate of $n^{- 1 / 2}$ , and can be seen as an extension of the work of Hsieh and Turnbull²⁰ to the LROC study. Second, we extend the LROC curve so that it can be applied to the case where each case can be marked with any number of locations, i.e. the free-response tasks. Compared to the AFROC curve, the extended LROC curve takes the mislocation rate into consideration. Moreover, with asymptotic normality, we can make inferences for the indices related to the LROC curve, such as AUC and TPR at a fixed FPR. Third, we extend Bertail et al.’s²¹ work to the LROC curve, proposing a smoothed bootstrap method to construct a confidence band for the proposed LROC curve.

The rest of the paper is organized as follows. In Section 2, we propose the empirical estimation of the LROC curve and its extension to free-response tasks. In Section 3, we propose asymptotic properties of the proposed LROC curve and make inferences using our theoretical results. In Section 4 we illustrate our method through simulations. In Section 5 we apply our method to a real dataset from a clinical trial of a Software as a Medical Device (SaMD) by InferRead CT Lung, Infervision, Beijing, China. Section 6 is a discussion. Technical details are postponed to the supporting material.

2. Model

2.1. Definition of the LROC curve

In an LROC study (e.g. a radiology imaging test), for each case, the reader is asked to mark one suspected lesion and assign a confidence score to it.⁵ The gold standard, usually determined by a separate group of expert radiologists, consists of two parts, namely the disease status of all subjects and the location of the lesion in diseased subjects.

For diseased cases, let $L$ denote the localization information determined by comparing the lesion location reported by the reader to the gold standard, i.e. $L = 0$ if the location is inconsistent with the gold standard, and $L = 1$ otherwise. Let $X$ denote the confidence score assigned to the marked lesion. For the normal cases, let $Y$ denote the confidence score. Let $F$ denote the distribution function of $X$ condition on $L = 1$ and $Q$ denote the probability of correct localization. Let $G$ denote the distribution function of $Y$ . $F$ and $G$ can be presented as:

{\begin{aligned} F (t) = P (X \leq t | L = 1) \\ G (t) = P (Y \leq t) \end{aligned}

(1)

When the threshold is set as

c

, the TPR and the FPR of the LROC curve are defined as:

{\begin{aligned} TPR (c) = P (X > c, L = 1) = (1 - F (c)) Q \\ FPR (c) = P (Y > c) = 1 - G (c) \end{aligned}

(2)

The LROC curve is defined as a plot of the TPR on the vertical axis versus the FPR on the horizontal axis as the $c$ varies from $+ \infty$ to $- \infty$ , i.e.

LROC (t) = TPR ({FPR}^{- 1} (t)) = (1 - F (G^{- 1} (1 - t))) Q

(3)

The area under the LROC curve (AUC) summarizes the information of the LROC curve and is often used to compare different diagnostic methods.^7,8 We can write the AUC of the LROC curve as a probability:

AUC = \int_{0}^{1} (1 - F (G^{- 1} (1 - t))) Q d t = P (X > Y, L = 1)

(4)

So the AUC can be explained as the probability of the event that a disease case is correctly localized and its confidence score

X

is larger than a normal case’s confidence score

Y

2.2. Estimation

Suppose we have $m$ diseased cases and $n$ normal cases. For the $i$ th diseased case, let $L_{i}$ denote the localization information and $X_{i}$ denote the confidence score. For the $j$ th normal case, let $Y_{j}$ denote the confidence score. Let $l = \sum_{i = 1}^{m} L_{i}$ . For convenience, we assign the first $l$ disease cases is correctly located, i.e. $L_{i} = 1$ $⟺$ $i \in {1, 2, \dots l}$ .

The empirical estimates of $F$ , $G$ and $Q$ are

{\begin{aligned} \hat{F} (t) = \frac{1}{l} \sum_{i = 1}^{l} I (X_{i} \leq t) \\ \hat{G} (t) = \frac{1}{n} \sum_{j = 1}^{n} I (Y_{j} \leq t) \\ \hat{Q} = \frac{l}{m} \end{aligned}

And the nonparametric estimation of the LROC curve and the AUC are:

\begin{aligned} \hat{LROC} (t) & = \hat{Q} (1 - \hat{F} ({\hat{G}}^{- 1} (1 - t))) \\ \hat{AUC} & = \int_{0}^{1} (1 - \hat{F} ({\hat{G}}^{- 1} (1 - t))) \hat{Q} d t = \frac{\hat{Q}}{n l} \sum_{i = 1}^{l} \sum_{j = 1}^{n} I (X_{i} \geq Y_{j}) \end{aligned}

2.3. The extension to the free-response tasks

Before extending the LROC curve, we discuss the limitations of the existing LROC and AFROC curves and how the extended LROC curve can solve these limitations from the types of potential errors in free-response tasks. There are three kinds of mistakes in free-response tasks: (i) failure to mark locations on diseased cases where there is a lesion; (ii) incorrect marking of locations on diseased cases where there are no lesions; (iii) incorrect marking of locations on normal cases. The existing LROC curve neglects the third kind of mistakes since it assumes all cases, including normal cases, have one marked location.¹⁰ The AFROC curve treats the first kind of mistakes as false negatives and the third kind of mistakes as false positives,²² but neglects the second kind of mistakes.

Swensson¹⁰ proposed the first-choice hypothesis: for a given case, the reader first determines whether the case is diseased, and then marks all possible locations. The determination of disease status depends on whether the maximum score assigned by the reader exceeds a threshold. Thus, in free-response tasks, the maximum score among the marked locations for each case represents the reader’s report for that case. We further extend this by using the score and localization information of the marked location with the highest score in each case as the case-level judgment, and use it to define the TPR and the FPR. Unlike the AFROC curve, which overlooks erroneous localizations in diseased individuals, our method treats a diseased case as having a localization error if an incorrect localization’s score exceeds that of all correct localizations. Our approach captures three types of errors in free-response tasks and demonstrates greater sensitivity than the AFROC curve in detecting localization errors in diseased cases. We will illustrate this in the following estimation and in simulations.

Suppose we have $m$ diseased cases and $n$ normal cases. Let $m_{i}$ denote the number of marks in the $i$ th diseased case and $n_{j}$ denote the number of marks in the $j$ th normal case. Let $m_{i}^{(T)}$ denote the number of correctly marked locations compared to the gold standard and $m_{i}^{(F)}$ denote the number of wrongly marked locations compared to the gold standard in the $i$ th diseased case, clearly $m_{i}^{(T)} + m_{i}^{(F)} = m_{i}$ . Let $X_{i s}^{(T)}$ ( $s = 1, 2 \dots m_{i}^{(T)}$ ) denote the confidence score given to the $s$ th correct location on the $i$ th diseased case and $X_{i q}^{(F)}$ ( $q = 1, 2 \dots m_{i}^{(F)}$ ) denote the confidence score given to the $q$ th wrong location on the $i$ th diseased case. Let $n_{j}$ denote the number of marked locations in the $j$ th normal case. Let $Y_{j r}$ ( $r = 1, 2, \dots n_{j}$ ) denote the confidence score given to the $r$ th marked location on the $j$ th normal case. Note that if $m_{i}^{(T)}$ ( $m_{i}^{(F)}$ ; $n_{j}$ ) is equal to $0$ , $X_{i s}^{(T)}$ ( $X_{i q}^{(F)}$ ; $Y_{j r}$ ) does not exist.

Then we can define the case-level rate as follows:

In normal cases,

\begin{aligned} Y_{j} & = {\begin{aligned} max_{r} (Y_{j r}) & n_{j} > 0 \\ does not exist & n_{j} = 0 \end{aligned} \end{aligned}

\begin{aligned} L_{0 j} & = {\begin{aligned} 0 & n_{j} > 0 \\ 1 & n_{j} = 0 \end{aligned} \end{aligned}

In diseased cases,

\begin{aligned} L_{1 i} & = {\begin{aligned} 1 & \begin{aligned} m_{i}^{(T)} > 0, m_{i}^{(F)} > 0 and \\ max_{s} (X_{i s}^{(T)}) \geq max_{q} (X_{i q}^{(F)}) \\ or m_{i}^{(T)} > 0, m_{i}^{(F)} = 0 \end{aligned} \\ 0 & \begin{aligned} m_{i}^{(T)} > 0, m_{i}^{(F)} > 0 and \\ max_{s} (X_{i s}^{(T)}) < max_{q} (X_{i q}^{(F)}) \\ or m_{i}^{(T)} = 0, m_{i}^{(F)} \geq 0 \end{aligned} \end{aligned} \end{aligned}

\begin{aligned} X_{i} & = {\begin{aligned} max_{s} (X_{i s}^{(T)}) & \begin{aligned} m_{i}^{(T)} > 0, m_{i}^{(F)} > 0 and \\ max_{s} (X_{i s}^{(T)}) \geq max_{q} (X_{i q}^{(F)}) \\ or m_{i}^{(T)} > 0, m_{i}^{(F)} = 0 \end{aligned} \\ doesn't exist & \begin{aligned} m_{i}^{(T)} > 0, m_{i}^{(F)} > 0 and \\ max_{s} (X_{i s}^{(T)}) < max_{q} (X_{i q}^{(F)}) \\ or m_{i}^{(T)} = 0, m_{i}^{(F)} \geq 0 \end{aligned} \end{aligned} \end{aligned}

Let $l_{0} = n - \sum_{j = 1}^{n} L_{0 j}$ and $l_{1} = \sum_{i = 1}^{m} L_{1 i}$ . For convenience, by changing the order of diseased and normal cases, we can suppose that $L_{1 i} = 1 iff i = 1, 2, \dots l_{1}$ and $L_{0 j} = 0 iff j = 1, 2, \dots l_{0}$ .

The case-level data can then be presented as follows:

\begin{aligned} {X_{i}, i = 1, 2, \dots l_{1}}, {L_{1 i}, i = 1, 2, \dots m} \\ {Y_{j}, j = 1, 2 \dots l_{0}}, {L_{0 j}, j = 1, 2, \dots n} \end{aligned}

For the diseased case, let $L_{1}$ denote the localization information and $X$ denote the confidence score. For the normal case, let $L_{0}$ denote the localization information and $Y$ denote the confidence score.

We then define the case-level TPR and the FPR in free-response tasks when the threshold is set as $c$ :

{\begin{aligned} TPR = P (X > c, L_{1} = 1) \\ FPR = P (Y > c, L_{0} = 0) \end{aligned}

(5)

We further define:

{\begin{aligned} F_{0} (c) & = P (Y \leq c | L_{0} = 0), P_{0} = P (L_{0} = 0) \\ F_{1} (c) & = P (X \leq c | L_{1} = 1), P_{1} = P (L_{1} = 1) \end{aligned}

The extended LROC curve is defined as a plot of the TPR on the vertical axis versus the FPR on the horizontal axis as the

c

varies from

+ \infty

- \infty

, i.e.

LROC (t) = {\begin{aligned} TPR ({FPR}^{- 1} (t)) & t \in [0, P_{0}] \\ P_{1} & t \in (P_{0}, 1] \end{aligned}

(6)

The mathematical expression for the extended LROC curve and its AUC are:

LROC (t) = {\begin{aligned} P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) & t \in [0, P_{0}] \\ P_{1} & t \in (P_{0}, 1] \end{aligned}

(7)

and

\begin{aligned} AUC = & P_{1} - \int_{0}^{P_{0}} P_{1} F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) d t \\ = & P (X \geq Y, L_{1} = 1, L_{0} = 0) \end{aligned}

(8)

The AUC can be explained as the probability that a diseased case has a test result with true localization and is no less than the test result of a normal case with false localization.

We get the empirical estimation of $F_{0}$ , $F_{1}$ , $P_{0}$ and $P_{1}$ from the data:

{\begin{aligned} \hat{F_{0}} (t) & = \frac{1}{l_{0}} \sum_{j = 1}^{l_{0}} I (Y_{j} \leq t) \\ \hat{F_{1}} (t) & = \frac{1}{l_{1}} \sum_{i = 1}^{l_{1}} I (X_{i} \leq t) \\ \hat{P_{0}} & = \frac{l_{0}}{n} \\ \hat{P_{1}} & = \frac{l_{1}}{m} \end{aligned}

Then the empirical LROC curve can be defined as:

\hat{LROC} (t) = {\begin{aligned} \hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) & , t \in [0, \hat{P_{0}}] \\ \hat{P_{1}} & , t \in (\hat{P_{0}}, 1] \end{aligned}

(9)

The area under the empirical LROC curve can be presented as:

\hat{AUC} = \hat{P_{1}} - \hat{P_{0}} \hat{P_{1}} \cdot \frac{1}{l_{0} l_{1}} \sum_{i = 1}^{l_{1}} \sum_{j = 1}^{l_{0}} I (X_{i} \leq Y_{j})

From the estimation, the extended LROC curve punishes the mislocation in diseased cases through the “first choice” report, i.e. the maximum score. If a marked disease case’s maximum score comes from a wrongly localized region, it will be seen as a mislocation case and reduce

l_{1}

. Moreover, all its confidence scores with true localization will be excluded from computing

\hat{F_{1}}

. Compared to the AFROC curve, the extended LROC curve takes

X_{i q}^{(F)}

into consideration. When

P_{0}

= 1, the extended LROC curve reduces to the existing LROC curve.

Similar to the ROC curve, the proposed LROC curve has the property of invariance to monotone transformations of the confidence scores. Specifically, let $ϕ (x)$ denote an arbitrary strictly increasing function, which is invertible by definition. If we apply this transformation to the original confidence scores to obtain the transformed scores $X^{1} = ϕ (X)$ and $Y^{1} = ϕ (Y)$ , both the proposed LROC curve and its AUC remain unchanged. That’s because in such a case,

{\begin{aligned} F_{0}^{1} (c) & = P (ϕ (Y) \leq c | L_{0} = 0) = F_{0} (ϕ^{- 1} (c)) \\ F_{1}^{1} (c) & = P (ϕ (X) \leq c | L_{1} = 1) = F_{1} (ϕ^{- 1} (c)) \end{aligned}

and

{LROC}^{1} (t) = {\begin{aligned} P_{1} (1 - F_{1}^{1} ({F_{0}^{1}}^{- 1} (1 - \frac{t}{P_{0}}))) & t \in [0, P_{0}] \\ P_{1} & t \in (P_{0}, 1] \end{aligned}

(10)

Note that

\begin{aligned} F_{1}^{1} ({F_{0}^{1}}^{- 1} (1 - \frac{t}{P_{0}})) & = F_{1} (ϕ^{- 1} (ϕ (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) \\ = F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) \end{aligned}

(11)

Then we can conclude that

{LROC}^{1} (t) = LROC (t)

, i.e. the LROC curve is identical under the strictly monotone increasing transformation.

3. Main results

3.1. Assumptions and statistical properties

In a FROC study, a common assumption is that the test results from different cases are independent. However, assuming that the test results from a single case are independent may be too strong in some situations and difficult to justify.¹³ Luckily, after transforming the data we only need to assume the independence of the test results from different cases. In detail, the following assumptions are needed to obtain the asymptotic properties:

Confidence scores from different cases, i.e. ${X_{i}}$ and ${Y_{j}}$ are independent. The accuracy of localization, i.e. ${L_{0 j}}_{j = 1}^{n}$ and ${L_{1 i}}_{i = 1}^{m}$ are independent.

The sample sizes m and n are such that ${l i m}_{m \to \infty} \frac{n}{m} = λ$ , where $0 < λ < \infty$ .

The distribution function $F_{0}$ and $F_{1}$ have continuous densities $f_{0}$ , $f_{1}$ , respectively. The derivative function of $F_{1} F_{0}^{- 1} (t)$ , i.e. $f_{1} (F_{0}^{- 1}) (t) / f_{0} (F_{0}^{- 1}) (t)$ is bounded on any subinterval $(a, b)$ of $(0, 1)$ .

Under the assumptions (1), (2), and (3), we have the following theorems:

Theorem 1
We derive the uniform consistency of the empirical LROC curve:
$\underset{0 \leq t \leq 1}{s u p} | \hat{LROC} (t) - LROC (t) | \to 0 a.s. as n \to \infty .$
(12)
Theorem 2
We obtain the convergence of the empirical LROC curve on the process level. In detail, We can define two Brownian bridges $B_{1}^{n} (t)$ and $B_{2}^{n} (t)$ as well as two normal random variable $W_{0}$ and $W_{1}$ , with $W_{i} \sim N (0, P_{i} (1 - P_{i}))$ . $B_{1}^{n} (t)$ , $B_{2}^{n} (t)$ , $W_{0}$ and $W_{1}$ are independent and
$\begin{aligned} \sqrt{n} (\hat{LROC} (t) - LROC (t)) \\ \overset{d}{\to} & {\begin{aligned} \begin{aligned} \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} \\ + P_{1} (F_{1} (F_{0})^{- 1})^{'} (1 - \frac{t}{P_{0}}) \frac{t}{P_{0}^{2}} W_{2} \\ + \sqrt{λ P_{1}} B_{1}^{n} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) \\ + \frac{P_{1}}{\sqrt{P_{0}}} (F_{1} (F_{0}^{- 1}))^{'} (1 - \frac{t}{P_{0}}) B_{2}^{n} (1 - \frac{t}{P_{0}}) \end{aligned} & t \in (0, P_{0}] \\ \sqrt{λ} W_{1} & t \in (P_{0}, 1] \end{aligned} \end{aligned}$
(13)
Corollary 1
With Theorem 2, we obtain the asymptotic normality of the empirical AUC:
$\sqrt{n} (\hat{AUC} - AUC) \overset{d}{\to} N (0, σ^{2})$
where
$\begin{aligned} σ^{2} & = λ {(1 - P_{0} \int F_{1} F_{0}^{- 1})}^{2} P_{1} (1 - P_{1}) \\ + P_{1}^{2} {(\int F_{1} F_{0}^{- 1})}^{2} P_{0} (1 - P_{0}) \\ + λ P_{1} P_{0}^{2} | | F_{1} F_{0}^{- 1} (m) | | + P_{0} P_{1}^{2} | | F_{0} F_{1}^{- 1} (m) | | \end{aligned}$
(14)
and $| | h | | = \int h^{2} - (\int h)^{2}$ .

Proofs for Theorems 1 and 2 are provided in Appendix A. When $P_{0} = 1$ , the extended LROC curve reduces to the existing LROC curve, so Theorems 1 and 2 also apply to the empirical estimation of the existing LROC curve.
3.2. Inference for TPR at a given FPR and the AUC

We first focus on the TPR at a given FPR, i.e. $LROC (t)$ at a fixed t, which is important in some diagnostic research⁵ and receives attention in decision curve analysis.²³ Using Theorem 2, the confidence interval for $LROC (t)$ of level $1 - α$ is given by

[\hat{LROC (t)} - \frac{{\hat{σ}}_{t} Φ^{- 1} (1 - (α / 2))}{\sqrt{n}}, \hat{LROC (t)} + \frac{{\hat{σ}}_{t} Φ^{- 1} (1 - (α / 2))}{\sqrt{n}}]

where

{\hat{σ}}_{t}^{2}

is an estimation of the variance of

\sqrt{n} (L R O C (t) - \hat{L R O C (t)})

and can be obtained by the bootstrap method.

Also, using Corollary 1, the confidence interval for the AUC of level $1 - α$ is given by

[\hat{AUC} - \frac{\hat{σ} Φ^{- 1} (1 - (α / 2))}{\sqrt{n}}, \hat{AUC} + \frac{\hat{σ} Φ^{- 1} (1 - (α / 2))}{\sqrt{n}}]

Again, we can compute

\hat{σ}

by the bootstrap method.

Specifically, suppose we aim to estimate the variance ${\hat{σ}}_{t_{0}}^{2}$ for a fixed $t_{0}$ and ${\hat{σ}}^{2}$ , the bootstrap method is as follows.

Based on original data compute empirical estimation $\hat{F_{0}}$ , $\hat{F_{1}}$ , $\hat{P_{0}}$ , $\hat{P_{1}}$ and $\hat{AUC}$ .

Generate a bootstrap sample of $n$ normal cases and $m$ diseased cases as follows.

For normal cases, draw a sample ${L_{0 j}^{*}}$ of size $n$ from a Bernoulli distribution with success probability $1 - \hat{P_{0}}$ (hereafter denoted as $B (1, 1 - \hat{P_{0}})$ ). Let $l_{0}^{*} = n - \sum_{j = 1}^{n} L_{0 j}^{*}$ , draw a sample ${Y_{j}^{*}}_{j = 1}^{l_{0}^{*}}$ of size $l_{0}^{*}$ from $\hat{F_{0}}$ .

For diseased cases, draw a sample $L_{1 i}^{*}$ of size $m$ from $B (1, \hat{P_{1}})$ . Let $l_{1}^{*} = \sum_{i = 1}^{m} L_{1 i}^{*}$ , draw a sample ${X_{i}^{*}}_{i = 1}^{l_{1}^{*}}$ of size $l_{1}^{*}$ from $\hat{F_{1}}$ .

Based on the bootstrap sample, compute empirical estimates ${LROC}^{*} (t)$ and ${AUC}^{*}$ . Then compute $α$ and $β (t_{0})$ , where $α = \sqrt{n} ({AUC}^{*} - \hat{AUC})$ and $β (t_{0}) = \sqrt{n} ({LROC}^{*} (t_{0}) - \hat{LROC} (t_{0}))$ .

Repeat (2) and (3) for $T$ times to obtain $T$ bootstrap samples of $α$ and $β (t_{0})$ , denoted as $α_{1}, α_{2}, \dots, α_{T}$ and $β_{1} (t_{0}), β_{2} (t_{0}), \dots, β_{T} (t_{0})$ . Compute their sample variance, respectively, as ${\hat{σ}}^{2}$ and ${\hat{σ}}_{t_{0}}^{2}$ .

3.3. Confidence bands for the LROC curve

We have already gained the confidence interval of $LROC (t)$ for any given t. Varying t from 0 to 1, and we can obtain a pointwise confidence interval for the LROC curve. However, to build a confidence band, we need to simulate the distribution of the process $R_{n} (t) = \sqrt{n} (\hat{LROC} (t) - LROC (t))$ and a smoothed bootstrap method is then proposed.

In ROC analysis, Macskassy et al.²⁴ first applied the bootstrap approach to build a confidence band for ROC. Hall et al.²⁵ and Ghosal and Gu²⁶ discussed the statistical properties of the nonparametric confidence band for ROC. The inspiration for our method comes from Bertail et al.,²¹ Who proposed a bootstrap method for constructing confidence bands for the ROC curve. According to Bertail et al.,²¹ naive bootstrap (i.e. resampling from the raw empirical distribution) provides an unsatisfactory approximation as the convergence rate of a given quantile is $O (n^{- 1 / 4})$ . To solve the problem of estimating $R_{n}$ , here we follow Hall et al.²⁵ and Bertail et al.’s²¹ work on ROC curve and adopt a smoothed version of the empirical LROC curve for resampling. Take $\hat{F_{0}} (t) = (1 / l_{0}) \sum_{j = 1}^{l_{0}} I (Y_{j} \leq t)$ for example, to smooth $\hat{F_{0}}$ , a typical choice is to use $\int_{- \infty}^{t} K_{h} (u - Y_{j}) d u$ to substitute $I (Y_{j} \leq t)$ , where $K_{h} (x) = (1 / h) K (x / h)$ and $K$ is a kernel density. The details for the bootstrap method are as follows:

Based on original data compute empirical estimation $\hat{F_{0}}$ , $\hat{F_{1}}$ , $\hat{P_{0}}$ and $\hat{P_{1}}$ . Using $\hat{F_{0}}$ and $\hat{F_{1}}$ to construct smoothed versions $\tilde{F_{0}}$ and $\tilde{F_{1}}$ .

Generate a bootstrap sample of $n$ normal cases and $m$ diseased cases as follows.

For normal cases, draw a sample ${L_{0 j}^{*}}$ of size $n$ from $B (1, 1 - \hat{P_{0}})$ . Let $l_{0}^{*} = n - \sum_{j = 1}^{n} L_{0 j}^{*}$ , draw a sample ${Y_{j}^{*}}_{j = 1}^{l_{0}^{*}}$ of size $l_{0}^{*}$ from $\tilde{F_{0}}$ .

Based on the bootstrap sample, compute ${LROC}^{*} (t)$ . Based on ${LROC}^{*} (t)$ and $\hat{LROC} (t)$ , we compute $R_{n}^{*} (t)$ , where

R_{n}^{*} (t) = \sqrt{n} ({LROC}^{*} (t) - \hat{LROC} (t))

Get the bootstrap confidence bands at level $1 - δ$ defined by the ball of center $\hat{LROC}$ and radius $r (δ) / \sqrt{n}$ , with $r (δ)$ defined by

P^{*} {sup_{t \in [ϵ, 1 - ϵ]} | R_{n}^{*} (t) | \leq r (δ) |} = 1 - δ

where

P^{*}

denotes the conditional probability given the original data.

For the smoothed bootstrap method, we have the following theorem whose proof is provided in Appendix A of the Supplemental Material.

Theorem 3
If both $\tilde{F_{0}}$ and $\tilde{F_{1}}$ are computed using kernel $K_{h_{n}} (x)$ with $h_{n} \to 0$ , $n h_{n}^{3} \to \infty$ and $n h_{n}^{5} (\log n)^{2} \to 0$ as $n \to \infty$ , We have
$s u p_{u \in R} | P^{} (| | R_{n}^{} | |_{\infty} \leq u) - P (| | R_{n} | |_{\infty} \leq u) | = O (\frac{l o g (n)}{\sqrt{n} h_{n}})$

If we choose $h_{n} \sim n^{- 1 / 5} l o g {(n)}^{- (2 / 5 + η)}$ with $η$ being a small constant, we will have a convergence rate close to $n^{- 3 / 10}$ .
4. Simulation

In this section, we conduct three simulations. The first is to assess the bias of the empirical LROC curve in finite samples. The second is to assess the coverage rate (CR) and average width (AW) of the confidence band for the LROC curve and the confidence interval for the AUC. Here, the coverage rate is the proportion of simulation runs where the true LROC curve lies entirely within the confidence band or where the true AUC lies within the confidence interval, and the average width refers to the mean of width of the confidence interval for AUC or the confidence band for the LROC curve across all simulation runs. The third is to show that the LROC curve can detect the reader’s mislocation mistakes in diseased cases, which the AFROC curve ignores.

4.1. Simulation for the bias of the empirical LROC curve in finite sample

From the mathematical definition, $L R O C (t)$ is a function defined on the interval $[0, 1]$ . We then adopt two approaches to measuring the distance between the empirical LROC curve and the true LROC curve.

The Integrate Bias, which is $\int_{0}^{1} | LROC (t) - \hat{LROC} (t) | d t .$

The Integrate MSE, which is $\int_{0}^{1} (LROC (t) - \hat{LROC} (t))^{2} d t .$

Let

N = m + n

be the total number of diseased and normal cases. Let

λ

denote the ratio of the number of normal cases to the number of diseased cases. First we set

λ = 1

and change

N

from

100

2000

to check the consistency and convergence rate of the empirical estimation. Then we set

N = 600

and change

λ

from

0.1

10

to check the empirical estimation’s ability to deal with different ratios of diseased cases and normal cases. In each simulation, after setting

λ

and

N

, we set

P_{0} = 0.3

P_{1} = 0.9

and generate the data using the following process:

\begin{aligned} L_{0 j} \sim B (1, 1 - P_{0}), L_{1 i} \sim B (1, P_{1}); \\ X_{i} | L_{1 i} = 1 \sim F_{1}, Y_{j} | L_{0 j} = 0 \sim F_{0} \end{aligned}

One of the advantages of our method over the parametric model is that our model makes fewer assumptions about the form of

F_{0}

and

F_{1}

. Here we choose

F_{0}

and

F_{1}

as (i) Normal distribution:

X_{i} | L_{1 i} = 1 \sim N (1, 1), Y_{j} | L_{0 j} = 0 \sim N (0, 1)

; (ii) Lognormal distribution:

l n (X_{i}) | L_{1 i} = 1 \sim N (1, 1), l n (Y_{j}) | L_{0 j} = 0 \sim N (0, 1)

and (iii) Student’s-t distribution:

X_{i} | L_{1 i} = 1 \sim t (10), Y_{j} | L_{0 j} = 0 \sim t (1)

Here we show the simulation result for the normal distribution in Figure 1 and leave other results in Appendix B. Simulation results of Integrated Bias and Integrated MSE for these three distributions of $X$ and $Y$ are consistent with Theorem 1 and Theorem 2. In all cases, the bias reaches a minimum when the ratio is close to 1. Although Theorem 1 claims that consistency holds for any $0 < λ < \infty$ , simulation results show that an extreme ratio needs larger $N$ to get an acceptable bias. So in practice, we recommend controlling the $λ$ close to $1$ . In next simulations, we fix $λ = 1$ .

Figure 1.

Integrate bias and MSE for normal distribution as $N$ (top) and $λ$ (bottom) change.

4.2. Simulation for the confidence intervals of the AUC and confidence bands of the LROC curve

In the second simulation, we illustrate our bootstrap methods. First, we compute the coverage rate and the average width of the AUC confidence interval and the LROC confidence band. We choose confidence level $1 - δ = 95 %$ and $L_{1 i} \sim B (1, P_{1}), L_{0 j} \sim B (1, P_{0})$ , and simulate under three distributions:

Normal distribution: $X_{i} | L_{1 i} = 1 \sim N (1, 1), Y_{0, j} | L_{0 j} = 0 \sim N (0, 1)$ .

Lognormal distribution: $l n (X_{i}) | L_{1 i} = 1 \sim N (2, 1)$ , $l n (Y_{0, j}) | L_{0 j} = 0 \sim N (0, 1) .$

Beta distribution: $X_{i} | L_{1 i} = 1 \sim B e t a (20, 5), Y_{0, j} | L_{0 j} = 0 \sim B e t a (5, 5) .$

We choose $N = 200, 400, 800, 1600$ and $(P_{0}, P_{1}) = (0.3, 0.9), (0.5, 0.7)$ . For the smoothing parameter in the construction of the confidence band, we choose $K$ as the standard Gaussian density function and $h$ as $C \cdot n^{- 1 / 5} l o g {(n)}^{- 1 / 2}$ , where $C$ is a constant.

Table 1 shows the simulation result for the AUC confidence interval. The coverage rates are close to 0.95 in all cases. Even with a small value of N, the coverage rate can be consistently maintained within the range of 0.94 to 0.96. This result indicates that our proposed approach remains valid and practicable in settings with limited sample sizes. We also notice that the average width is influenced by $N$ , $P_{0}$ , and $P_{1}$ rather than the test result’s distribution.

Table 2 shows the simulation result for the LROC confidence band. The average width of the confidence bands consistently decreases as the total sample size increases, which aligns with the expected behavior of a consistent inference procedure. For the coverage rate (CR), the results are overall close to the nominal $95 %$ level across all simulation scenarios, which aligns with the asymptotic validity of our proposed method. Meanwhile, we observe that for some settings, even at a sample size of $N = 800$ , the coverage rates have not yet fully converged to the nominal level, with minor fluctuations both above and below $0.95$ . For instance, the CR reaches $0.963$ for the normal distribution with $(P_{0}, P_{1}) = (0.3, 0.9)$ , while it drops to $0.941$ for the lognormal distribution with $(P_{0}, P_{1}) = (0.5, 0.7)$ , both showing a deviation of around 0.01 from the nominal $95 %$ level. We attribute this incomplete convergence in finite samples to the slow convergence rate of the supremum norm adopted for uniform confidence band construction. Based on our theoretical derivation, the convergence rate of the supremum norm is around $n^{- 3 / 10}$ . This slow convergence naturally leads to minor fluctuations in coverage rates around the nominal level in finite samples, and this phenomenon is also documented on uniform confidence band construction for ROC curves.²¹ Moreover, the simulation results at the larger sample size of $N = 1600$ show that the coverage rates become very close to the nominal $95 %$ level across all distributions and parameter settings, which further validates the asymptotic properties of our method.

Table 1.
Simulation for AUC confidence interval.

Distribution $P_{0}$ $P_{1}$ $N$ CR AW

Normal 0.3 0.9 200 0.957 0.127

400 0.945 0.090

800 0.951 0.064

1600 0.948 0.045

0.5 0.7 200 0.947 0.172

400 0.949 0.122

800 0.944 0.086

1600 0.949 0.061

Lognormal 0.3 0.9 200 0.940 0.118

400 0.941 0.084

800 0.956 0.059

1600 0.947 0.042

0.5 0.7 200 0.948 0.175

400 0.941 0.124

800 0.953 0.088

1600 0.949 0.062

Beta 0.3 0.9 200 0.940 0.117

400 0.939 0.084

800 0.945 0.059

1600 0.951 0.042

0.5 0.7 200 0.946 0.177

400 0.959 0.125

800 0.948 0.089

1600 0.952 0.063

Distribution	$P_{0}$	$P_{1}$	$N$	CR	AW
Normal	0.3	0.9	200	0.957	0.127
			400	0.945	0.090
			800	0.951	0.064
			1600	0.948	0.045
	0.5	0.7	200	0.947	0.172
			400	0.949	0.122
			800	0.944	0.086
			1600	0.949	0.061
Lognormal	0.3	0.9	200	0.940	0.118
			400	0.941	0.084
			800	0.956	0.059
			1600	0.947	0.042
	0.5	0.7	200	0.948	0.175
			400	0.941	0.124
			800	0.953	0.088
			1600	0.949	0.062
Beta	0.3	0.9	200	0.940	0.117
			400	0.939	0.084
			800	0.945	0.059
			1600	0.951	0.042
	0.5	0.7	200	0.946	0.177
			400	0.959	0.125
			800	0.948	0.089
			1600	0.952	0.063

Table 2.

Simulation for the LROC confidence band.

Distribution	$P_{0}$	$P_{1}$	$N$	CR	AW
Normal	0.3	0.9	200	0.963	0.262
			400	0.959	0.185
			800	0.963	0.128
			1600	0.955	0.089
	0.5	0.7	200	0.975	0.201
			400	0.970	0.139
			800	0.959	0.096
			1600	0.954	0.067
Lognormal	0.3	0.9	200	0.952	0.119
			400	0.953	0.081
			800	0.949	0.056
			1600	0.947	0.038
	0.5	0.7	200	0.967	0.118
			400	0.952	0.080
			800	0.941	0.055
			1600	0.951	0.038
Beta	0.3	0.9	200	0.948	0.090
			400	0.962	0.058
			800	0.958	0.039
			1600	0.953	0.026
	0.5	0.7	200	0.959	0.111
			400	0.944	0.075
			800	0.955	0.051
			1600	0.952	0.035

4.3. Simulation for comparison of the AFROC curve and the LROC curve

In the third simulation, the LROC and AFROC curves are used to analyze the free-response tasks data to show that the LROC curve is superior to the AFROC curve in detecting mislocation mistakes in diseased cases. We set $m = n = 300$ . The data generation is as follows:

For $i$ th diseased case set the number of the lesions $l_{i}$ as $l_{i} \sim B (p_{l}, M) + 1$ and then the number of correctly marked locations $m_{i}^{(T)}$ as $m_{i}^{(T)} \sim B (l_{i}, p_{1})$ . If $m_{i}^{(T)} > 0$ , we draw ${X_{i s}^{(T)}}_{s = 1}^{m_{i}^{(T)}}$ from a multivariate normal distribution with $E X_{i s}^{(T)} = μ_{T}$ , $V a r (X_{i s}^{(T)}) = 1$ and $C o r (X_{i s_{1}}^{T}, X_{i s_{2}}^{T}) = ρ$ . The number of wrongly marked locations $m_{i}^{(F)}$ is set as $m_{i}^{(F)} \sim P o i (λ_{F})$ . If $m_{i}^{(F)} > 0$ , set $X_{i q}^{(F)} \sim N (μ_{F}, 1)$ .

For $j$ th normal cases, set the number of marked locations $n_{j} \sim P o i (λ)$ .If $n_{j} > 0$ , set $Y_{j r} \sim N (0, 1)$

Since the LROC curve and the AFROC curve have the same FPR, we focus on parameters related to diseased cases and fix other parameters. In detail, we fix

p_{l} = 0.2

M = 10

p_{1} = 0.9

ρ = 0.3

λ = 1

and change

μ_{T}

μ_{F}

λ_{F}

in turn. Our goal is to find out whether the LROC curve and the AFROC curve can detect the changes of these parameters. The results are as follows.

Figure 2 shows the results of the two curves at different $μ_{T}$ , i.e. the mean of $X_{i s}^{(T)}$ (confidence score of a lesion). It can be observed that as $μ_{T}$ increases, both curves rise significantly, indicating that both can detect the ability of readers to score actual lesions higher. Figure 3 shows the results of the two curves at different $μ_{F}$ , i.e. the mean of $X_{i q}^{(F)}$ (confidence score of a mislocated location). It can be observed that as $μ_{F}$ increases, our LROC curve decreases significantly, while the AFROC curve remains essentially unchanged. This indicates that only the LROC curve can detect the ability of readers to score normal positions on diseased individuals lower. Figure 4 shows the results of the two curves at different $λ_{F}$ , i.e. the mean of $m_{i}^{(F)}$ (the number of mislocation mistakes in a diseased case). It can be seen that as $λ_{F}$ increases, the LROC curve decreases significantly, while the AFROC curve remains essentially unchanged. This indicates that only the LROC curve can detect the ability of readers to make fewer incorrect marks on diseased individuals. The simulation results are consistent with our analysis that the LROC curve can detect the reader’s ability to make fewer mislocation mistakes in diseased cases.

Figure 2.

The empirical LROC (top) and AFROC (bottom) curve for different $μ_{T}$ .

Figure 3.

The empirical LROC (top) and AFROC (bottom) curve for different $μ_{F}$ .

Figure 4.

The empirical LROC (top) and AFROC (bottom) curve for different $λ_{F}$ .

5. Example

We illustrate the LROC method using the data from a clinical trial of a SaMD by InferRead CT Lung, Infervision, Beijing, China, coming from Liu et al.¹ The SaMD, which aims to help readers find pulmonary lesions, can mark all suspicious lesions in an image and assign confidence scores to them.

This study tries to demonstrate superiority in the performance of a radiologist aided by the software, as well as in the performance of this software when it works alone.¹ Therefore, the study consists of a multi-reader, multi-case reader study and a SaMD working alone study. We focus on the SaMD working alone study and want to prove its effectiveness. The data, consisting of 161 normal and 183 diseased cases, is continuous. The gold standard is set by three experienced experts. The first and second experts independently read the image and recorded the locations, sizes, and categories of the lesions. The last expert acts as an arbitrator. If the first two experts have different opinions, the last expert makes the final decision. The software then analyzes all subjects, distinguishes diseased and normal subjects, and indicates suspicious lesions in those diseased subjects.

The study is a free-response task since from the design researchers didn’t order the software not to mark more than one lesion for each image. In the original study, researchers considered three indices: LDR NSMR and cAFAUC. These indices are tightly associated with the AFROC curve and have clear clinical interpretations. (For more details we refer to Liu et al.¹) However, these indices do not consider the reader’s mislocation mistakes in diseased cases. Although the SaMD performed well in these indices, the ability of the SaMD to avoid marking normal locations in diseased cases remains unknown. For an extreme example, consider another reader (denoted as A) performing the same in normal cases and those marked lesions as the SaMD. However, the confidence scores assigned by A to those mislocated locations and the marked lesions are from the same distribution, i.e. A can hardly distinguish between lesions and normal locations in diseased cases. A is worse than the SaMD, but their empirical AFROC curves and the three indices above are the same, which means the original study can’t distinguish A and the SaMD. The LROC curve can then be used to complement the SaMD’s research.

We use the method proposed in Section 2 to deal with the data. Our results are shown in Figure 5, which shows the pointwise confidence interval and the confidence bands for the LROC curve. Since our dataset is small, the bootstrap confidence band over the interval $[0, 1]$ is too wide and has little reference value. However, we find that the average width will observably decline if we build the confidence band for $t \in (ϵ, 1)$ rather than $t \in (0, 1)$ , where $ϵ$ is a small positive number close to 0. So we recommend first building the confidence band over $[ϵ, 1]$ and then extending it to [0,1] using the monotony of the LROC curve and the fact that $LROC (t) \in [0, 1]$ . In this case, we can first build a $95 %$ confidence band for $t \in [0.08, 1]$ and then extend it to $[0, 1]$ .

Figure 5.

Pointwise confidence intervals (top) and confidence band (bottom).

In addition, we compute that the AUC of the empirical LROC curve is $0.813$ and construct the confidence interval of the AUC using the bootstrap method, which is $[0.762, 0.864]$ . According to Section 3, we obtain an asymptotically consistent estimate of $AUC = P_{1} - P_{0} P_{1} P (X \leq Y)$ . In detail, we compute their $95 %$ confidence intervals: $P_{1} :\ [0.792, 0.896]$ ; $P_{0} :\ [0.261, 0.410]$ and $P (X \leq Y) :\ [0.069, 0.176]$ . These results show that the SaMD has a high probability of correctly locating lesions in diseased cases; a low probability of marking normal cases and a low probability of giving a normal location higher confidence scores than a lesion, thus verifying the software’s ability to detect and locate pulmonary lesions.

Compared to the methods used in Liu et al.,¹ the LROC curve is a more comprehensive method that simultaneously reflects the SaMD’s three abilities: (1) to correctly locate lesions in the diseased cases; (2) to give higher confidence scores to lesions than those marked locations in normal cases; (3) to give higher confidence scores to lesions than to those wrongly marked locations in diseased cases. What’s more, we give the uniform band and pointwise intervals of the LROC curve so researchers can evaluate the ability of the SaMD both uniformly and at a fixed FPR.

6. Discussion

Free-response observer performance assessment plays an important role in diagnostic radiology,¹⁹ but the studies of its theory, i.e. the location-specific ROC curves and their estimators’ statistical properties, are scarce in comparison to its wide range of applications. In this paper, we propose an empirical estimation for the LROC curve to solve the problem of existing estimation methods for the LROC curve. Our method does not need extra assumptions on the specific distribution of the confidence scores to maintain its statistical properties, so it is theoretically reliable for continuous test results. We further extend the LROC curve to deal with the free-response tasks. The proposed LROC curve considers the confidence scores of those wrongly located regions in diseased cases, which are ignored by the AFROC curve. In practice, it punishes those who blindly give high scores to suspicious regions with wrong localization and prevents them from obtaining undue compliments, which may happen if the AFROC curve is used alone. Moreover, the AUC of the LROC curve has a specific probability meaning, which can serve as a proper summary of the reader’s ability and be used to compare different readers.

The limitation of our method is that we need to drop out some test results, which also happens in the AFROC and inferred ROC studies. Also, our method can only deal with continuous data. For future work, we want to extend our method to the case of ordinal data and binary data.

Supplemental Material

sj-zip-1-smm-10.1177_09622802261468028 - Supplemental material for Nonparametric inference for the localization receiver operating characteristic curve and its extension to free-response image localization tasks

Supplemental material, sj-zip-1-smm-10.1177_09622802261468028 for Nonparametric inference for the localization receiver operating characteristic curve and its extension to free-response image localization tasks by Kaiyuan Liu and Xiao-Hua Zhou in Statistical Methods in Medical Research

Footnotes

Acknowledgments

The authors thank the anonymous reviewers for their valuable suggestions. The authors thank the Beijing Infervision Technology Co. Ltd. for providing the dataset of the SaMD example.

ORCID iDs

Kaiyuan Liu

Xiao-Hua Zhou

Ethical approval and informed consent statements

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: the National Natural Science Foundation of China (Grant number 82173623).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The data that supports the findings of this study are available from the Beijing Infervision Technology Co. Ltd. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from the corresponding author with the permission of the Beijing Infervision Technology Co. Ltd.

Supplemental material

Supplemental material for this article is available online.

We begin our proof by referring to two results proposed by Hsieh and Turnbull.²⁰

Suppose $X_{i}, i = 1, 2, \dots m$ and $Y_{j}, j = 1, 2, \dots n$ are two i.i.d sequences and have distribution functions $F$ and $G$ , respectively. If both $F$ and $G$ have continuous densities $f$ , $g$ , respectively and the derivative function of $F G^{- 1} (t)$ , i.e. $f (G^{- 1}) (t) / g (G^{- 1}) (t)$ is bounded on any subinterval $(a, b)$ of $(0, 1)$ , $0 < a < b < 1$ , then for the empirical distribution function $\hat{F}$ and $\hat{G}$ , we have: (15)

sup_{t \in [0, 1]} | \hat{F} {\hat{G}}^{- 1} (t) - F G^{- 1} (t) | \to 0 a.s. as n \to \infty .

And there exists a probability space on which one can define sequences of two independent versions of Brownian bridges

B_{1}

and

B_{2}

such that for

t \in [a, b]

(16)

\begin{aligned} \sqrt{n} (\hat{F} {\hat{G}}^{- 1} (t) - F G^{- 1} (t)) = \sqrt{λ} B_{1} (F G^{- 1} (t)) + (F G^{- 1})^{'} (t) B_{2} (t) + o (n^{- \frac{1}{2}} (l o g n)^{2}) a.s. \end{aligned}

Since $LROC (t)$ is a piecewise function and when $t > P_{0}$ it is a constant that does not rely on $t$ , we first focus on the case that $t \leq m i n (P_{0}, \hat{P_{0}})$ . Under the assumptions (1), (2), and (3), we have Lemma 1 and 2: Lemma 1

The empirical LROC curve is consistent when $t \leq m i n (P_{0}, \hat{P_{0}})$ . Let $A$ denote the interval $[0, min (P_{0}, \hat{P_{0}})]$ ,

\underset{t \in A}{s u p} | \hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) | \to 0

a.s. as n

\to

\infty

Lemma 2

We can define two Brownian bridges $B_{1}^{n} (t)$ and $B_{2}^{n} (t)$ as well as two normal random variable $W_{0}$ and $W_{1}$ , with $W_{i} \sim N (0, P_{i} (1 - P_{i}))$ . $B_{1}^{n} (t)$ , $B_{2}^{n} (t)$ , $W_{0}$ and $W_{1}$ are independent and for $0 < t \leq m i n (P_{0}, \hat{P_{0}})$ , (17)

\begin{aligned} \sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) \\ \overset{d}{\to} \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} + P_{1} (F_{1} (F_{0})^{- 1})^{'} (1 - \frac{t}{P_{0}}) \frac{t}{P_{0}^{2}} W_{2} \\ + \frac{P_{1}}{\sqrt{P_{0}}} (F_{1} (F_{0}^{- 1}))^{'} (1 - \frac{t}{P_{0}}) B_{2}^{n} (1 - \frac{t}{P_{0}}) \\ + \sqrt{λ P_{1}} B_{1}^{n} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) . \end{aligned}

Proof Proof of Lemma 1

(18)

\begin{aligned} | \hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) | \\ \leq | {\hat{P}}_{1} - P_{1} | | 1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ + P_{1} | F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ \leq | {\hat{P}}_{1} - P_{1} | | 1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ + P_{1} | F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ + P_{1} | F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) - F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ \leq 2 | \hat{P_{1}} - P_{1} | + | F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) | \\ + | (F_{1} F_{0}^{- 1})^{'} (1 - \frac{t}{P_{0}}) (P_{0} - \hat{P_{0}}) + o (P_{0} - \hat{P_{0}}) | \end{aligned}

The first term and the last term are controlled by

| \hat{P_{1}} - P_{1} |

and

| \hat{P_{0}} - P_{0} |

respectively. Apply (15) to the second term and we complete the proof.

Proof Proof of Lemma 2

(19)

\begin{aligned} \sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) \\ = \sqrt{n} ({\hat{P}}_{1} - P_{1}) (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ + \sqrt{n} P_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ + \sqrt{n} P_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) - F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \end{aligned}

We treat these three parts respectively. Using the central limit theorem, Equation (16) and Slutsky’s theorem we have: (20)

\begin{aligned} \begin{aligned} \sqrt{n} ({\hat{P}}_{1} - P_{1}) (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ \to \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} \end{aligned} \end{aligned}

(21)

\begin{aligned} \begin{aligned} \sqrt{n} P_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}})) - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ \to \sqrt{λ P_{1}} B_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) \\ + \frac{P_{1}}{\sqrt{P_{0}}} (F_{1} (F_{0}^{- 1}))^{'} (1 - \frac{t}{P_{0}}) B_{2} (1 - \frac{t}{P_{0}}) \end{aligned} \end{aligned}

and (22)

\begin{aligned} \sqrt{n} P_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})) - F_{1} (F_{0}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ \to P_{1} (F_{1} (F_{0})^{- 1})^{'} (1 - \frac{t}{P_{0}}) \frac{t}{P_{0}^{2}} W_{2} . \end{aligned}

Since we have assumed that ${X_{i}}$ , ${Y_{j}}$ , ${L_{0 j}}_{j = 1}^{n}$ and ${L_{1 i}}_{i = 1}^{m}$ are independent, $W_{1}$ , $W_{2}$ , $B_{1}$ and $B_{2}$ are independent and we complete the proof.

Now we begin to prove the theorems in the article.

Proof Proof of Theorem 1

By proving Lemma 1 and Lemma 2, we have obtained the properties of $\hat{L R O C (t)}$ on $[0, m i n (P_{0}, \hat{P_{0}})]$ . Moreover, when $t \in [m a x (P_{0}, \hat{P_{0}}), 1]$ , $\hat{L R O C} (t) - L R O C (t) = \hat{P_{1}} - P_{1}$ whose asymptotic properties are obvious. So the only problem left is that $\hat{L R O C (t)}$ and $L R O C (t)$ have different turning point, i.e. $\hat{P_{0}}$ and $P_{0}$ .

Note that $l_{0} \sim B (n, P_{0})$ , $\hat{P_{0}} = \frac{l_{0}}{n} \to P_{0} a.s. as n \to \infty$ . So we first prove the properties piecewise and then combine them using the convergence of $\hat{P_{0}}$ .

For any $ϵ$ that $0 < ϵ < m i n (P_{0}, 1 - P_{0})$ :

(1)For $t \in [0, P_{0} - ϵ],$ (23)

\begin{aligned} P (lim_{n \to \infty} sup_{t \in [0, P_{0} - ϵ]} | \hat{L R O C (t)} - L R O C (t) | \to 0) \\ = P (lim_{n \to \infty} sup_{t \in [0, P_{0} - ϵ]} | \hat{L R O C (t)} - L R O C (t) | \to 0 lim_{n \to \infty} \hat{P_{0}} < P_{0} - \frac{1}{2} ϵ) \\ + P (lim_{n \to \infty} sup_{t \in [0, P_{0} - ϵ]} | \hat{L R O C (t)} - L R O C (t) | \to 0 lim_{n \to \infty} \hat{P_{0}} \geq P_{0} - \frac{1}{2} ϵ) \\ \geq P (lim_{n \to \infty} sup_{t \in [0, P_{0} - ϵ]} | \hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) | \to 0, lim_{n \to \infty} \hat{P_{0}} \geq P_{0} - \frac{1}{2} ϵ) \\ = 1 \end{aligned}

(2)For

t \in [P_{0} + ϵ, 1],

(24)

\begin{aligned} P (lim_{n \to \infty} sup_{t \in [P_{0} + ϵ, 1]} | \hat{L R O C (t)} - L R O C (t) | \to 0) \\ = P (lim_{n \to \infty} sup_{t \in [P_{0} + ϵ, 1]} | \hat{L R O C (t)} - L R O C (t) | \to 0, lim_{n \to \infty} \hat{P_{0}} > P_{0} + \frac{1}{2} ϵ) \\ + P (lim_{n \to \infty} sup_{t \in [P_{0} + ϵ, 1]} | \hat{L R O C (t)} - L R O C (t) | \to 0 lim_{n \to \infty} \hat{P_{0}} \leq P_{0} + \frac{1}{2} ϵ) \\ \geq P (lim_{n \to \infty} | \hat{P_{1}} - P_{1} | \to 0, lim_{n \to \infty} \hat{P_{0}} \leq P_{0} + \frac{1}{2} ϵ) = 1 \end{aligned}

We then let $ϵ \to 0$ and complete the proof.

Proof Proof of Theorem 2

Similar to the proof of Theorem 1, we first deal with $\hat{L R O C} (t) - L R O C (t)$ piecewise and then combine the results.

for $t \in [0, P_{0}),$ (25)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ) & = P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} \leq \hat{P_{0}}) \\ + P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} > \hat{P_{0}}) \end{aligned}

For the first term, (26)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} \leq \hat{P_{0}}) \\ = P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ, P_{0} \leq \hat{P_{0}}) \\ = P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ) + P (P_{0} \leq \hat{P_{0}}) \\ - P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ or P_{0} \leq \hat{P_{0}}) \end{aligned}

n \to \infty

: (27)

\begin{aligned} (26) & = lim_{n \to \infty} P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ) + 1 - 1 \\ = lim_{n \to \infty} P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ) \end{aligned}

For the second term, as

n \to \infty

, (28)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} > \hat{P_{0}}) \leq P (P_{0} > \hat{P_{0}}) = 0 \end{aligned}

for $t \in (P_{0}, 1],$ (29)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ) & = P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} \geq \hat{P_{0}}) \\ + P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} < \hat{P_{0}}) \end{aligned}

For the first term, (30)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} \geq \hat{P_{0}}) & = P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ, P_{0} \geq \hat{P_{0}}) \\ = P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ) + P (P_{0} \leq \hat{P_{0}}) \\ - P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ or P_{0} \geq \hat{P_{0}}) \end{aligned}

n \to \infty

, (31)

\begin{aligned} (30) & = lim_{n \to \infty} P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ) + 1 - 1 \\ = lim_{n \to \infty} P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ) \end{aligned}

For the second term, (32)

\begin{aligned} P (\sqrt{n} (L R O C (t) - \hat{L R O C (t)}) < δ, P_{0} > \hat{P_{0}}) \leq P (P_{0} > \hat{P_{0}}) \end{aligned}

and as

n \to \infty

lim_{n \to \infty} P (P_{0} > \hat{P_{0}}) = 0

In conclusion, we have (33)

\begin{aligned} lim_{n \to \infty} P (\sqrt{n} (\hat{L R O C} (t) - L R O C (t)) < δ) \\ = & {\begin{aligned} \begin{aligned} lim_{n \to \infty} P (\sqrt{n} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) < δ) \end{aligned}, & t \in (0, P_{0}] \\ lim_{n \to \infty} P (\sqrt{n} (P_{1} - \hat{P_{1}}) < δ), & t \in (P_{0}, 1] \end{aligned} \end{aligned}

Using Lemma 2, the proof is completed.

Proof Proof of Corollary 2

We first rewrite $\hat{AUC} - AUC$ as an integral and then use the dominated convergence theorem and the Ito isometry. (34)

\begin{aligned} \hat{AUC} - AUC & = \int_{0}^{\hat{P_{0}}} (\hat{P_{1}} (1 - \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) d t + \hat{P_{1}} (1 - \hat{P_{0}}) \\ - \int_{0}^{P_{0}} P_{1} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) d t \\ - P_{1} (1 - P_{0}) \\ = (\hat{P_{1}} - P_{1}) - (\int_{0}^{1} \hat{P_{1}} \hat{P_{0}} \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - m)) d m \\ - \int_{0}^{1} P_{1} P_{0} F_{1} (F_{0}^{- 1} (1 - u)) d u) \\ = (\hat{P_{1}} - P_{1}) (1 - P_{0} \int_{0}^{1} F_{1} (F_{0}^{- 1} (1 - u)) d u) \\ + (P_{0} - \hat{P_{0}}) \hat{P_{1}} \int_{0}^{1} F_{1} (F_{0}^{- 1} (1 - u)) d u \\ - \hat{P_{0}} \hat{P_{1}} (\int_{0}^{1} \hat{F_{1}} ({\hat{F_{0}}}^{- 1} (1 - m)) d m - \int_{0}^{1} F_{1} (F_{0}^{- 1} (1 - u)) d u) \end{aligned}

Since

F

and

\hat{F}

are distribution functions. With Lemma 2, the central limit theorem and the dominated convergence theorem we obtain that

\sqrt{n} (\hat{AUC} - AUC)

is asymptotically normal. (35)

\begin{aligned} \sqrt{n} (\hat{AUC} - AUC) \\ \to \sqrt{λ} W_{1} (1 - P_{0} \int_{0}^{1} F_{1} (F_{0}^{- 1} (1 - u)) d u) \\ + W_{2} P_{1} \int_{0}^{1} F_{1} (F_{0}^{- 1} (1 - u)) d u \\ + P_{0} P_{1} \int_{0}^{1} \sqrt{\frac{λ}{P_{1}}} B_{1} (F_{1} F_{0}^{- 1} (t)) + \frac{1}{\sqrt{P_{0}}} (F_{1} F_{0}^{- 1})^{'} (t) B_{2} (t) d t \end{aligned}

Since

\hat{P_{0}}

\hat{P_{1}}

\hat{F}

and

\hat{G}

are independent, the asymptotic variance of

\sqrt{n} (\hat{AUC} - AUC)

can be computed as: (36)

\begin{aligned} σ^{2} = & λ {(1 - P_{0} \int F_{1} F_{0}^{- 1})}^{2} P_{1} (1 - P_{1}) \\ + P_{1}^{2} {(\int F_{1} F_{0}^{- 1})}^{2} P_{0} (1 - P_{0}) \\ + λ P_{1} P_{0}^{2} | | F_{1} F_{0}^{- 1} (m) | | + P_{0} P_{1}^{2} | | F_{0} F_{1}^{- 1} (m) | | \end{aligned}

where

| | h | | = \int h^{2} - (\int h)^{2}

Proof Proof of Theorem 3

We have already derived the asymptotic process of $R_{n} (t)$ and $R_{n}^{*} (t)$ .

$t \in (0, P_{0})$ , (37)

\begin{aligned} R_{n} (t) & \to \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} \\ + P_{1} (F_{1} (F_{0})^{- 1})^{'} (1 - \frac{t}{P_{0}}) \frac{t}{P_{0}^{2}} W_{2} \\ + \sqrt{λ P_{1}} B_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) \\ + \frac{P_{1}}{\sqrt{P_{0}}} (F_{1} (F_{0}^{- 1}))^{'} (1 - \frac{t}{P_{0}}) B_{2} (1 - \frac{t}{P_{0}}) \end{aligned}

and (38)

\begin{aligned} R_{n}^{*} (t) & \to \sqrt{λ} (1 - \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) W_{1}^{*} \\ + \hat{P_{1}} (\tilde{F_{1}} {\tilde{(F_{0})}}^{- 1})^{'} (1 - \frac{t}{\hat{P_{0}}}) \frac{t}{{\hat{P_{0}}}^{2}} W_{2}^{*} \\ + \sqrt{λ \hat{P_{1}}} B_{1} (\tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ + \frac{\hat{P_{1}}}{\sqrt{\hat{P_{0}}}} (\tilde{F_{1}} ({\tilde{F_{0}}}^{- 1}))^{'} (1 - \frac{t}{\hat{P_{0}}}) B_{2} (1 - \frac{t}{\hat{P_{0}}}) \end{aligned}

We will treat each term respectively.

(39)

\begin{aligned} \sqrt{λ} (1 - \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) W_{1}^{*} \\ - \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} \\ = (\sqrt{λ} (1 - \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) \\ - \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}})))) W_{1} \\ + \sqrt{λ} (1 - \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) (W_{1}^{*} - W_{1}) \end{aligned}

(40)

\begin{aligned} | P (W_{1}^{*} \leq u) - P (W_{1} \leq u) | \\ = | \int_{u \sqrt{P_{1} (1 - P_{1})}}^{u \sqrt{\hat{P_{1}} (1 - \hat{P_{1}})}} \frac{1}{\sqrt{2 π}} e^{- \frac{1}{2} x^{2}} d x | \\ = O (| P_{1} - \hat{P_{1}} |) = O (\frac{1}{\sqrt{n}}) \end{aligned}

From Chapter 2 of Nonparametric Functional Estimation,²⁷ under the condition $n h^{5} l o g^{2} n \to 0$ as $n \to 0$ we have:

| \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (t)) - F_{1} (F_{0}^{- 1} (t)) | = O (\frac{l o g n}{\sqrt{n} h_{n}})

Then: (41)

\begin{aligned} \sqrt{λ} (1 - \tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) W_{1}^{*} - \sqrt{λ} (1 - F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) W_{1} = O (\frac{l o g n}{\sqrt{n} h_{n}}) \end{aligned}

Using a similar approach, we can deal with other terms; here, we directly give the convergence rates of them:

\begin{aligned} | P_{1} - \hat{P_{1}} | & = O (\frac{1}{\sqrt{n}}) \\ | P_{0} - \hat{P_{0}} | & = O (\frac{1}{\sqrt{n}}) \\ | P (W_{2}^{*} \leq u) - P (W_{2} \leq u) | & = O (| P_{0} - \hat{P_{0}} |) = O (\frac{1}{\sqrt{n}}) \end{aligned}

From Bertail et al.²¹ we have:

\begin{aligned} (F_{1} F_{0}^{- 1})^{'} (1 - \frac{t}{P_{0}}) - (\tilde{F_{1}} {\tilde{(F_{0})}}^{- 1})^{'} (1 - \frac{t}{P_{0}}) = O (\frac{l o g (h_{n}^{- 1})}{\sqrt{n h_{n}}}) \\ s u p | B_{1} (\tilde{F_{1}} ({\tilde{F_{0}}}^{- 1} (1 - \frac{t}{\hat{P_{0}}}))) - B_{1} (F_{1} (F_{0}^{- 1} (1 - \frac{t}{P_{0}}))) | = O (\frac{l o g n}{\sqrt{n}}) \end{aligned}

With some algebra, Taylor expansion and

δ

-method, we can show that:

| R_{n}^{*} - R_{n} | = O (\frac{l o g n}{\sqrt{n} h_{n}})

holds uniformly when

t \in (0, P_{0}]

, so

sup_{u \in R} | P^{*} (| | R_{n}^{*} | |_{\infty} \leq u) - P (| | R_{n} | |_{\infty} \leq u) = O (\frac{l o g (n)}{\sqrt{n} h_{n}})

holds for

t \in (0, P_{0}]

$t \in (P_{0}, 1]$ , we already have (42)

R_{n} (t) = \sqrt{λ} W_{1}

and in a similar way, we have (43)

R_{n}^{*} (t) = \sqrt{λ} W_{1}^{*}

We have shown that:

| P (W_{1}^{*} \leq u) - P (W_{1} \leq u) | = O (\frac{1}{\sqrt{n}})

Then it is obvious that

sup_{u \in R} | P^{*} (| | R_{n}^{*} | |_{\infty} \leq u) - P (| | R_{n} | |_{\infty} \leq u) | = O (\frac{l o g (n)}{\sqrt{n} h_{n}})

holds for

t \in (P_{0}, 1]

See Figures 6 and 7. Figure 6.

Integrate bias and MSE for lognormal distribution as N (top) and $λ$ (bottom) change.

Figure 7.

Integrate bias and MSE for t distribution as N (top) and $λ$ (bottom) change.

References

Liu

Sun

Zhou

. A novel regression method for the analysis of multireader multicase-free-response receiver operating characteristics studies. Stat Med 2022; 41: 3022–3038.

Baker

Bogoni

Obuchowski

et al. Computer-aided detection of colorectal polyps: can it improve sensitivity of less-experienced readers? preliminary findings. Radiology 2007; 245: 140–149.

Starr

Metz

Lusted

et al. Visual detection and localization of radiographic images. Radiology 1975; 116: 533–538.

Bunch

Hamilton

Sanderson

et al. A free response approach to the measurement and characterization of radiographic observer performance. In Application of Optical Instrumentation in Medicine VI, volume 127. SPIE, pp.124–135.

Zhou

McClish

Obuchowski

. Statistical methods in diagnostic medicine. John Wiley & Sons, 2009.

Seltzer

Swensson

Nawfel

et al. Visualization and detection–localization on computed tomographic images. Invest Radiol 1991; 26: 285–294.

Gifford

Wells

King

. LROC analysis of human detection performance in PET and time-of-flight PET. In Medical Imaging 1999: Image Perception and Performance, volume 3663. SPIE, pp.187–198.

Surti

Scheuermann

El Fakhri

et al. Impact of time-of-flight pet on whole-body oncologic studies: a human observer lesion detection and localization study. J Nucl Med 2011; 52: 712–719.

Kadrmas

Casey

Conti

et al. Impact of time-of-flight on pet tumor detection. J Nucl Med 2009; 50: 1315–1323.

10.

Swensson

. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 1996; 23: 1709–1725.

11.

Popescu

. Nonparametric roc and lroc analysis. Med Phys 2007; 34: 1556–1564.

12.

Tang

Balakrishnan

. A random-sum Wilcoxon statistic and its application to analysis of ROC and LROC data. J Stat Plan Inference 2011; 141: 335–344.

13.

Bandos

Rockette

Song

et al. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 2009; 65: 247–256.

14.

Chakraborty

. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989; 16: 561–568.

15.

Popescu

. Nonparametric signal detectability evaluation using an exponential transformation of the froc curve. Med Phys 2011; 38: 5690–5702.

16.

Song

Zhou

. A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data. Biostatistics 2005; 6: 303–312.

17.

Chakraborty

Berbaum

. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004; 31: 2313–2330.

18.

Chakraborty

. Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008; 15: 1554–1566.

19.

Chakraborty

. Observer performance methods for diagnostic imaging: foundations, modeling, and applications with r-based examples. Boca Raton, FL: CRC Press, 2017.

20.

Hsieh

Turnbull

. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat 1996; 24: 25–40.

21.

Bertail

Clémençcon

Vayatis

. On bootstrapping the roc curve. Adv Neural Inf Process Syst 2008; 21: 137–144.

22.

Chakraborty

Winter

. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology 1990; 174: 873–881.

23.

Vickers

van Calster

Steyerberg

. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Prognostic Res 2019; 3: 18.

24.

Macskassy

Provost

Rosset

. ROC confidence bands: an empirical evaluation. In Proceedings of the 22nd international conference on Machine learning. pp.537–544.

25.

Hall

Hyndman

Fan

. Nonparametric confidence intervals for receiver operating characteristic curves. Biometrika 2004; 91: 743–750.

26.

Ghosal

. Strong approximations for resample quantile process and application to ROC methodology. J Nonparametr Stat 2008; 20: 229–240.

27.

Rao

. Nonparametric functional estimation. Academic press, 2014.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.01 MB