An efficient glomerular object locator for renal whole slide images using proposal-free network and dynamic scale evaluation method

Abstract

In the diagnosis of chronic kidney disease, glomerulus as the blood filter provides important information for an accurate disease diagnosis. Thus automatic localization of the glomeruli is the necessary groundwork for future auxiliary kidney disease diagnosis, such as glomerular classification and area measurement. In this paper, we propose an efficient glomerular object locator in kidney whole slide image(WSI) based on proposal-free network and dynamic scale evaluation method. In the training phase, we construct an intensive proposal-free network which can learn efficiently the fine-grained features of the glomerulus. In the evaluation phase, a dynamic scale evaluation method is utilized to help the well-trained model find the most appropriate evaluation scale for each high-resolution WSI. We collect and digitalize 1204 renal biopsy microscope slides containing more than 41000 annotated glomeruli, which is the largest number of dataset to our best knowledge. We validate the each component of the proposed locator via the ablation study. Experimental results confirm that the proposed locator outperforms recently proposed approaches and pathologists by comparing $F_{1}$ and run time in localizing glomeruli from WSIs at a resolution of 0.25 μm/pixel and thus achieves state-of-the-art performance. Particularly, the proposed locator can be embedded into the renal intelligent auxiliary diagnosis system for renal clinical diagnosis by localizing glomeruli in high-resolution WSIs effectively.

Keywords

Glomerulus object locator whole slide image proposal-free network dynamic scale evaluation

1. Introduction

Chronic kidney disease (CKD) is much more widespread than people realize. It has the characteristic of substantial morbidity and mortality, but low awareness and prevention rate. Glomeruli are spherical clusters of capillaries (The examples of glomeruli in pathological images were shown in Fig. 1) which are responsible for expelling unnecessary substances of human body. An initial solution to diagnose CKD is that pathologists need to observe glomerular injury in the renal biopsy microscope slides by optical microscope [30]. Unfortunately, there are few qualified pathologists for diagnosing such diseases because of nonstandard training system, low wages, and high risk [17]. In addition, the statistical work of glomeruli in the pathological biopsy slides is extremely complicated and the location of the glomerulus is flexible, which results that the diagnostic process is extremely laborious and highly depend on experience of pathologists. To reduce the risk of misdiagnosis, pathologists have to conduct a thorough inspection of the whole biopsy slide which makes the diagnosis quite cumbersome [10]. Therefore, it is urgent to design automatic technology in CKD diagnosis.

Fig. 1.

Glomeruli are labeled through red boxes in pathological images.

The development of digital pathology allows pathologists to digitalize a biopsy slide to a high-resolution whole slide image (WSI), a new form of high-resolution image storage, for diagnosis [6] based on image data obtained by scanning pathological tissue samples. Computer aided diagnostics in digital pathology can not only alleviate pathologists’ workloads, but also help to reduce the diagnostic mistakes. Automatic analysis on the WSI has become the mainstream in digital pathology and achieved some promising results. To date, many researches are conducted to implement classification, segmentation and recognition of the tumor [3,12] based on the WSIs data. The implementation of these technologies in the clinical diagnosis and their impact on medical field attracted more attention [4]. For many CKDs, the glomerulus is the most important region of interest and thus localization of glomerulus is the significant groundwork to future automated glomerular analysis such as classification and area measurement [7]. However in the localization of glomeruli, the critical limitations are also obvious when adopting the current machine learning and deep learning algorithms directly: Firstly, high-resolution WSIs which have too many pixels are difficult to be processed by current algorithms directly. If WSIs are down-sampled to the size that computers can process, a lot of fine-grained information lost will affect the localization results. Secondly, existing glomerular locators adopt proposal-based network, such as Faster R-CNN [21], to ensure the accuracy. As a consequence, in the proposal phase, tremendous of proposals are selected due to the high-resolution of WSIs, which leads to the proposal-based network involving a huge number of parameters to learn.

To overcome the aforementioned defects, we propose an efficient glomerular object locator (GOL) by using intensive proposal-free network and dynamic scale evaluation method for 400× WSIs. Motivated by the previous works of [20,29], we design an intensive proposal-free by integrating a more powerful loss function, Distance-IoU loss function [29], into the proposal-free network. Compared with the proposal-based network, such as Faster R-CNN [21], the proposed intensive proposal-free network is not only more free to input image sizes, but also faster to process images. In addition, we employ the dynamic scale evaluation to process high-resolution images in the evaluation stage, which can effectively improve the speed and accuracy of processing high-resolution WSIs through optimal scale evaluation. To evaluate the performance of the proposed GOL model, a dataset including more than 41000 glomeruli annotated from 1204 human renal whole slide images are collected and used to conduct extensive experiments. Experimental results confirm that the proposed GOL locator achieves competing overall F1-score of $98.1 %$ in localizing glomeruli, which is much higher than that of compared methods and pathologists. Meanwhile, the localizing speed of the proposed method is much faster than the pathologists, which confirms the proposed method can be applied to the actual glomeruli localization due to its powerful real-time performance.

The reminder of this paper is structured as follows. Section 2 sketches the related works. Section 3 introduces the proposed network structure and training process of the proposed method in detail. Section 4 gives the experimental results and Section 5 draws the conclusion.

2. Related work

In recent years, a series of approaches based on computer vision technology have been proposed to process renal pathological images by scanning renal biopsy microscope slides to WSIs through digital slide scanner. One of the most fundamental steps is to localize the region of interest in the WSIs, such as glomeruli, renal tubules and interstitial cells. In this section, we briefly introduce the glomerular localization methods in terms of traditional methods and deep learning based methods.

2.1. Traditional methods

Traditional methods usually extract hand-crafted features by machine learning algorithms. For example, Kakimoto et al. [13] proposed a method of localization glomeruli based on the support vector machine (SVM) [26]. For this method, the candidate sub-image is first cropped by sliding a fixed detection window on the renal WSI. Then the HOG feature vector [5] is extracted from the sub-image and used as the pre-trained SVM classifier to predict whether a glomerulus exists in the center of the cropped sub-image. Afterward, Kato et al. [14] developed an improved HOG features extractor named as segmental HOG that adaptively fit to input images and acquired robustness for the localization of the glomeruli. Recently, Simon et al. [25] developed an automated detector of glomeruli. The authors utilized local binary pattern (LBP) [28] as feature descriptors and calculate multi-radial color LBP, which is used to train a SVM classifier. However, renal biopsy microscope slides are actually the tangential 2D sectioning levels of the spherical 3D structure. Different tangent planes lead to different glomerular structures. Moreover, due to the different staining methods, the appearance of tissues are also diverse. Manually defined features are difficult to capture the changes in glomerular morphology and color, so it do not have good generalization to locate heterogeneous glomeruli.

2.2. Deep learning based methods

With the advance of deep neural networks, deep learning-based models are proposed and achieve promising performance in image processing field, which can learn deeper and richer representations than hand-crafted representations. By utilizing the automatic feature extraction method, many deep neural networks, including SPPnet [11], You Only Look Once (YOLO) [18] and its improved versions [19,20], Faster R-CNN [21] and other deep learning methods have been proposed for object detection. Some previous studies proposed the glomerular locators via adopting deep learning methods. Gallego et al. [8] proposed a method of slicing each WSI to many small patches by using a fixed-size sliding window. Then they classified the patches which were glomeruli and background by convolution neural networks(CNN). Kawazoe et al. [15] firstly used object localization model, Faster R-CNN, to detect the glomeruli in WSIs with multiple stains of human renal biopsy slices, which improves performance significantly. Liu et al. [16] improved the Faster R-CNN algorithm to achieve better results. Bukowy et al. [2] constructed Faster R-CNN with modified AlexNet model to detect glomeruli in patches. Previous works solve the problem of high-resolution processing through slicing each WSI into many small patches by sliding window method and then patch-wise searching the location of glomeruli [1,2,15,16]. In this way, nevertheless, it also takes a long time in processing a high-resolution WSI for invoking the locator frequently. Therefore, all recently developed methods have not yet reached the desired performance in glomerular localization and have high time consumption.

3. Material and method

In this section, we first introduce the dataset of glomeruli, then present the whole architecture of our proposed glomerular object locator (GOL) based on two important design principles.

3.1. Dataset description

Firstly, the 1250 renal biopsy samples we used were used to the periodic acid silver methenamine(PASM) stains used for staining structure containing more details of mesangial and basement membranes. All samples were from the Second Hospital of Shanxi Medical University (SHSXMU) between 2014 and 2019 and from the Shanxi Provincial People’s Hospital (SXPPH) between 2017 and 2019. The personal information of patients involved in the data was removed. Then, the biopsy microscope slides were digitized to WSIs by using a KF-PRO-005-EX digital slide scanner (KFbio, Ningbo, China) with a $40 \times$ objective and $10 \times$ eyepiece at a resolution of 0.25 μm/pixel and stored in .SVS format at full resolution. Finally, 1204 WSIs containing complete renal tissue were preserved and used as analysis data after data cleaning by pathologists in total. The data collected was approved by the Ethics Committee on Human Research of the Shanxi Provincial People’s Hospital (No. 127) and the Second Hospital of Shanxi Medical University (YX. 026).

3.1.1. Training dataset

To ensure the universally applicability of training model, 955 slides from two hospitals and different years were selected as the train/validation dataset. The used $400 \times$ WSIs have very high quality with average size of $20000 \times 20000$ pixels. However, the high resolution WSIs cannot be directly used to train the convolution neural network. In the training phase, therefore, we down-sampled the $400 \times$ WSIs at the resolution of 0.25 μm/pixel to $200 \times$ WSIs at the resolution of 0.50 μm/pixel with the help of Openslide [9] for improving the processing efficiency of model. All of $200 \times$ WSIs were cropped into 5646 patches by sliding a $3000 \times 3000$ window in an overlapping fashion. These patches with the resolution of 0.50 μm/pixel were clear enough to identify the glomerulus in renal biopsy for pathologists. Glomeruli, including sclerosing glomeruli, were boxed in the patches using Labelme [23] by three pathologists. Furthermore, the annotations were checked by two chief pathologists via deleting wrong annotations, adding missing annotations and adjusting the scope of the annotated boxes. Finally, the whole dataset included 41440 glomeruli. Figure 2 shows the process of generating training dataset.

Fig. 2.

Process of generating training dataset.

3.1.2. Testing dataset

For pathological examination of renal biopsy, different hospitals have different staining methods even with the same staining type. Meanwhile, the stained slides would fade over time. Figure 3 shows some slides images collected from different hospitals and years. We can observe that these slides have different colors with each other. Therefore, it is necessary for us to evaluate the performance of our model in different slides’ appearance.

Fig. 3.

Examples of slides from different hospitals and years.

Overall, 250 slides were evenly selected from 2016 to 2018 in SHSXMU and SXPPH as the testing dataset to evaluate the performances of the models. Then we divided the testing dataset into five parts by different years and hospitals. Each testing dataset had 50 randomly selected WSIs. The testing dataset collected from different hospitals and years are summarized in Table 1.

Table 1

Testing dataset from different hospitals and years

	Hospital	Year	Glomeruli
Test-1	SHSXMU	2016	1123
Test-2	SHSXMU	2017	1165
Test-3	SHSXMU	2018	1077
Test-4	SXPPH	2017	1321
Test-5	SXPPH	2018	1263
Total			5949

3.2. Architecture of glomerular object locator

The overall framework of glomerular object locator is shown in Fig. 4. It consists of two parts involving the training phase and the testing phase. In the training phase, we employed one-stage localization network to learn glomerular features efficiently via combining multi-scale and proposal-free localization principle with more efficient Distance-IoU loss function. In order to keep the compression scale of WSIs,in the testing phase, same with that of patches in the training dataset, we proposed a dynamic scale evaluation (DSE) method to search the most appropriate compression scale for each WSI in the testing dataset before being fed into the well trained module. The used designed principle in each part will be introduced in the following.

Fig. 4.

Overall framework of glomerular object localizer.

3.2.1. Intensive proposal-free

Existing deep learning based glomerular object localization methods usually rely on region-proposal-network (RPN), such as Faster R-CNN [15], to generate relatively region of interests. These proposal-based networks have some inevitable limitations in processing the WSIs. Firstly, the RPN network has low localization speed in processing WSIs with high-resolution. Secondly, the proposal-based methods are difficult to converge successfully without pre-trained models [24]. However, there are few public datasets about renal pathology up to now, and the pre-trained process cannot be applied for renal pathology WSIs which have essential difference with origin images. Contrarily, proposal-free object localization networks, such as YOLO [18], handle the object localization problem with a regression analysis. This one-stage method (without RPN) is faster and more flexible to process WSIs with high-resolution, fine-grained features from scratch.

Various proposal-free localization frameworks predict the bounding box of locating object through $l_{p}$ -norm loss function to define the difference between bounding box and ground truth. However, the optimization of $l_{p}$ -norm loss is not equivalent to intersection over Union (IoU) [22]. Although IoU loss is used in Unitbox [27], and then GIoU [22] loss is proposed to solve the problem of gradient vanishing for non-overlapping boxes, these methods are still facing the issues of difficult convergence and imprecise regression [29].

To overcome these limitations, a more powerful loss function named as Distance-IoU (DIoU) [29] is proposed and its expression is formulated as $\begin{matrix} (1) & L_{DIoU} = 1 - DIoU, \end{matrix}$ with $\begin{matrix} (2) & DIoU = IoU - \frac{ρ^{2} (b, b^{gt})}{c^{2}}, \end{matrix}$ where b is the center of predicted box B, $b^{gt}$ is the center of ground truth $B^{gt}$ , $ρ (\cdot)$ is the calculation of Euclidean distance, and c is the diagonal distance of the smallest enclosing box that can contain both predicted box and ground truth. The DIoU loss function avoids divergence in the training process through considering the distance, the overlap rate and the scale between B and the $B^{gt}$ .

According to the above analysis, we constructed a three-scale proposal training framework from the YOLO-v3 [20] and replaced the loss function and non-maximum suppression (NMS) by the powerful DIoU loss function and DIoU NMS, respectively. It increases the aggregate localization speed and thus it is effective for improving the regression accuracy in glomerular localization. The intensive proposal-free of GOL is detailed in Fig. 5.

Fig. 5.

Intensive proposal-free module of GOL.

3.2.2. Dynamic scale evaluation

Due to the limitation of computing power, existing object localization methods can not process directly the high-resolution and fine-grained WSIs. Usually, the WSI with gigapixel are compressed by down-sampling before feeding into well-trained model. It is important to set an appropriate down-sampled coefficient to obtain high performance in localizing accuracy. In other word, excessive and inadequate down-sampling both make the ability of localizing the glomeruli unstable. Meanwhile, it is difficult to find a suitable down-sampled coefficient for all WSIs with different size. Alternatively, some methods slice the large WSI into many small patches by sliding windows to train and test the modules. However, these methods are inefficient since they have to invoke the locator many times to process each WSI. In addition, due to the slicing method may cut out incomplete glomeruli (see Fig. 6(a)), the locator would deem them as morphologically incomplete glomeruli (see Fig. 6(b)), resulting that the locator has poor performance.

Fig. 6.

Examples of incomplete glomeruli. (a) Incomplete glomerular caused by slicing; (b) Morphologically incomplete glomerular.

To overcome the aforementioned defects in evaluation phase, we proposed a dynamic scale evaluation (DSE) method, which can make the tested WSI have consistent compressed scale with the training process. We first calculate the compress scale of width and height for training image by $\begin{matrix} (3) & α = \frac{w_{train}}{w_{compress} / s^{n}}, β = \frac{h_{train}}{h_{compress} / s^{n}}, \end{matrix}$ where $w_{train}$ and $h_{train}$ denote the width and height of patch in the training dataset, $w_{compress}$ and $h_{compress}$ represent the width and height of compressed patch before sending to locator in training phase. s and n denotes the down-sampled step size and frequency in proposal-free module. For our proposed model, we set the down-sampling size step as 2 and use 5 layers of down-sampling. The final width and height of compressed patch for each tested WSI can be calculated as $\begin{matrix} (4) & w_{final} = s^{n} \cdot ⌊ \frac{w_{test}}{α} ⌋, h_{final} = s^{n} \cdot ⌊ \frac{h_{test}}{β} ⌋, \end{matrix}$ where $w_{test}$ and $h_{test}$ denote the width and height of tested WSI, $⌊ \cdot ⌋$ is the floor function.

The proedure of DSE is illustrated in Fig. 7. For each tested WSI, we first compresses it by the DSE method, and obtain a compressed image. Then, the compressed image is fed into the locator to infer the location of glomeruli. Finally, the compressed image with extracted bounding box by locator are mapped to original WSI through multiplying the corresponding down-sampled coefficient. The DSE method ensures that the object localization scale of the model in the testing phase keeps consistent with that of learning in the training phase. Therefore, the DSE method is particularly effective in localizing object from images with variable size and high-resolution such as medical WSI and high-resolution satellite imagery.

Fig. 7.

Procedure of dynamic scale evaluation.

4. Experimental results

In this section, the proposed GOL is evaluated on the testing dataset and compared with related locators and pathologists in performance of localization accuracy and speed.

4.1. Experimental settings

We constructed the training module of GOL based on a Linux serve platform with 4 NVIDIA Tesla V100s. We down-sampled the $3000 \times 3000$ pixel/patches in the training dataset to the image size of $800 \times 800$ pixel/image. It ensures the speed of training and not affects the general glomerular morphological characteristics. Then anchors were renewedly designed to use prior knowledge. The anchor boxes were set to $(10, 13)$ , $(29, 25)$ , $(40, 35)$ , $(50, 39)$ , $(64, 66)$ , $(55, 47)$ , $(116, 90)$ , $(156, 198)$ , $(373, 326)$ based on k-means. Carefully designed smaller boxes could achieve great performance since the size of glomeruli are different from natural object. The training data were enhanced by changing saturation, exposure and hue for the aim of well-training the network. For the hyperparameter setting, the maximum batch was set to 52000, the initial learning rate was set to $7 . 5^{- 3}$ , decrease to $7 . 5^{- 4}$ after 40000 batches and decrease to $7 . 5^{- 5}$ after 45000 batches, the batch size was assigned as 8, and subdivisions was set to 4 to save memory. Weight decay and momentum were set to $2 . 5^{- 4}$ and 0.9, respectively.

4.2. Evaluation metrics

In our experiments, we employ the well-known metrics including $precision$ , $recall$ and $F_{1}$ -score to evaluate the localization performance of the proposed method and these metrics are listed as follows: $\begin{array}{l} (5) & Precision = \frac{TP}{TP + FP}, \\ (6) & Recall = \frac{TP}{TP + FN}, \\ (7) & F_{1} = \frac{2 Recall \times Precision}{Recall + Precision}, \end{array}$ where $TP$ , $FP$ and $FN$ are true positive, false positive and false negative, respectively. True positive means that localization the glomerulus correctly detect into the glomerulus, false positive indicates that localization other object such as background mistakenly detect into the glomerulus and false negative denotes that glomerulus is missed in the localization.

4.3. Ablation study

In this subsection, we ablate various design choices for our proposed GOL framework.

Effectiveness of powerful loss function. We trained two proposal-free models by using the same parameters setting excepting the selection of loss function to compare the performance. In the training phase, the loss curves of the two models during training are shown in Fig. 8. We can observe that the DIoU accelerates the convergence of training, and decreases the loss value by 0.6 than that of original proposal-free network.In the testing phase, the values of three metrics are listed in Table 2. We can see that the proposal-free network with DIoU achieves 15% higher $F_{1}$ than that of original proposal-free network in similar scale.

Effectiveness of DSE. We compare the cases with DSE and the cases without DSE (fixed input image size) in the testing phase. As shown in Table 2, the cases with DSE achieves 14.3% and 17.4% percent higher $F_{1}$ than ones of original proposal-free network and intensive proposal-free network with fixed input image sizes, respectively. Besides, the time consumption of models are shown in the last four columns of Table 2. The DSE shorten the run time of localization a glomerulus from 0.74 s to 0.37 s. The average run time of localization a whole WSI is 3.88 s which is shorter than that of fixed input image size, although the maximum run time is a little bit longer.

Fig. 8.

Comparison of MSE and DIoU loss function curves.

Table 2

Ablation study

DIoU	DSE	$Recall$	$Precision$	$F_{1}$	${Time}_{avg}$	${Time}_{max}$	${Time}_{min}$	${Time}_{perglo}$
		0.712	0.983	0.823	12.46 s	20.78 s	8.69 s	0.74 s
	✓	0.944	0.991	0.966	8.58 s	23 s	1.93 s	0.37 s
✓		0.711	0.951	0.807	12.46 s	20.78 s	8.69 s	0.74 s
✓	✓	0.970	0.992	0.981	8.58 s	23 s	1.93 s	0.37 s

To visually compare the localization results, we display the localization results by original proposal-free work, proposal-free network using dynamic scale evaluation (proposal-free+DSE), proposal-free network using DIoU loss function and DIoU non-maximum suppression (intensive proposal-free) and ours GOL.The direct comparison of are shown in Fig. 9. From the last column of Fig. 9, we observe that our proposed GOL achieves the best localization performance via merging DIoU and DSE.

Fig. 9.

Comperison of the localization results. (a) original proposal-free work. (b) proposal-free network using dynamic scale evaluation (proposal-free+DSE). (c) proposal-free network using DIoU loss function and DIoU non-maximum suppression (intensive proposal-free). (d) GOL.

4.4. Performance on different testing datasets

To show an universal applicability of the model for slices from different hospitals and in different years, GOL has been evaluated in five testing datasets. The values of three metrics for the proposed model are shown in Table 3. We can see that $F_{1}$ metric for all five testing datasets are greater than 0.96. whereas, $F_{1}$ metric for the first four testing datasets is greater than 0.98. The $F_{1}$ metric for test-5 is slightly lower than others because the color of glomeruli is more similar with the background in these slides for poor staining method. In a word, our GOL has great performance and can avoid the impact of staining method with differences and slides fading over time.

Table 3
Performance on five testing datasets from different hospitals and years

Datasets Recall Precision $F_{1}$

Test-1 0.986 0.997 0.991

Test-2 0.974 0.990 0.982

Test-3 0.975 0.988 0.981

Test-4 0.976 0.998 0.989

Test-5 0.938 0.985 0.961

Average 0.970 0.992 0.981

Datasets	Recall	Precision	$F_{1}$
Test-1	0.986	0.997	0.991
Test-2	0.974	0.990	0.982
Test-3	0.975	0.988	0.981
Test-4	0.976	0.998	0.989
Test-5	0.938	0.985	0.961
Average	0.970	0.992	0.981

4.5. Performance comparison with related works

In this section, we compared the performance of GOL with recently published approaches including traditional and deep learning methods. The information of dastasets from previous works is shown in Table 4. All recently published approaches down-sampled WSIs to at least $200 \times$ because the original $400 \times$ WSIs are too large to be directly processed efficiently for these methods. Furthermore, we construct a largest dataset to the best of our knowledge, including 41440 glomeruli after annotating. The detailed results are shown in Table 5. We can observe that the performance of GOL is slightly better than relevant proposed methods with $F_{1}$ of 0.981. Furthermore, the run time of GOL is conspicuously better than the locators based on Faster R-CNN [15] and LBP [25]. The results confirm that our proposed method has advantage over other competing methods.

Table 4
Dataset comparison with related works

Methods Class of WSI Amount of WSI Amount of glomeruli Magnification

RHOG+SVM [14] Rat renal tissue 20 – –

mrcLBP+SVM [25] Rat and human renal tissue 55 8673 200×

Classification of sliding window [8] Human renal biopsy 108 4300 200×

Faster R-CNN with inception v2 [15] Human renal biopsy 800 33409 50×

Faster R-CNN with ResNet [16] Human renal biopsy 100 4506 50×

Faster R-CNN with modified AlexNet [2] Rat and human renal tissue 72 23540 200×

GOL Human renal biopsy 1204 41440 400×

Methods	Class of WSI	Amount of WSI	Amount of glomeruli	Magnification
RHOG+SVM [14]	Rat renal tissue	20	–	–
mrcLBP+SVM [25]	Rat and human renal tissue	55	8673	200×
Classification of sliding window [8]	Human renal biopsy	108	4300	200×
Faster R-CNN with inception v2 [15]	Human renal biopsy	800	33409	50×
Faster R-CNN with ResNet [16]	Human renal biopsy	100	4506	50×
Faster R-CNN with modified AlexNet [2]	Rat and human renal tissue	72	23540	200×
GOL	Human renal biopsy	1204	41440	400×

Table 5

Performance comparison with related works

Model	Recall	Precision	$F_{1}$	${Time}_{avg}$	${Time}_{max}$	${Time}_{min}$
RHOG+SVM [14]	0.911	0.777	0.838	–	–	–
mrcLBP+SVM [25]	0.761	0.917	0.832	120s	–	–
Classification of sliding window [8]	1.00	0.881	0.937	–	–	–
Faster R-CNN with inception v2 [15]	0.929	0.977	0.952	64s	364s	2s
Faster R-CNN with ResNet [16]	0.918	0.939	0.928	–	–	–
Faster R-CNN with modified AlexNet [2]	0.969	0.968	0.968	–	–	–
GOL	0.970	0.992	0.981	8.58s	23.00s	1.93s

4.6. Performance comparison with pathologists

We compared the GOL with two pathologists in term of localization accuracy and time consumption per glomerular. Firstly, 250 slides in the testing dataset are tested by a primary pathologist with two years of working experience and a senior pathologist with ten years of working experience. Localization results from pathologists are provided by the Shanxi Provincial People’s Hospital. The comparison results of average speed in each glomerulus and performance are given in Table 6. The GOL achieves higher performance in $F_{1}$ metric and the faster processing speed in localizing glomeruli than pathologists. Meanwhile, we note that some glomeruli influenced by sectioning (tangential 2D sectioning levels of the spherical 3D structure) are especially easy to be missed by pathologists (see Fig. 10) in the experiment, but our proposed method can successfully localizing them. Therefore, the application of GOL can significantly improve the work efficiency of pathologists and reduce the misdiagnosis rate for patients with renal disease.

Table 6
Performance comparison with pathologists

Time per glomerulus Recall Precision $F_{1}$

Primary pathologist 22.32 s 0.822 1.000 0.902

Senior pathologist 11.50 s 0.933 1.000 0.965

GOL 0.37 s 0.970 0.992 0.981

	Time per glomerulus	Recall	Precision	$F_{1}$
Primary pathologist	22.32 s	0.822	1.000	0.902
Senior pathologist	11.50 s	0.933	1.000	0.965
GOL	0.37 s	0.970	0.992	0.981

Fig. 10.

Examples of glomeruli easy to be missed.

5. Conclusion

In this work, we proposed an efficient glomerular object locator named as GOL by combining intensive proposal-free network and DSE method. The intensive proposal network and the DSE method can effectively improve the speed and accuracy of processing high-resolution WSIs. We constructed a large dataset of 1204 high-quality renal pathological WSIs with PASM stain containing 41440 glomeruli. Experimental results confirm that GOL has universal applicability for different WSIs from different hospitals and years and achieves higher F-measures of 0.981 in glomerular localization compared with recently proposed approaches and pathologists. In addition, GOL locates and slices all glomeruli from each WSI only in average time of 8.58 seconds, and one glomerulus in an average time of 0.374 seconds. Thus, the proposed method can be embedded into the intelligent auxiliary diagnosis system to assist pathologists in clinical disease diagnosis. In the future, we would continue to implement the classification of glomerular based on the GOL and expand the GOL to easily deal with other stained pathology slices.

Footnotes

Acknowledgement

The authors sincerely acknowledge the doctors who offering great assistance in the Second Hospital of Shanxi Medical University and in the Shanxi Provincial People’s Hospital. We also sincerely acknowledge the editor and any reviewers for their valuable advice. This work was supported by the National Natural Science Foundation of China (Grant NO. 11472184, No. 11771321, No. 61901292); the National Youth Science Foundation of China (Grant NO.11401423); the ShanXi province plan project on Science and Technology of social Development (Grant No. 201703D321032); and the Natural Science Foundation of Shanxi Province, China (Grant No. 201901D211080).

References

Bueno ,

M.M.

Fernandez-Carrobles ,

Gonzalez-Lopez and

Deniz , Glomerulosclerosis identification in whole slide images using semantic segmentation, Computer methods and programs in biomedicine 184 (2020), 105273. doi:10.1016/j.cmpb.2019.105273.

J.D.

Bukowy ,

Dayton ,

Cloutier ,

A.D.

Manis ,

Staruschenko ,

J.H.

Lombard ,

L.C.S.

Woods ,

D.A.

Beard and

A.W.

Cowley , Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections, Journal of the American Society of Nephrology 29(8) (2018), 2081–2088. doi:10.1681/ASN.2017111210.

Cabitza ,

Rasoini and

G.F.

Gensini , Unintended consequences of machine learning in medicine, Jama 318(6) (2017), 517–518. doi:10.1001/jama.2017.7797.

Coudray ,

P.S.

Ocampo ,

Sakellaropoulos ,

Narula ,

Snuderl ,

Fenyö ,

A.L.

Moreira ,

Razavian and

Tsirigos , Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning, Nature medicine 24(10) (2018), 1559–1567. doi:10.1038/s41591-018-0177-5.

Dalal and

Triggs , Histograms of oriented gradients for human detection, in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1, IEEE, 2005, pp. 886–893.

Farahani ,

A.V.

Parwani and

Pantanowitz , Whole slide imaging in pathology: Advantages, limitations, and emerging perspectives, Pathology and Laboratory Medicine International 7 (2015), 23–33. doi:10.2147/PLMI.S59826.

A.B.

Fogo , Renal Pathology, Pediatric Nephrology, 2009.

George ,

Skaria ,

Varun et al., Using yolo based deep learning network for real time detection and localization of lung nodules from low dose ct scans, in: Medical Imaging 2018: Computer-Aided Diagnosis, Vol. 10575, International Society for Optics and Photonics, 2018, p. 105751I.

Goode ,

Gilbert ,

Harkes ,

Jukic and

Satyanarayanan , Openslide: A vendor-neutral software foundation for digital pathology, Journal of pathology informatics 4 (2013), 27. doi:10.4103/2153-3539.119005.

10.

Haas , Donor kidney biopsies: Pathology matters, and so does the pathologist, Kidney international 85(5) (2014), 1016–1019. doi:10.1038/ki.2013.439.

11.

He ,

Zhang ,

Ren and

Sun , Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE transactions on pattern analysis and machine intelligence 37(9) (2015), 1904–1916. doi:10.1109/TPAMI.2015.2389824.

12.

Janowczyk and

Madabhushi , Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases, Journal of Pathology Informatics 7 (2016), 29. doi:10.4103/2153-3539.186902.

13.

Kakimoto ,

Okada ,

Hirohashi ,

Relator ,

Kawai ,

Iguchi ,

Fujitaka ,

Nishio ,

Kato and

A.A.

Fukunari , Automated image analysis of a glomerular injury marker desmin in spontaneously diabetic torii rats treated with losartan, Journal of Endocrinology 222(1) (2014), 43–51. doi:10.1530/JOE-14-0164.

14.

Kato ,

Relator ,

Ngouv ,

Hirohashi ,

Takaki ,

Kakimoto and

Okada , Segmental hog: New descriptor for glomerulus detection in kidney microscopy image, Bmc Bioinformatics 16(1) (2015), 1–16. doi:10.1186/s12859-014-0430-y.

15.

Kawazoe ,

Shimamoto ,

Yamaguchi ,

Shintani-Domoto ,

Uozaki ,

Fukayama and

Ohe , Faster r-cnn-based glomerular detection in multistained human whole slide images, Journal of Imaging 4(7) (2018), 91. doi:10.3390/jimaging4070091.

16.

Liu ,

Wang ,

He and

Chen , Towards staining independent segmentation of glomerulus from histopathological images of kidney, bioRxiv (2019), 821181.

17.

D.M.

Metter ,

T.J.

Colgan ,

S.T.

Leung ,

C.F.

Timmons and

J.Y.

Park , Trends in the us and Canadian pathologist workforces from 2007 to 2017, JAMA network open 2 5 (2019), e194337–e194337. doi:10.1001/jamanetworkopen.2019.4337.

18.

Redmon ,

Divvala ,

Girshick and

Farhadi , You only look once: Unified, real-time object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 779–788.

19.

Redmon and

Farhadi , Yolo9000: Better, faster, stronger, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7263–7271.

20.

Redmon and

Farhadi , Yolov3: An incremental improvement, 2018, arXiv preprint arXiv:1804.02767.

21.

Ren ,

He ,

Girshick and

Sun , Faster r-cnn: Towards real-time object detection with region proposal networks, IEEE transactions on pattern analysis and machine intelligence 39(6) (2016), 1137–1149. doi:10.1109/TPAMI.2016.2577031.

22.

Rezatofighi ,

Tsoi ,

Gwak ,

Sadeghian ,

Reid and

Savarese , Generalized intersection over union: A metric and a loss for bounding box regression, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 658–666.

23.

B.C.

Russell ,

Torralba ,

K.P.

Murphy and

W.T.

Freeman , Labelme: A database and web-based tool for image annotation, International journal of computer vision 77(1–3) (2008), 157–173. doi:10.1007/s11263-007-0090-8.

24.

Shen ,

Liu ,

Li ,

Y.-G.

Jiang ,

Chen and

Xue , Object detection from scratch with deep supervision, IEEE transactions on pattern analysis and machine intelligence 42(2) (2019), 398–412. doi:10.1109/TPAMI.2019.2922181.

25.

Simon ,

Yacoub ,

Jain ,

J.E.

Tomaszewski and

Sarder , Multi-radial lbp features as a tool for rapid glomerular detection and assessment in whole slide histopathology images, Scientific reports 8(1) (2018), 1–11.

26.

J.A.

Suykens and

Vandewalle , Least squares support vector machine classifiers, Neural processing letters 9(3) (1999), 293–300. doi:10.1023/A:1018628609742.

27.

Yu ,

Jiang ,

Wang ,

Cao and

Huang , Unitbox: An advanced object detection network, in: Proceedings of the 24th ACM International Conference on Multimedia, 2016, pp. 516–520. doi:10.1145/2964284.2967274.

28.

Zhang ,

Gao ,

Zhao and

Liu , Local derivative pattern versus local binary pattern: Face recognition with high-order local pattern descriptor, IEEE transactions on image processing 19(2) (2009), 533–544. doi:10.1109/TIP.2009.2035882.

29.

Zheng ,

Wang ,

Liu ,

Li ,

Ye and

Ren , Distance-iou loss: Faster and better learning for bounding box regression, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 12993–13000.

30.

H.U.

Zollinger and

M.J.

Mihatsch , Renal Pathology in Biopsy: Light, Electron and Immunofluorescent Microscopy and Clinical Aspects, Springer Science & Business Media, 2012.

An efficient glomerular object locator for renal whole slide images using proposal-free network and dynamic scale evaluation method

Abstract

Keywords

1. Introduction

2.1. Traditional methods

2.2. Deep learning based methods

3. Material and method

3.1. Dataset description

3.1.1. Training dataset

4.1. Experimental settings

4.2. Evaluation metrics

4.3. Ablation study

Table 3 Performance on five testing datasets from different hospitals and years Datasets Recall Precision F 1 Test-1 0.986 0.997 0.991 Test-2 0.974 0.990 0.982 Test-3 0.975 0.988 0.981 Test-4 0.976 0.998 0.989 Test-5 0.938 0.985 0.961 Average 0.970 0.992 0.981

Table 6 Performance comparison with pathologists Time per glomerulus Recall Precision F 1 Primary pathologist 22.32 s 0.822 1.000 0.902 Senior pathologist 11.50 s 0.933 1.000 0.965 GOL 0.37 s 0.970 0.992 0.981

Footnotes

Acknowledgement

References

Table 3
Performance on five testing datasets from different hospitals and years

Datasets Recall Precision $F_{1}$

Test-1 0.986 0.997 0.991

Test-2 0.974 0.990 0.982

Test-3 0.975 0.988 0.981

Test-4 0.976 0.998 0.989

Test-5 0.938 0.985 0.961

Average 0.970 0.992 0.981

Table 6
Performance comparison with pathologists

Time per glomerulus Recall Precision $F_{1}$

Primary pathologist 22.32 s 0.822 1.000 0.902

Senior pathologist 11.50 s 0.933 1.000 0.965

GOL 0.37 s 0.970 0.992 0.981