An Automatic Biopsy Needle Detection and Segmentation on Ultrasound Images Using a Convolutional Neural Network

Abstract

Needle visualization in the ultrasound image is essential to successfully perform the ultrasound-guided core needle biopsy. Automatic needle detection can significantly reduce the procedure time, false-negative rate, and highly improve the diagnosis. In this paper, we present a CNN-based, fully automatic method for detection of core needle in 2D ultrasound images. Adaptive moment estimation optimizer is proposed as CNN architecture. Radon transform is applied to locate the needle. The network’s model was trained and tested on the total of 619 2D images from 91 cases of breast cancer. The model has achieved an average weighted intersection over union (the weighted Jaccard Index) of 0.986, F1 Score of 0.768, and angle RMSE of 3.73°. The obtained results exceed the other solutions by at least 0.27 and 7° in case of F1 score and angle RMSE, respectively. Finally, the needle is detected in a single frame averagely in 21.6 ms on a modern PC.

Keywords

core-needle biopsy convolutional neural networks needle detection needle segmentation

Introduction

The core-needle biopsy is a necessary procedure helping to diagnose or to rule out medical condition. During this procedure, a hollow needle is inserted and a sample of tissue collected. Correct diagnosis highly depends on the accuracy of the needle puncture. Therefore, biopsies are performed with the assistance of medical imaging and are regarded as a highly accurate method. Breast cancer is one of the most common diseases diagnosed by biopsy.

Breast cancer is a disease in which a breast tissue cells proliferate excessively, forming larger and larger tumor. It usually originates in the glandular or ductal cells and affects mostly women. It is stated that in 2019 in the United States, there were 316,700 women diagnosed with breast cancer and 2670 men. It caused death of 41,760 women and 500 men in 2019¹ in the USA. In recent years, there has been an increasing trend in both morbidity and mortality caused by breast cancer.^2,3

To obtain an accurate and reliable diagnosis of breast tumor and start the right treatment it is necessary to perform various medical imaging techniques. Ultrasound-guided core-needle biopsy is especially important. During this procedure, the breast is punctured with a biopsy gun equipped with an 18 to 14-gauge cutting needle (a core-needle) and ca. 3 to 5 tissue samples are taken. The correct diagnosis highly depends on the accuracy of the needle puncture. The samples are assessed histopathologically and molecularly, which leads to a final diagnosis.^4,5 The diagnostic accuracy reaches 95.5%,⁶ if the procedure is performed correctly.

The ultrasound-assisted core-needle biopsy is also used to diagnose other organs’ focal lesions, for example, lungs, kidneys, liver, prostate, or lymphatic nodules. Furthermore, ultrasonography is employed during therapeutic punctures, such as the pneumothorax decompression⁷ or the abdominal paracentesis.⁸ Snapshots captured at the procedure may be included in patient medical record.

The clear visualization of the needle in the ultrasound image is only possible if it is correctly positioned in relation to the ultrasound beam. If the needle and the beam are perpendicular, there is a strong specular reflection that enhances the needle’s image quality. The image is often accompanied by a reverberation artifact (so-called “comet-tail artifact”).⁹ Additionally, adjacent clear longitudinal anatomical structures (e.g., borders of bones or fasciae) may disturb the process of needle segmentation and prevent its correct location.⁹ If the needle does not lie parallel to the beam’s plane, it is visible only partially at the intersection with the plane. Additionally, the process of needle segmentation and its correct localization may be disturbed by adjacent clear longitudinal anatomical structures (e.g., borders of bones or fasciae).⁹

There are two main approaches for needle detection: hardware and software-based one. The hardware-based approach is determined by a tracking device and sensors attached to the needle and ultrasound probe. The sensors are localized by an optical or electromagnetic technique.¹⁰ A full system calibration needs to be performed beforehand.¹¹ Hardware-based methods are sensitive to errors caused by the needle’s flexibility unless the sensor is attached directly to its tip.¹²

On the other hand, software-based approaches cover a wide range of image processing algorithms, from classic to novel deep learning methods. These methods detect a needle in two- or three-dimensional ultrasound images. They are summarized in Tables 1 and 2 for 2D and 3D images, respectively. To enhance bright linear structures, these methods employ mainly the following pre-processing: Otsu tresholding,¹³ Top-hat filter,¹⁴ Hough transform,¹⁵ or Log-Gabor wavelets.¹⁶ The machine-learning methods are widely used for needle segmentation process.^14,17–21 The needle covers less than 1% of the image’s surface, so its segmentation becomes a very challenging task. Therefore the next step of image processing involves the computation of its trajectory. A variety of methods have been proposed from a simple selection of the maximal segmented object¹⁷ to more advanced techniques as Kalman filter,^13,22 log-Gabor filter,¹⁸ Gabor filter,¹⁵ or Radon transform.¹⁶

Table 1.

Summary of Needle Detection Approaches in 2D Ultrasound Images.

Ref.	Preprocessing	Candidate region	Final segmentation
Lee et al.¹⁷	Cropping	Light deep learning	Max contour algorithm
Agarwal et al.¹³	Median filtering; Otsu thresholding	Canny edge detection	Kalman filtering
Groves et al.¹⁴	Resizing; depth normalization	CNN	—
Mwikirize et al.¹⁸	Top-hat filter	Fully convolutional network	R-CNN; 2D Log-Gabor filter bank
Wijata et al.¹⁵	Hough transform	Shock filter	Gabor filter
Czajkowska et al.²³	Gaussian smoothing	Histogram of Gradients	Kernelized Weighted C-Means (KWCM)
Hatt et al.¹⁶	Log-Gabor wavelets	Adaboost	Radon transform; second Gaussian derivative filter

Table 2.

Summary of Needle Detection Approaches in 3D Ultrasound Images.

Ref.	Preprocessing	Candidate region	Final segmentation
Yang et al.¹⁹	Slice-based UNet	Direction-fused UNet; 3D Pyramid-UNet	—
Arif et al.²⁰	—	CNN	Principal component analysis
Pourtaherian et al.²¹	Patch classification; FCN	ShareCNN, IndepCNN	RANSAC algorithm
Zhao et al.²²	Hessian matrix	Principal component analysis; Hough transform	Kalman filter
Zhao et al.²⁴	Hessian based line filtering method; Frangi’s line filter	RANSAC algorithm	Principal component analysis; fast normalized cross-correlation; Kalman filter

Other needle segmentation approaches and the datasets, accompanied by the obtained segmentation results, are described in Tables 3 and 4 for 2D and 3D images, respectively. The vast number of automatic needle detection methods is based on animal phantoms, mainly porcine and chicken.^{15,16,18,19,21,23} A few thousands of phantom images were used for training these methods. The phantom-based databases are very common. Nevertheless, it is vital to remember that human tissues are more heterogeneous with a number of artifacts, therefore it is important to provide methods based on clinical data, even if their number is smaller. Unfortunately researches employing human in vivo images are infrequent and do not process more than a thousand 2D^16,17 or 3D^19,20 images.

Table 3.

Summary of Needle Detection Approaches in 2D Ultrasound Images—Reported Results.

Ref.	Image data	No. of images	Measure	Value
Lee et al.¹⁷	8 patients	996	IoU	41.01%
			Dice accuracy	56.65%
			$Δ θ_{r m s}$	13.3°
			$Δ λ_{r m s}$	9.5 pixels
Agarwal et al.¹³	1 tissue phantom	ca. 160	RMSE	0.598 mm
Groves et al.¹⁴	3 phantoms	5958	RMSE	0.62 and 0.74 mm (6.08 and 7.62 pixels)
Groves et al.¹⁴	3 phantoms	5958	MAE	0.50 ± 0.40 mm and 0.51 ± 0.54 mm (4.90 ± 3.96 and 5.24 ± 5.52 pixels) for the axial and lateral directions
Mwikirize et al.¹⁸	2 phantoms (porcine and bovine tissue)	2900	$Δ λ$	0.38 ± 0.1 mm
Mwikirize et al.¹⁸	2 phantoms (porcine and bovine tissue)	2900	$Δ θ$	0.82° ± 0.4°
Wijata et al.¹⁵	8 phantoms (porcine tissue)	1440	Euclidean distance	17.60 ± 18.44 pixels
Czajkowska et al.²³	2 phantoms (pork and chicken tissue)	442	Sensitivity	84.6%
Czajkowska et al.²³	2 phantoms (pork and chicken tissue)	442	Specificity	99.0%
Hatt et al.¹⁶	1 phantom (pork and chicken tissue)	45	TE ( $Δ λ$ ) NLSR	0.48 mm
	1 phantom (pork and chicken tissue)	45	TE ( $Δ λ$ ) NLSR	86.2%
	6 clinical dataset	577	TE ( $Δ λ$ ) NLSR	0.19 mm
	6 clinical dataset	577	TE ( $Δ λ$ ) NLSR	99.80%

Table 4.

Summary of Needle Detection Approaches in 3D Ultrasound Images—Reported Results.

Ref.	Image data	No. of images	Measure	Value
Yang et al.¹⁹	8 phantoms (porcine heart)	92	Dice score	70.5% ± 9.2%
			Hausdorff distance	7.3 ± 4.1 voxels
			Prediction time	4.1 s
	18 patients	18 volumes	Dice score	66.0% ± 8.3%
			Hausdorff distance	8.2 ± 2.9 voxels
			Prediction time	1.4 s
Arif et al.²⁰	17 patients	149	Detection time	190 ms
			Position distance	1.59 mm
			Orientation angle	1.74°
Pourtaherian et al.²¹	1 phantom (chicken and porcine tissue)	ca. 1000	Recall	89.6 ± 4.2
			Precision	79.8 ± 5.5
			Specificity	99.97 ± 0.0001
			Recall	87.9 ± 4.2
			Precision	83.0 ± 3.7
			Specificity	99.99 ± 0.00001
Zhao et al.²²	20 phantoms	411,800	Axis accuracy	0.4 ± 0.1 mm
			Angle $α$	0.0° ± 0.0°
			Angle $β$	0.1° ± 1.2°
			Tip error	0.6 ± 0.3 mm
			Calculation time	120.7 ± 12.4 s
Zhao et al.²⁴	55 phantom sets (lamb heart)	788,480	Axis accuracy	1.8 mm
			RMSE tip localization	1.6 mm
			RMSE of angle $α$	8°
			RMSE of angle $β$	5°
			Localization time	0.1 s

In this paper, a method using convolutional neural networks to detect a core needle in the 2D ultrasound images is presented. The method enables determination of the needle trajectory in real-time, and also when only partially visible in the imaging plane. The method was evaluated on a database containing 619 images from 91 cases of a breast cancer. No artificial or phantom images were used in the workflow.

Materials

The images used in this study were obtained from two databases. The first one, containing 81 ultrasound images recorded for 58 patients, is publicly available.²⁵ Images in the database feature various dimensions and acquisition parameters. There is no information about the devices, the images were taken with. The database was used in other research studies, for example by Kriti et al.²⁶ to evaluate segmentation methods.

The second database consists of 538 two-dimensional ultrasound images from 204 core-needle biopsies performed by radiologists on 33 patients.²⁷ The images were recorded using an S-Video to USB converter (DFG/USB2pro, Imaging Source, Bremen, Germany) connected to an ultrasound device (EZU-MT25-S1, Hitachi, Tokyo, Japan) with a linear transducer EUP-L65. The ultrasound beam’s depth ranged from 25 to 40 mm depending on the lesion’s location, and the frequency of the ultrasound device ranged from 6 to 14 MHz. The images were captured with a resolution of 768 × 576 at 25 fps.

We merged both databases, and an expert delineated the needle in each image on a PC workstation using a freehand drawing tool on a graphical editor. Convex hull was computed for each delineation, and the internal pixels were filled. These segmented sets of pixels are the ground truth data.

Methods

The developed method for detecting the needle in ultrasound images is fully automatic, consisting of initial pre-processing, image segmentation using a convolutional neural network (CNN), and finally, the needle trajectory detection. These steps are presented in Figure 1.

Figure 1.

A workflow of presented method for a needle detection in ultrasound images.

Image Pre-processing

The images from the second database described in Section Materials had contained superfluous data at their peripherals, so they were cropped at the appropriate borders. Motion artifacts of the video signal were deinterlaced using nearest-neighbor interpolation.

All the images were converted from RGB colorspace to 8-bit grayscale and resized to fit a CNN requirements (256 × 256 pixels) using nearest-neighbor interpolation. A pre-processed image is denoted in the paper as $I (x, y)$ .

Convolutional Neural Network in Needle Region Segmentation

The architecture and the parameters of the developed convolutional network were chosen experimentally. We tested various types of semantic segmentation architecture (LinkNet, U-Net, SegNet). Eventually, the parameters featuring enough effectiveness were found. The final network architecture and the parameters are summarized in Figure 2. All the network source files along with a MATLAB script have been published on Mendeley site.²⁸

Figure 2.

A convolutional neural network architecture developed for segmentation of the needle. $I (x, y)$ —input pre-processed image, $S (x, y)$ —output binary image. Abbreviations used to describe processing blocks: ks = kernel size, f = number of filters, st = step. Next to the arrows, the data dimensions entering the next processing block are given (width × height × no. of channels).

The input layer accepts single-channel grayscale images $I (x, y)$ with a size of 256 × 256 pixels. The output is a binary image with the segmented needle $S (x, y)$ of the same dimension.

The proposed network is divided into two parts: convolution and deconvolution. Convolutional blocks consist of a convolution layer, a normalization layer, and a rectified linear unit (ReLU layer). In the first and third block, the max-pooling layer (2 × 2 stride over a 3 × 3 pool) was added. Sizes of the kernels of the subsequent convolutional layers are 11 × 11, 9 × 9, and 7 × 7. The size was chosen in order to detect locally larger objects. The number of filters for successive convolution layers increases with the power of 2 starting at 16. The padding takes the same values.

The deconvolution blocks consist of a transposed convolution layer, a normalization layer, and a rectified linear unit (ReLU layer). Similarly, the kernel sizes are 7 × 7 and 9 × 9 with 64 and 32 filters, respectively. The second convolution block has a 2 × 2 stride to increase the output image.

The final classification is done using the convolution layer, softmax, and pixel classification layer using generalized Dice loss for semantic segmentation.

The network was trained using the adaptive moment estimation (ADAM) optimizer. Data augmentation was also performed with the following parameters: translation by a random number of pixels from an integer set $[- 8, \dots, 8]$ in each direction, a random reflection in the left-right direction, and an isotropic scaling to the $I (x, y)$ using uniformly distributed values from a range $[0.8.1.2]$ . The batch size was set to 64 with 20 epochs. The initial learning rate was set to 0.005 with the drop factor of 0.5 and the period 3. The validation frequency was 2 per epoch.

The result of the developed CNN is a binary mask $S (x, y)$ which contains segmented objects: the needle and additional inaccurately classified linear-shaped structures.

Radon Transform in Needle Trajectory Detection

The needle’s trajectory is finally determined in the image by applying the Radon transform²⁹ on the $S (x, y)$ . The obtained image, called a sinogram, comprises a set of 1-D projections at various angles, each row of the sinogram contains one projection. The brightest parts of the sinogram correspond to the segmented objects in the binary image $S (x, y)$ . The brighter the area, the greater the collinearity of the objects in $S (x, y)$ . We assume that the coordinates of the sinogram’s maximum value relate to a straight line passing through the needle—its trajectory. The trajectory is back-transformed to a final binary image $R (x, y)$ . This image contains a single one-pixel-wide straight line across a black background.

Evaluation Methods

The evaluation methods can be divided into two parts: The former evaluate the quality of segmentation by comparing the area of the ground truth ( $G T (x, y)$ ) with the segmentation result ( $S (x, y)$ ) and the latter indicates the correctness of the needle trajectory localization by comparing the straight lines determined by the CNN ( $R (x, y)$ ) and the expert’s delineation ( $R_{G T} (x, y)$ ).

We assessed the segmentation result using the following measures: global accuracy, accuracy, intersection over union (IoU), weighted IoU, and F1 score:

Global accuracy reports the number of correctly classified pixels, regardless of class, up to the total number of pixels. It is defined as follows:

A C C = \frac{T P + T N}{T P + T N + F P + F N}

(1)

where TP, TN, FP, and FN are the numbers of true positive, true negative, false positive, and false negative pixels, respectively.

Accuracy indicates the fraction of correctly identified pixels averaged over the classes.

IoU, also known as the Jaccard similarity coefficient, is the most commonly used metric. IoU metric measures the similarity between two sets. It is defined as the quotient of the cardinality of the sets’ intersection and union.

I o U = \frac{T P}{(T P + F P + F N)}

(2)

Weighted IoU of each class, weighted by the number of pixels in that class. This measure is appropriate for unevenly sized classes because it can reduce the impact of errors in small classes. This situation occurs in our data, as the needle area is very little compared to the background.

F1 Score indicates how well the predicted boundary of each class matches the actual boundary. F1 Score is the metric used when comparing human assessed data. The formula is as follows:

F_{1} = 2 \cdot \frac{P R E \cdot S E N}{P R E + S E N} = \frac{T P}{T P + \frac{1}{2} \cdot (F P + F N)}

(3)

where PRE is the precision that determines the repeatability of the obtained results:

P R E = \frac{T P}{T P + F P}

(4)

and SEN is the sensitivity that determines the ratio of correctly identified pixels to the sum of true positives and false negatives:

S E N = \frac{T P}{T P + F N} .

(5)

Furthermore, the needle localization success rate (NLSR) was computed. It shows the ratio of $N$ images for which the method correctly indicates the line passing through the needle.¹⁶

N L S R = \frac{1}{N} \sum_{i = 1}^{N} N L S R_{i}

(6)

A case $N L S R_{i}$ is treated as a correct one, if the intersection of the segmented needle image ( $S_{i} (x, y)$ ), the ground truth ( $G T_{i} (x, y)$ ) and the trajectory image ( $R_{i} (x, y)$ ) is nonempty.

N L S R (i) = {\begin{cases} 0 for (G T_{i} (x, y) \cap S_{i} (x, y) \cap R_{i} (x, y)) = \emptyset \\ 1 for (G T_{i} (x, y) \cap S_{i} (x, y) \cap R_{i} (x, y)) = \emptyset \end{cases}

(7)

The evaluation values indicating the correctness of the needle trajectory detection (distance and angle) are $Δ λ$ , $Δ θ$ , $Δ λ_{r m s}$ , and $Δ θ_{r m s}$ . They are defined as follows:

Δ λ = | λ_{R_{G T}} - λ_{R} |

(8)

Δ θ = | θ_{R_{G T}} - θ_{R} |

(9)

where $λ_{R_{G T}}$ is the smallest distance between the line corresponding to the needle trajectory and the central pixel of the image; $λ_{R}$ is the smallest distance between the line determined by the classification and the central pixel of the image; $θ_{R_{G T}}$ is the angle between the line corresponding to the needle trajectory and the horizontal axis of the image, while $θ_{R}$ is the angle between the line corresponding to the segmented structure and the horizontal axis of the image. These parameters are presented in Figure 3.

Figure 3.

An explanation of the evaluation parameters. Red line—the needle trajectory computed by the proposed method. White line—the expert’s delineation (the ground truth). $P_{C}$ is the central point of the image, $θ_{R}$ and $θ_{R_{G T}}$ are the slopes of the computed trajectory and the expert-based one. $λ_{R}$ and $λ_{R_{G T}}$ are the distances from the two trajectories to $P_{C}$ .

Finally, the root mean square distance error ( $Δ θ_{r m s}$ ) and the root mean square angle error ( $Δ λ_{r m s}$ ) are defined accordingly with the above indications:

Δ λ_{r m s} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} {(λ_{R_{G T_{i}}} - λ_{R_{i}})}^{2}}

(10)

Δ θ_{r m s} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} {(θ_{R_{G T_{i}}} - θ_{R_{i}})}^{2}} .

(11)

Experiments

The combined database was randomly divided into three sets based on patient identifiers: a training set, a validation set, and a test set, in the relation 0.8:0.1:0.1. Along with the training and validation sets, we conducted the proposed model training, and using the test set, we checked the model effectiveness. We repeated the data sampling, training, and testing procedures 15 times.

Under the same assumptions, we conducted the training of a network proposed by Lee et al.¹⁷ and a statistical classifier described by Hatt et al.¹⁶ These methods were chosen for comparison because they also employed databases with real human tissue images: Lee et al. gathered 996 clinical images from eight patients, while Hatt et al. worked with a database consisting of 577 clinical images (and a few dozen phantom images). We compared our segmentation and classification results to these methods.

The proposed and the Lee methods were implemented in MATLAB 2020b environment. The Hatt method was implemented in Python 3.8.5. The experiments were run on Windows 10 with a hardware platform containing AMD Ryzen 9 3900 (CPU), 32 GB RAM 3 GHz, NVIDIA RTX Titan 24 GB (GPU).

Running a single experiment on a trained network took averagely 22 ms for the proposed method, 35 ms for the Lee method, and 7.64 seconds for the Hatt method.

Results of the Needle Segmentation

The achieved segmentation results were divided into three categories: the best (correct), avarage, and incorrect. They are all are presented in Figures 4 to 6, respectively. Even for average values of IoU, the presented trajectory lies very close to the ground truth.

Figure 4.

The best result of needle segmentation using the proposed method in relation to IoU. (a) Original ultrasound image. (b) The expert’s delineation (blue) with computed trajectory of the needle (green). (c) Segmentation result of the proposed method (blue) with computed trajectory of the needle (red). (d) The computed trajectories from (b) and (c) presented together.

Figure 5.

The result of segmentation by the proposed method corresponding to the average values of $Δ λ$ and $Δ θ$ . (a) Original ultrasound image. (b) The expert’s delineation (blue) with computed trajectory of the needle (green). (c) Segmentation result of the proposed method (blue) with computed trajectory of the needle (red). (d) The computed trajectories from (b) and (c) presented together.

Figure 6.

An incorrect result of needle segmentation using the proposed method. (a) Original ultrasound image. (b) The expert’s delineation (blue) with computed trajectory of the needle (green). (c) Segmentation result of the proposed method (blue) with computed trajectory of the needle (red). (d) The computed trajectories from (b) and (c) presented together.

The segmentation process may be disturbed by long bright elements presented in the breast tissue such as borders of the lobes. These structures may be incorrectly recognized as a needle, for example as in Figure 6.

An assessment of the segmentation results was conducted for the methods based on convolutional neural networks, that is the proposed method (PM) and the one by Lee et al.¹⁷ We also compared the results with the method described by Hatt et al.¹⁶ The following metrics were computed: global accuracy, accuracy, IoU, weighted IoU, and F1 score. The results for the experiments are summarized in Table 5.

Table 5.

Evaluation of the Needle Segmentation for the Proposed Method (PM), the Lee Method, and the Hatt Method.

Model	Statistics	Global accuracy	Accuracy	IoU	Weighted IoU	F1 score
PM	Average	$0.992$	$0.742$	$0.655$	$0.986$	$0.768$
	Median	$0.993$	$0.756$	$0.658$	$0.988$	$0.801$
	Std	$0.004$	$0.130$	$0.087$	$0.006$	$0.146$
	Min	$0.973$	$0.495$	$0.490$	$0.952$	$0.391$
	Max	$0.998$	$0.991$	$0.882$	$0.996$	$1.000$
Lee	Average	$0.573$	$0.718$	$0.296$	$0.566$	$0.102$
	Median	$0.563$	$0.695$	$0.287$	$0.556$	$0.087$
	Std	$0.143$	$0.100$	$0.078$	$0.142$	$0.046$
	Min	$0.196$	$0.452$	$0.100$	$0.189$	$0.041$
	Max	$0.917$	$0.958$	$0.519$	$0.907$	$0.310$
Hatt	Average	$0.990$	$0.504$	$0.499$	$0.982$	$0.498$
	Median	$0.991$	$0.500$	$0.496$	$0.985$	$0.440$
	Std	$0.004$	$0.012$	$0.011$	$0.008$	$0.140$
	Min	$0.978$	$0.497$	$0.490$	$0.959$	$0.375$
	Max	$0.997$	$0.577$	$0.568$	$0.995$	$0.918$

The global accuracy of the proposed method and the Hatt method are similar (0.992 ± 0.004 vs. 0.990 ± 0.004). The global accuracy for these methods are significantly higher than for the Lee method (0.573 ± 0.143). As the needle area is very little compared to the background, this result reflects primarily the background classification. On average, the region of the segmented needle represents 0.76% ± 0.38% of the ultrasound image. The relatively low value of the Lee method results from spilling the segmentation mask of the needle.

The accuracy does not differ significantly between the proposed method and the Lee one (0.742 ± 0.130 vs. 0.718 ± 0.100). In case of the Hatt method, the accuracy is lower than in other cases (0.504 ± 0.012). This parameter is a measure of all correctly classified pixels in the classes.

Much higher values of IoU for the proposed method than for the Hatt and the Lee method (0.655 ± 0.087 vs. 0.499 ± 0.011 and vs. 0.296 ± 0.078) prove lower amount of false positives and false negatives obtained by our method.

The weighted IoU for the proposed network is similar to the Hatt one (0.982 ± 0.008 vs. 0.986 ± 0.006) but we observed higher F1 score for our method (0.768 ± 0.146) than for the Hatt method (0.498 ± 0.140).

Significantly higher values of Weighted IoU and F1 Score for our and the Hatt methods than for the Lee one also indicate that the proposed method was trained properly and leads to correct results.

After proving the non-normality for all the analyzed parameters distributions using the Shapiro-Wilk test, we applied the Wilcoxon-signed rank test to compare the measures of the segmentation results between all three methods. The test showed that our method yielded significantly larger values for all the parameters described in Table 5 with $p < 0.05$ .

Results of the Needle Trajectory Detection

The average angle of insertion of the needle into the breast in the images of the database used is 70.61° ± 9.14°. Very similar values were recorded by Mwikirize et al.¹⁸ in their research.

The needle localization success rate (NLSR), $Δ λ_{r m s}$ , $Δ θ_{r m s}$ , and average image processing time ( $t$ ) were computed for the proposed method as well as for the Lee and Hatt methods. The obtained results are summarized in Table 6. NLSR has the highest value for the proposed method (89.51%). The values of the remaining metrics summarizing the needle trajectory location were determined for cases where NLSR was 1. The smallest errors $Δ λ_{r m s}$ , $Δ θ_{r m s}$ are also observed for the proposed method. The average processing time per image was significantly lower with the proposed method and the Lee method than that required by the Hatt method (22 vs. 35 vs. 76,400 ms, respectively).

Table 6.

Needle Trajectory Errors for the Proposed Method (PM), the Lee, and the Hatt Method. The Symbols are Explained in Section Evaluation Methods.

Method	NLSR (%)	$Δ λ_{r m s}$ (pixel)	$Δ θ_{r m s}$ (°)	$t$ (ms)
PM	$89.51$	$9.25$	$3.73$	$21.62$
Lee	$73.15$	$22.88$	$10.93$	$35.08$
Hatt	$29.94$	$30.78$	$13.51$	$76, 400$

The average trajectory detection errors calculated for our method are 2.24 pixels and 1.29° for $Δ λ$ and $Δ θ$ , respectively. These parameters take values 10.55 pixels and 6.08° for the Lee, and 20.37 pixels and 10.19° for the Hatt method (Table 7).

Table 7.

Evaluation of the Needle Trajectory Localization Errors for the Proposed Method (PM), the Lee Method, and the Hatt Method.

Statistics	$Δ λ_{P M}$	$Δ θ_{P M}$	$Δ λ_{L e e}$	$Δ θ_{L e e}$	$Δ λ_{H a t t}$	$Δ θ_{H a t t}$
Average	$2.24$	$1.29$	$10.55$	$6.08$	$20.37$	$10.19$
Median	$1.05$	$0.93$	$2.70$	$2.09$	$14.34 .$	$8.86$
Std	$6.69$	$2.34$	$16.12$	$8.57$	$22.04$	$8.26$
Min	$0.00$	$0.00$	$0.02$	$0.00$	$0.01$	$0.00$
Max	$91.56$	$20.05$	$80.05$	$59.80$	$98.99$	$36.98$

We performed a statistical analysis of the obtained results. Verification of normal data distribution was performed using the Shapiro-Wilk test that proved no normality. The Wilcoxon signed rank test was performed to verify differences in the determined values $Δ λ_{R}$ and $Δ θ_{R}$ between the methods. According to the test, our method errors are the smallest.

Discussion

An automatic detection of a biopsy needle is a challenging task so numerous attempts have been made to address this issue. Nevertheless, all of them have some shortcomings.

After implementing the Lee method¹⁷ and training it on our database, we received significantly lower segmentation values but higher final results of localization. The method was the same but databases were different. The results confirm the importance of providing database diversity. Similarly, the Groves method¹⁴ was tested on a very uniform database. The authors used water-filled phantoms. Although the method yields good results, it is hard to judge how this would work with real images. The limitation of the Mwikirize method¹⁸ is a restricted range of injection angles. This range was specified by the authors as 45° to 75°. In our database, we obtained an average injection angle equal to $73.60 \pm 7.92$ , so this method is not fully universal. The Hatt method¹⁶ also yielded good results, but the algorithm was not resistant to the needle injection direction. In their work, the authors assumed that the needle was always injected from the left side of the image.

A workflow for a needle trajectory detection based solely on its segmentation does not lead to success. The needle covers few pixels, so an under-segmentation or over-segmentation occurs very often. The under-segmentation is preferable as even the smallest segmented part of the needle is usually sufficient to compute needle trajectory using the Radon transform.

The presented method has some limitation. Although the segmentation process yields very accurate results for clean input images, it is sensitive to superimposed text, arrows, or lines as well as high-intensity borders from adjacent soft tissue or bone along the needle trajectory. They may interfere with needle visibility. Therefore, accurate needle localization remains an ongoing challenge at steep angles and deep insertions.¹⁸ Reverberations artifacts from the needle also significantly decrease the quality of the results.

Moreover, during the process of puncturing the tissue, at the very first time, the needle cannot be recognized. Only when at least a few millimeters of the needle is visible, the method begins to detect it. The single image processing time is around 22 ms, which is sufficient for real-time implementation.

As the needle is not completely rigid, it may bend when insertion force is applied. In this situation, the proposed method computes the trajectory with a higher error that depends on the grade of the needle curvature. The same statement applies to the other methods.

Despite the difficulties indicated above, the presented method gives valuable results that exceed other solutions. This method may be beneficial in the future when the needle trajectory is recorded. If a real-time automated needle detection is implemented into the clinical workflow, one should expect such benefits as a decrease in the number of punctures, higher accuracy, or shorter procedure time.

In the paper, clinical data were collected during the ultrasound-assisted core needle breast biopsy. The dataset comprised various types of tumors, and its quantity is comparable to other researches depending on clinical data. Although there are researches based on larger databases, their images are acquired from phantoms. These datasets feature a significantly higher resolution of images, fewer artifacts, more homogeneous background. Phantom datasets are easier to be acquired than clinical ones, but they do not capture the real-world artifacts contained in clinical data and thus may cause CNNs to overfit if not trained properly.

Conclusion

This paper presents an automatic method for needle detection and segmentation in the ultrasound image during a core-needle biopsy. The high accuracy of the method and the processing rate encourage us to work further on eliminating current limits described in the Discussion. The method can be used online, during a procedure of core-needle biopsy.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research has been founded by the Silesian University of Technology, Faculty of Biomedical Engineering grants Nos. 07/010/BKM20/0048 and 07/010/BKM20/0049.

ORCID iD

Bartłomiej Pyciński

References

Breast Cancer Facts & Figures 2019-2020. American Cancer Society. Atlanta: American Cancer Society, Inc.; 2019.

Azamjah

Soltan-Zadeh

Zayeri

Global trend of breast cancer mortality rate: a 25-year study. Asian Pac J Cancer Prev. 2019;20(7):2015-20. doi:10.31557/APJCP.2019.20.7.2015.

Desantis

Gaudet

Newman

Miller

Goding Sauer

, et al. Breast cancer statistics, 2019. CA Cancer J Clin. 2019;69:438-51. doi:10.3322/caac.21583.

Slubowski

Slubowska

Wojciechowski

Diagnostic techniques in breast cancer detection. part II: Physical and biopsy methods. Ginekologia Polska. 2007;78:479-83.

Rocha

Pinto

Tavares

Gonçalves

CFA

. Step-by-step of ultrasound-guided core-needle biopsy of the breast: review and technique. Radiologia Brasileira. 2013;46:234-41. doi:10.1590/S0100-39842013000400010.

Fattahi

Tavassoli

Kalantari

Noorshafiee

Rahmani

Evaluation of the value of core needle biopsy in the diagnosis of a breast mass. Arch Breast Cancer. 2016;3(2):56-61. doi:10.19187/abc.20163256-61.

Dancel

Schnobrich

Puri

Franco-Sadud

Cho

Grikis

, et al. Recommendations on the use of ultrasound guidance for adult thoracentesis: a position statement of the society of hospital medicine. J Hosp Med. 2018;13(2):126-35. doi:10.12788/jhm.2940.

Cho

Jensen

Reierson

Mathews

Bhagra

Franco-Sadud

, et al. Recommendations on the use of ultrasound guidance for adult abdominal paracentesis: a position statement of the society of hospital medicine. J Hosp Med. 2019;14:E7-15. doi:10.12788/jhm.3095.

Huang

Triedman

Vasilyev

Suematsu

Cleveland

Dupont

PE.

Imaging artifacts of medical instruments in ultrasound-guided interventions. J Ultrasound Med. 2007;26(10):1303-22. doi:10.7863/jum.2007.26.10.1303.

10.

Franz

Haidegger

Birkfellner

Cleary

Peters

Maier-Hein

Electromagnetic tracking in medicine–a review of technology, validation, and applications. IEEE Trans Med Imaging. 2014;33(8):1702-25. doi:10.1109/TMI.2014.2321777.

11.

Lasso

Heffter

Rankin

Pinter

Ungi

Fichtinger

PLUS: open-source toolkit for ultrasound-guided intervention systems. IEEE Trans Biomed Eng. 2014;61(10):2527-37. doi:10.1109/TBME.2014.2322864.

12.

Xia

West

Finlay

Mari

J-M

Ourselin

David

, et al. Looking beyond the imaging plane: 3D needle tracking with a linear array ultrasound probe. Sci Rep. 2017;7(1):3674. doi:10.1038/s41598-017-03886-4.

13.

Agarwal

Yadav

Gupta

Orlando

MF.

Real-time needle tip localization in 2D ultrasound images using Kalman filter. In: 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2019, pp. 1008-12.

14.

Groves

VanBerlo

Peters

Chen

ECS

, et al. Deep learning approach for automatic out-of-plane needle localisation for semi-automatic ultrasound probe calibration. Healthc Technol Lett. 2019;6(6):204-9. doi:10.1049/htl.2019.0075.

15.

Wijata

Ranosz

Galińska

Juszczyk

Czajkowska

Detection and tracking of the biopsy needle using ultrasound images. In: Gzik

Tkacz

Paszenda

Piętka

, eds. Innovations in Biomedical Engineering. Cham: Springer International Publishing; 2018, pp. 33-41.

16.

Hatt

Parthasarathy

Enhanced needle localization in ultrasound using beam steering and learning-based segmentation. Comput Med Imaging Graph. 2015;41:46-54. doi:10.1016/j.compmedimag.2014.06.016.

17.

Lee

Islam

Woh

Washeem

TSM

Ngoh

LYC

Wong

, et al. Ultrasound needle segmentation and trajectory prediction using excitation network. Int J Comput Assist Radiol Surg. 2020;15:437-43. doi:10.1007/s11548-019-02113-x.

18.

Mwikirize

Nosher

Hacihaliloglu

Convolution neural networks for real-time needle detection and localization in 2D ultrasound. Int J Comput Assist Radiol Surg. 2018;13(5):647-57. doi:10.1007/s11548-018-1721-y.

19.

Yang

Shan

Bouwman

Kolen

de With

PHN

. Efficient and robust instrument segmentation in 3D ultrasound using patch-of-interest-fusenet with hybrid loss. Med Image Anal. 2021;67:101842. doi:10.1016/j.media.2020.101842.

20.

Arif

Moelker

van Walsum

Automatic needle detection and real-time bi-planar needle visualization during 3D ultrasound scanning of the liver. Med Image Anal. 2019;53:104-10. doi:10.1016/j.media.2019.02.002.

21.

Pourtaherian

Ghazvinian Zanjani

Zinger

Mihajlovic

Korsten

HHM

, et al. Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks. Int J Comput Assist Radiol Surg. 2018;13(9):1321-33. doi:10.1007/s11548-018-1798-3.

22.

Zhao

Shen

Bernard

Cachard

Liebgott

Evaluation and comparison of current biopsy needle localization and tracking methods using 3D ultrasound. Ultrasonics. 2017;73:206-20. doi:10.1016/j.ultras.2016.09.006.

23.

Czajkowska

Pycinski

Pietka

HoG feature based detection of tissue deformations in ultrasound data. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015, pp. 6326-29.

24.

Zhao

Bernard

Cachard

Liebgott

Biopsy needle localization and tracking using ROI-RK method. Abstr Appl Anal. 2014;2014:1-7. doi:10.1155/2014/973147.

25.

SonoSkills

. Free ultrasound library offered to you by SonoSkills and Hitachi Medical Systems Europe. Available from: https://www.ultrasoundcases.info/cases/breast-and-axilla/ (accessed June 2020).

26.

Kirti, Virmani

Agarwal

Characterization of breast tumors using selected laws’ mask texture features. In: 2017 Fourth International Conference on Image Information Processing (ICIIP), 2017, pp. 1-6.

27.

Pyciński

Juszczyk

Wijata

Galinska

Czajkowska

Pietka

Image guided core needle biopsy of the breast. In: Pietka

Badura

Kawa

Wieclawek

, eds. Information Technology in Biomedicine, Advances in Intelligent Systems and Computing, volume 762. Cham: Springer International Publishing; 2019, pp. 160-71.

28.

Wijata

Andrzejewski

Pyciński

A convolutional neural network for an automatic biopsy needle detection and segmentation on ultrasound images. Mendeley Data, 2021. doi:10.17632/zk6scwv52p.

29.

Toft

PA.

The Radon transform – theory and implementation. Technical University of Denmark, 1996.