Automated lung tumor segmentation robust to various tumor sizes using a consistency learning-based multi-scale dual-attention network

Abstract

BACKGROUND:

It is often difficult to automatically segment lung tumors due to the large tumor size variation ranging from less than 1 cm to greater than 7 cm depending on the T-stage.

OBJECTIVE:

This study aims to accurately segment lung tumors of various sizes using a consistency learning-based multi-scale dual-attention network (CL-MSDA-Net).

METHODS:

To avoid under- and over-segmentation caused by different ratios of lung tumors and surrounding structures in the input patch according to the size of the lung tumor, a size-invariant patch is generated by normalizing the ratio to the average size of the lung tumors used for the training. Two input patches, a size-invariant patch and size-variant patch are trained on a consistency learning-based network consisting of dual branches that share weights to generate a similar output for each branch with consistency loss. The network of each branch has a multi-scale dual-attention module that learns image features of different scales and uses channel and spatial attention to enhance the scale-attention ability to segment lung tumors of different sizes.

RESULTS:

In experiments with hospital datasets, CL-MSDA-Net showed an F1-score of 80.49%, recall of 79.06%, and precision of 86.78%. This resulted in 3.91%, 3.38%, and 2.95% higher F1-scores than the results of U-Net, U-Net with a multi-scale module, and U-Net with a multi-scale dual-attention module, respectively. In experiments with the NSCLC-Radiomics datasets, CL-MSDA-Net showed an F1-score of 71.7%, recall of 68.24%, and precision of 79.33%. This resulted in 3.66%, 3.38%, and 3.13% higher F1-scores than the results of U-Net, U-Net with a multi-scale module, and U-Net with a multi-scale dual-attention module, respectively.

CONCLUSIONS:

CL-MSDA-Net improves the segmentation performance on average for tumors of all sizes with significant improvements especially for small sized tumors.

Keywords

Chest CT lung tumor segmentation deep learning size normalization consistency learning

1 Introduction

Lung cancer is the most common malignant tumor and leading cause of cancer deaths worldwide [1]. To evaluate treatment response and disease progression, it is essential to measure tumor response, such as changes in tumor size, on chest CT images. The RECIST (Response Evaluation Criteria in Solid Tumors) 1.1 criteria for tumor response evaluation is a one-dimensional measure of tumor diameter based on the assumption that tumors are spherical and that there is a proportional change in tumor volume and a parallel change in tumor diameter [2]. However, there are limitations in evaluating the response to treatment in lung cancer patients with large, irregularly shaped lesions, in which volumetric measurements are required [3, 4]. In addition, automated segmentation of lung tumors from chest CT images is essential for reproducible volumetric measurements.

Recently, convolutional neural networks (CNN) based on deep learning have been widely used for image segmentation and have shown remarkable performance. However, CNNs are limited by perceived local information, making it difficult to capture structural dissimilarities and spatial dependencies due to inter-patient variability. To solve this problem, several studies on lung tumor segmentation have proposed multi-scale strategies and attention mechanism. Jiang et al. [5] proposed a multi-scale framework that simultaneously combines features across multiple image resolutions and feature levels through residual connection. Yang et al. [6] proposed a multi-level and multi-scale deep supervision framework that adjusts the weights of the layers by generating multiple predictions from multiple layers and monitoring the local depth of the learned features. Chen et al. [7] proposed a multi-attention framework incorporating multiple attention mechanisms for modeling spatial and channel-dimensional semantic interdependencies from global and local perspectives. Banu et al. [8] proposed a multi-attention framework that improves the ability to discriminate between nodular and non-nodular feature representations. Zhang et al. [9] proposed a multi-scale and multi-attention framework to effectively integrate channel attention modules and the U-Nets to segment malignant lung areas from surrounding chest areas. However, despite the application of multi-scale and attention, it is difficult to accurately segment small-sized tumors due to the lack of semantic information inside lung tumors. Several studies have tried to differentiate lung tumors from surrounding structures by generating patches with different window settings. Byun et al. [10] proposed a dual-coupling net with a shape-focused prior based on lung and mediastinal window images to accurately extract shape information from lung tumors. Lee et al. [11] proposed a dual-window capsule network that learns the relative spatial relationship between lung tumors and surrounding structures based on ensemble learning. However, these studies using dual-window patches have limitations such as requiring manual labeling for each patch and still showing poor segmentation performance in small-sized tumors.

Despite several recent lung tumor segmentation techniques, as shown in Fig. 1, lung tumors vary in size making it difficult to consistently segment lung tumors on chest CT images. This is because of the characteristics of lung tumors, in which small lung tumors less than 1 cm and large lung tumors greater than 7 cm are distributed according to the T stage [12]. In this paper, we propose a consistency learning-based multi-scale dual-attention lung tumor segmentation method to accurately segment lung tumors of various sizes on chest CT images. The main contributions of this paper can be summarized as follows. First, we propose a size-invariant patch generated by normalizing the tumor in the training data to the ratio of the average tumor size to avoid under- and over-segmentation caused by different ratios of lung tumors and surrounding structures in the input patch. Second, we integrate a multi-scale module and dual-attention module to learn image features at different scales and enhances the scale-attention ability to segment lung tumors of different sizes. Third, we propose a consistency loss that trains a network branch using size-variant patches and a network branch using size-invariant patches to have similar predictions. Fourth, we evaluate the efficacy of our framework on the publicly available NSCLC-Radiomics dataset and the Veterans Health Service Medical Center-approved chest CT dataset through ablation studies.

Fig. 1

Examples of lung tumors of various types, as well as size, location, and shape on chest CT images: (a) small-sized lung tumor (b) medium-sized lung tumor (c) large-sized lung tumor (d) mediastinum-attached tumor (e) chest wall-attached tumor (f) vessel-attached tumor. (red area indicates lung tumor).

2 Materials and methods

2.1 Materials

In this study, we used two datasets, the hospital dataset from the Veterans Health Service Medical Center (VHSMC) and the NSCLC-Radiomics public dataset from The Cancer Imaging Archive (TCIA) [13]. The patient characteristics for each dataset used in this study are shown in Table 1, and the study using the hospital dataset was approved by the Institutional Review Board of the Veterans Health Service Medical Center (IRB File Number: BOHUN 2018-07-009-005).

Table 1
Patient characteristics for each dataset used in this study

Patient characteristics VHSMC NSCLC-Radiomics

(n = 259) (n = 154)

Age 69.3±8.1 69.7±10.0

Gender

Male 251 (97%) 107 (69%)

Female 8 (3%) 47 (31%)

Histology

Adenocarcinomas 148 (57%) 39 (25%)

Squamous cell carcinomas 111 (43%) 115 (75%)

Size Group (Tumor size criteria)

1 (<3 cm) 80 (31%) 30 (20%)

2 (≥3 cm and < 5 cm) 98 (38%) 39 (25%)

3 (≥5 cm) 81 (31%) 85 (55%)

Patient characteristics	VHSMC	NSCLC-Radiomics
Age	69.3±8.1	69.7±10.0
Gender
Male	251 (97%)	107 (69%)
Female	8 (3%)	47 (31%)
Histology
Adenocarcinomas	148 (57%)	39 (25%)
Squamous cell carcinomas	111 (43%)	115 (75%)
Size Group (Tumor size criteria)
1 (<3 cm)	80 (31%)	30 (20%)
2 (≥3 cm and < 5 cm)	98 (38%)	39 (25%)
3 (≥5 cm)	81 (31%)	85 (55%)

The VHSMC dataset obtained from 259 NSCLC patients who underwent curative surgical resection consisted of 148 lung adenocarcinomas (LUADs) and 111 lung squamous cell carcinomas (LUSCs). The tumor size was measured by measuring the longest diameter in the representative image slice with the largest tumor, and the distribution of the tumor sizes ranged from 0.76 to 14.33 cm with an average size of 4.6 cm and a standard deviation of 2.66 cm. The CT images were acquired from July 11, 2003 to June 29, 2015 from four different CT scanner manufacturers: BrightSpeed S, Discovery CT750 HD, LightSpeed Plus, LightSpeed Pro 16, LightSpeed Ultra, LightSpeed VCT, LightSpeed 16, ProSpeed Advantage (GE Healthcare), Brilliance 64, GEMINI TF 16, Ingenuity (Philips), SOMATOM Definition AS+, SOMATOM Emotion, SOMATOM Emotion 6, SOMATOM Sensation 16, SOMATOM Sensation 64, SOMATOM Definition AS+, SOMATOM Spirit, Biograph 6 (Siemens Healthineers) Aquilion, and Asteion (TOSHIBA). Each image slice had a matrix size of 512x512 pixels with an in-plane resolution ranging from 0.52 to 1.17 mm. The slice thickness ranged from 2.5 mm to 10 mm, and a dose protocol of 100 to 130 kVp at 60∼583 mAs was used. The CT images consisted of 245 contrast-enhanced images and 14 non-contrast-enhanced images. The lung tumor areas were semi-automatically segmented by one board-certified radiologist using in-house software [14], and the segmented areas were manually corrected by the radiologist as needed.

The NSCLC-Radiomics dataset obtained from 422 NSCLC patients consisted of 51 LUADs, 152 LUSCs, 114 large cell lung carcinomas, 63 not otherwise specified, and 42 not applicable, of which 39 LUADs and 115 LUSCs were selectively used. The distribution of the tumor sizes ranged from 1.36 to 14.63 cm with an average size of 5.58 cm and a standard deviation of 2.69 cm. The CT images were acquired from September 27, 2004 to January 1, 2014 from two different CT scanner manufacturers: Biograph 40, SOMATOM Sensation 10, SOMATOM Sensation 16, SOMATOM Sensation Open (Siemens Healthineers) and XiO (CMS Imaging, Inc). Each image slice had a matrix size of 512x512 pixels with an in-plane resolution of 0.97 mm. The slice thickness was 3 mm and a dose protocol of 120 to 140 kVp at 40∼491 mAs was used.

2.2 Data preprocessing

To reduce the differences in intensity and pixel spacing among the CT images, intensity rescaling and spacing normalization are performed on the entire set of the CT images. For intensity rescaling, the intensity values of the CT images with lung window settings (WW: 1500, WL: –600) are rescaled to gray-scale intensities ranging from 0 to 255 [15]. For spacing normalization, all CT images are resampled to the minimum pixel spacing value. For data augmentation, we performed random rotation (–20° to 20°), random translation (–20 pixel to 20 pixel) and random scaling (0.8 to 1.2 times), increasing the amount of training data by 5 times.

2.3 Input patch generation

Lung tumors have various size distributions from small lung tumors of less than 1 cm to large lung tumors greater than 7 cm in size. Generating an input patch with a size that encompasses all the various sizes suffers from class imbalance problems, with small tumors having a small tumor relative to their background, and large tumors having a small background relative to their tumor area. Therefore, to compensate for the degradation of the small tumor segmentation performance caused by such a class imbalance problem, we propose a size-invariant patch that is used simultaneously with a size-variant patch that contains various original size information.

To generate size-variant patches, input images are cropped to 256 × 256 pixels based on the center coordinates of the tumor slice with the largest area. To generate size-invariant patches, the size-normalization ratio calculates the average tumor size over the current tumor size. The average tumor size is determined by calculating the average of tumor sizes within the training set, and each tumor is normalized to the average tumor size. The size-invariant patches are cropped to the same size as the size-variant patches.

When the size-invariant patches are extracted, as shown in Fig. 2, small lung tumors expand into average-sized lung tumors and decrease the ratio of the surrounding structures, whereas large lung tumors shrink into average-sized lung tumors and expand the ratio of the surrounding structures. Therefore, size-invariant patches have the same scale retaining similar proportions of the tumor and background, while size-variant patches have variable scales by preserving the original size information.

Fig. 2

Examples of size-variant patches (first row) and their size-invariant patches (second row): (a) small-sized lung tumor with 0.76 cm (b) medium-sized lung tumor with 3.91 cm (c) large-sized lung tumor with 6.70 cm.

2.4 Consistency learning-based multi-scale dual-attention network

To simultaneously train pairs of two input patches, the size-variant patch and size-invariant patch, we propose a consistency learning-based multi-scale dual-attention network (CL-MSDA-Net). As shown in Fig. 3, the proposed network consists of dual-branch networks, each receiving different input patches. One network is for feature extraction and tumor segmentation using the size-variant patches, and the other network is for feature extraction and tumor segmentation using the size-invariant patches. A consistency learning framework enables training to obtain common feature representations without the need to compute optimized hyperparameter values, allowing different input patches to be considered simultaneously.

Fig. 3

An illustration of CL-MSDA-Net architecture for lung tumor segmentation.

As shown in Fig. 4, each branch network consists of a contracting path and an expansive path based on 2D U-Net [16] consists shared weights to learn the common features of each input patch. The contracting patch consists of two 3×3 convolution operations followed by a batch normalization and a rectified linear unit (ReLU) activation function, and a 2×2 max pooling with stride 2 for down-sampling. The expansive path concatenates the same level output feature maps of the layer in the contracting path via skip connections and then performs two 3×3 convolution operations followed by a batch normalization, ReLU activation function, and 2×2 up-sampling with bilinear interpolation. The number of feature channels is doubled after the convolution operation in the contrastive patch, and the feature size is halved after the down-sampling. In contrast, the number of feature channels is halved after the convolution operation in the expansive path, and the feature size is doubled after the up-sampling. Bilinear interpolation up-sampling in the decoder is used to restore the original image size for pixel level prediction but has the problem of not considering the correlation between each pixel label [17]. To obtain multi-scale features and improve the scale-attention ability considering the channel and spatial correlation, a multi-scale dual-attention (MSDA) module is proposed. Inspired by SA-Net [18], we use a multi-scale module consisting of convolutional blocks to have hierarchical residual connections to enlarge the receptive field range. The multi-scale module consists of three 3×3 convolution operations that generate multiscale hierarchical feature maps to generate larger receptive fields. In addition, the dual-attention module consisting of channel-attention and spatial-attention [19] was applied to each scale extracted using the multi-scale module to focus only on the tumor area. The spatial attention module aggregates tumor information from different locations to enhance the feature representation at locations, and the channel attention module aggregates tumor information from different channels to obtain a correlation between any two channel feature maps. The dual-attention module can segment lung tumors of various sizes by considering both channel attention and spatial attention to generate weighted maps that highlight the spatial and channel information of lung tumors. The detailed structure of the MSDA module is shown in Fig. 5.

Fig. 4

An Illustration of MSDA-Net architecture.

Fig. 5

An illustration of the proposed module structure: (a) MSDA module (b) Dual-attention module.

The CL-MSDA-Net is a consistency learning framework that contains two differently networks f_seg1 and f_seg2, both of which are trained with the two input patches generated by the input patch generation step. The CL-MSDA-Net is optimized using two supervised losses, L_variant and L_invariant, and two consistency losses, L_norm and L_restore. The total loss for CL-MSDA-Net is as follows: $L_{total} = L_{variant} + L_{invariant} + α \cdot L_{norm} + (1 - α) \cdot L_{restore}$ (1) where α-represents the weighting factor of the consistency loss and is set to 0.5.

The CL-MSDA-Net is trained with a supervised loss defined as a combination of Binary Cross-Entropy loss and Dice loss to penalize false-negative and false-positive predictions. The supervised loss function is formulated as follows: $L_{var iant} (X, Y) = \frac{1}{N} \sum_{i = 1}^{N} (L_{BCE} (f_{seg 1} (X_{i}), Y_{i}) + L_{dice} (f_{seg 1} (X_{i}), Y_{i}))$ (2) $L_{in var iant} (\hat{X}, \hat{Y}) = \frac{1}{N} \sum_{i = 1}^{N} (L_{BCE} (f_{seg 2} ({\hat{X}}_{i}), {\hat{Y}}_{i}) + L_{dice} (f_{seg 2} ({\hat{X}}_{i}), {\hat{Y}}_{i}))$ (3) where X_i, ${\hat{X}}_{i}$ , Y_i, ${\hat{Y}}_{i}$ , L_BCE and L_dice are the size-variant input patch, size-invariant input patch, ground truth of size-variant patch, ground truth of size-invariant patch, Binary Cross-Entropy loss, and Dice loss, respectively. N is the number of datasets.

The consistency loss simultaneously supervises each prediction from networks f_seg1 and f_seg2, respectively. The consistency loss computes the similarity between pairs of predictions to jointly learn common features. They are scaled by equal ratios to calculate the consistency of the predicted probability across input patches of different ratios. The prediction f_seg1 (X_i) is normalized by multiplying the size-normalization ratio, which is calculated as the average tumor size over the current tumor size, and the prediction $f_{seg 2} ({\hat{X}}_{i})$ is restored by multiplying the reciprocal of the size-normalization ratio. As a result, the predictions produce two pairs of the normalized size ratios and the original size ratios, and the consistency between the predictions of the same ratios is obtained. To measure the consistency between the two predictions, L_norm and L_restore are computed for the average difference between each prediction using the L2 loss. The consistency loss function is formulated as follows: $L_{norm} (X, \hat{X}) = \frac{1}{N} \sum_{i = 1}^{N} {(f_{seg 2} ({\hat{X}}_{i}) - f_{norm} (f_{seg 1} (X_{i})))}^{2}$ (4) $L_{restore} (X, \hat{X}) = \frac{1}{N} \sum_{i = 1}^{N} {(f_{seg 1} (X_{i}) - f_{restore} (f_{seg 2} ({\hat{X}}_{i})))}^{2}$ (5) where f_norm and f_restore are the size normalization and size restoration, respectively. N is the number of datasets.

3 Results

Our network was implemented with Python 3.7 and the PyTorch framework and trained with four NVIDIA GeForce RTX 3090 GPUs. We trained with the initial learning rate using the Adam optimizer, and the learning rate was reduced by a factor of 0.95 per epoch using the LambdaLR scheduler. The maximum number of epochs was set to 100, and training was stopped early if the performance did not improve after 25 epochs. The hyperparameters used in the experiments are summarized in Table 2.

Table 2
Hyperparameter settings for experiment: The same hyperparameter settings were set except for the loss function of the proposed method, and each dataset used a different initial learning rate for optimization

Hyperparameters Methods

U-Net (Baseline) U-Net+Multi-scale module U-Net+Multi-scale module+Dual-attention module U-Net+Multi-scale module+Dual-attention module+Consistency Learning (Ours)

Optimizer Adam Optimizer

Initial learning rate 1e-4 (VHSMC), 1e-3 (NSCLC-Radiomics)

Loss function Binary Cross Entropy loss+Dice loss Binary Cross Entropy loss+Dice loss+L2 loss

Epoch (patience) 100 (25)

Batch size 10

We evaluated the ablation studies for the effects of a multi-scale module, a multi-scale dual-attention module, and a consistency learning-based multi-scale dual-attention module using U-Net on two experimental datasets. The VHSCMC and NSCLC-Radiomics datasets were analyzed with a 5-fold cross validation and hold-out validation, respectively. The configurations of the train, validation and test set of each dataset are shown in Table 3. As an experimental plan, we analyzed the segmentation results according to the performance of the entire tumor group and tumor size groups, and the performance of the proposed method and other modules was evaluated by three evaluation criteria: F1-score, recall, and precision. The tumor size groups were established according to the clinical T-stage and categorized into three groups. Size group 1 represents tumors measuring≤3 cm (T1), size group 2 represents tumors measuring > 3 cm and≤5 cm (T2), and size group 3 represents tumors measuring > 5 cm (T3/T4).

Table 3

Patient characteristics for training, validation, and test of each dataset. The amount of data is provided for whole and for each group, and minimum and maximum tumor sizes are given in parentheses. The VHSCMC dataset and NSCLC-Radiomics were performed with 5-fold cross validation and hold-out validation, respectively. In the VHSCMC dataset, each fold has the same configuration

	Patient characteristics	VHSMC (n = 259)	NSCLC-Radiomics (n = 154)
Train	Size Group (Tumor size criteria)
	Whole	165 [0.75cm∼13.55cm]	94 [1.38cm∼10.60cm]
	Group 1 (<3 cm)	51	18
	Group 2 (≥3 cm and < 5 cm)	63	25
	Group 3 (≥5 cm)	51	51
Validation	Size Group (Tumor size criteria)
	Whole	42 [1.39cm∼14.32cm]	25 [2.09cm∼10.35cm]
	Group 1 (<3 cm)	13	5
	Group 2 (≥3 cm and < 5 cm)	16	4
	Group 3 (≥5 cm)	13	16
Test	Size Group (Tumor size criteria)
	Whole	52 [1.72cm∼12.34cm]	31 [1.36cm∼14.63cm]
	Group 1 (<3 cm)	16	6
	Group 2 (≥3 cm and < 5 cm)	19	8
	Group 3 (≥5 cm)	17	17

Tables 4 and 5 show the performance evaluation results for each method on the entire VHSMC dataset, as well as on tumor size groups within the dataset. The proposed method improved the segmentation performance compared to the other methods, and the proposed method showed the best results with an F1-score of 80.49%, and in particular, the precision improved the most with a score of 86.78% compared to the other methods. In the case of Group 1, the F1-score, recall, and precision of the proposed method were 81.53%, 80.49%, and 86.75%, respectively, which were significantly improved by 5.09%, 2.42%, and 5.92%, compared to U-Net using the multi-scale dual-attention module. When using the multi-scale dual-attention module, Group 1 showed lower performance. This was because the ratio to tumor area was very small and provided little semantic information, although it considered the relative spatial relationship with surrounding structures. However, the largest performance improvement was seen through size normalization and consistency learning, in which small tumors were expanded to average-sized tumors, and the proportion of the surrounding structures was decreased. Group 2 showed a smaller increase than Group 1 but still showed performance improvements of 4.43%, 2.7%, and 4.57%, respectively, compared to U-Net using multi-scale dual-attention module. Group 3 showed the best performance in terms of precision, but had a slightly decreased F1 score and recall by 0.95% and 2.84%, respectively, compared to the method without consistency learning. In Group 3 with the VHSMC dataset, the effect of size normalization seems difficult to ascertain because the variations in tumor size are variable and low in number.

Table 4

Results of the performance evaluation of lung tumor segmentation on the entire VMHSC dataset: Mean and standard values of the proposed and comparative methods in terms of F1-score, recall and precision are provided

Methods	Metrics	Tumor size group
		Group 1	Group 2	Group 3
U-Net (Baseline)	F1 score	75.82±17.06	77.18±19.86	76.62±19.52
	Recall	76.98±19.93	79.16±22.09	73.88±24.05
	Precision	80.11±18.81	79.50±18.87	85.02±12.69
U-Net+Multi-scale module	F1 score	76.57±15.65	76.67±18.97	78.20±16.80
	Recall	79.71±15.12	79.23±20.11	75.89±21.47
	Precision	78.44±20.99	79.74±19.22	85.23±12.97
U-Net+Multi-scale module+	F1 score	76.44±17.65	77.32±18.48	78.88 ± 16.07
Dual-attention module	Recall	78.07±17.29	79.80±18.99	76.33 ± 20.43
	Precision	80.83±21.55	80.14±19.08	85.86±11.87
U-Net+Multi-scale module+	F1 score	81.53 ± 12.56	81.75 ± 15.94	77.93±19.11
Dual-attention module+	Recall	80.49 ± 15.72	82.50 ± 18.78	73.49±23.38
Consistency Learning (Ours)	Precision	86.75 ± 13.48	84.71 ± 13.47	89.21 ± 09.60

Table 5

Results of the performance evaluation of lung tumor segmentation on the VMHSC dataset based on the tumor size groups: Mean and standard values of the proposed and comparative methods in terms of F1-score, recall and precision are provided

Methods	Metrics (%)
	F1-score	Recall	Precision
U-Net (Baseline)	76.58±19.86	76.83±22.09	81.41±18.87
Dual Coupling-Net [15]	75.70±18.56	75.79±21.16	-
U-Net+Multi-scale module	77.11±18.97	78.34±20.11	81.06±19.22
U-Net+Multi-scale module+Dual-attention module	77.54±18.48	78.18±18.99	82.14±19.08
U-Net+Multi-scale module+Dual-attention module+Consistency Learning (Ours)	80.49 ± 15.94	79.06 ± 18.78	86.78 ± 13.47

Table 6 shows the performance evaluation results for each method on the NSCLC-Radiomics dataset. The proposed method showed the best performance compared to the other methods with an F1-score of 71.70%, recall of 68.24%, and precision of 79.33%. In Group 1 with a small tumor size and Group 2 with a medium tumor size, the same trend as in the VHSMC dataset was observed. In Group 3, the proposed method showed the best results with an F1 score, recall, and precision of 72.52%, 68.43%, and 79.89%, respectively. This means that the NSCLC-Radiomics dataset has a relatively large amount of data for Group 3 compared to the VHSMC dataset, and the large-sized tumor distributions also has a small number of variations ranging from 6 to 11 cm.

Table 6

Results of the performance evaluation of lung tumor segmentation on NSCLC-Radiomics dataset: Mean and standard values of the proposed and comparative methods in terms of F1-score, recall and precision are provided

Methods	Metrics (%)	Whole	Tumor size group
			Group 1	Group 2	Group 3
U-Net (Baseline)	F1 score	68.04±18.87	55.77±22.80	64.98±18.87	70.15±14.28
	Recall	65.72±22.56	44.64±23.92	66.87±22.56	66.94±17.59
	Precision	76.55±17.04	86.37±07.51	67.02±17.04	78.73±14.35
U-Net+Multi-scale module	F1 score	68.09±15.41	56.66±12.62	66.62±15.88	70.63±11.12
	Recall	63.55±18.83	44.32±14.44	63.88±19.34	66.83±15.69
	Precision	79.01±16.61	84.31±16.06	75.14±16.90	79.44±13.28
U-Net+Multi-scale module+	F1 score	68.57±15.63	58.77±07.26	67.64±14.76	71.21±12.14
Dual-attention module	Recall	65.88±20.94	45.55±05.60	71.52±20.96	68.21±16.57
	Precision	78.05±15.15	84.47±16.72	73.49±13.75	79.02±12.30
U-Net+Multi-scale module+Dual-	F1 score	71.70 ± 12.35	67.04 ± 04.39	71.73 ± 12.55	72.52 ± 10.08
attention module+Consistency	Recall	68.24 ± 16.76	55.52 ± 08.63	74.08 ± 16.48	68.43 ± 14.43
Learning (Ours)	Precision	79.33 ± 14.93	86.89 ± 08.83	73.96 ± 16.16	79.89 ± 12.79

Figure 6 qualitatively evaluated the performance of each segmentation method compared to the ground truth on the VHSMC dataset. In U-Net and U-Net using the multi-scale module, under-segmentation occurred in most of the tumors, and outliers occurred in other structures similar in shape to the tumors, such as the mediastinum. U-Net using the multi-scale dual-attention module showed better results than U-Net and U-Net using the multi-scale module, but under-segmentation still occurred in Groups 1 and 3. The proposed method showed the most similar segmentation results to the ground truth in all groups with various tumor sizes.

Fig. 6

Results of the qualitative evaluation of lung tumor segmentations on VHSMC dataset: (a) Original CT image, (b) Ground truth, (c) Lung tumor segmentation using U-Net (d) Lung tumor segmentation using U-Net with multi-scale module, (e) Lung tumor segmentation using U-Net with multi-scale dual-attention module and (f) Lung tumor segmentation using proposed method.

Figure 7 qualitatively evaluated the performance of each segmentation method compared to the ground truth on the NSCLC-Radiomics dataset. In U-Net, U-Net using the multi-scale module and U-Net using the multi-scale dual-attention, more severe under-segmentation occurred in most tumors than in the segmentation results of the VHSMC dataset. The proposed method showed the best segmentation results in all groups with various tumor sizes.

Fig. 7

Results of the qualitative evaluation of lung tumor segmentations on NSCLC-Radiomics dataset: (a) Original CT image, (b) Ground truth, (c) Lung tumor segmentation using U-Net (d) Lung tumor segmentation using U-Net with multi-scale module, (e) Lung tumor segmentation using U-Net with multi-scale dual-attention module and (f) Lung tumor segmentation using proposed method.

4 Discussions and conclusions

In this paper, we proposed a CL-MSDA-Net that can accurately segment lung tumors of various sizes on chest CT images. The proposed method uses not only the size-variant patch that preserve original size information, but also the size-invariant patch normalized to the average size of tumors while maintaining a similar tumor-to-background ratio. Through this method, small-sized tumors can learn semantic information within the tumor, and large-sized tumors can learn the surrounding information together. In addition, a pair of features were simultaneously learned through consistency learning to increase the learning stability between the networks in each branch, resulting in consistent segmentation results in lung tumors of different sizes. Finally, a multi-scale dual-attention module focused on the tumor area rather than surrounding structures to distinguish important areas at multiple scales.

The CL-MSDA-Net model showed the best segmentation performance on both the VHSMC and NSCLC-Radiomics datasets and showed an F1 score of 80.49%, recall of 79.06%, and precision of 86.78% on the VHSMC dataset, respectively, and showed an F1 score of 71.7%, recall of 68.24%, and precision of 79.33% on the NSCLC-Radiomics dataset, respectively. The VHSMC dataset has a wide distribution with a small amount of data in a large-sized tumor group, while the NSCLC-Radiomics has a constant size with a large amount of data in a large-sized tumor group. Although the dataset distributions are different, the proposed method showed the best performance in both datasets. In particular, small-sized lung tumors in the VHSMC dataset showed significant performance improvements, and large-sized lung tumors in the NSCLC-Radiomics dataset showed significant performance improvements by applying consistency learning.

Despite the overall good performance on both datasets, our study has several limitations. Because our method processes data in 2D slices, it is difficult to consider spatial continuity information in lung tumors. For lung tumors attached to the mediastinum, vessels, or chest wall, spatial continuity information can also be an important factor in segmentation. In addition, the proposed method could not be compared with other SOTA methods because the experimental configuration was different from the public datasets. In the future, these limitations are complemented, and it is expected that the tumor volume can be measured to evaluate the treatment response which is useful for predicting the prognosis.

Footnotes

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00207947) and a sabbatical year (2022) and research grant (2023) from Seoul Women’s University.

References

Sung

, Ferlay

, Siegel

R.L.

et al., Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, CA: A Cancer Journal for Clinicians 71(3) (2021), 209–249.

Greenberg

, Lazarev

, Frank

et al., Semi-automatic volumetric measurement of response to chemotherapy in lung cancer patients: How wrong are we using RECIST? Lung Cancer 108 (2017), 90–95.

Hayes

S.A.

, Pietanza

M.C.

, O’Driscoll

et al., Comparison of CT volumetric measurement with RECIST response in patients with lung cancer, European Journal of Radiology 85(3) (2016), 524–533.

Mozley

P.D.

, Bendtsen

, Zhao

et al., Measurement of tumor volumes improves RECIST-based response assessments in advanced lung cancer, Translational Oncology 5(1) (2012), 19–25.

Jiang

, Hu

Y.C.

, Liu

C.J.

et al., Multiple resolution residually connected feature streams for automatic lung tumor segmentation from CT images, IEEE Transactions on Medical Imaging 38(1) (2018), 134–144.

Yang

, Wu

, Li

et al., MSDS-UNet: A multi-scale deeply supervised 3D U-Net for automatic segmentation of lung tumor in CT, Computerized Medical Imaging and Graphics 92 (2021), 101957.

Chen

, Yang

, Zhang

et al., MAU-Net: Multiple Attention 3D U-Net for Lung Cancer Segmentation on CT Images, Procedia Computer Science 192 (2021), 543–552.

Banu

S.F.

, Sarker

, Kamal

et al., An attention-aware weight excitation u-net for lung nodule segmentation, Applied Sciences 11(21) (2021), 10132.

Zhang

, Qi

, Wu

et al., Multi-Scale Segmentation Squeeze-and-Excitation UNet with Conditional Random Field for Segmenting Lung Tumor from CT Images, Computer Methods and Programs in Biomedicine (2022), 106946.

10.

Byun

, Jung

, Hong

, Kim

B.S.

Lung tumor segmentation using dual-coupling net with shape prior based on lung and mediastinal window images from chest CT images, Journal of X-Ray Science and Technology 30(6), (2022), 1067–1083.

11.

Lee

, Jung

, Hong

, Kim

B.S.

Automatic Segmentation of Lung Cancer in Chest CT Images through Capsule Network-based Dual-Window Ensemble Learning, Journal of KIISE 48(8) (2021), 905–912.

12.

Detterbeck

F.C.

, Boffa

D.J.

, Kim

A.W.

, Tanoue

L.T.

The eighth edition lung cancer stage classification, Chest 151(1) (2017), 193–203.

13.

Aerts

H.J.W.L.

, Velazquez

E.R.

, Leijenaar

R.T.H.

et al., Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach, Nature Communications 5(1) (2014), 1–9.

14.

Garzelli

, Goo

J.M.

, Ahn

S.Y.

et al., Improving the prediction of lung adenocarcinoma invasive component on CT: value of a vessel removal algorithm during software segmentation of subsolid nodules, European Journal of Radiology 100 (2018), 58–65.

15.

https://radiopaedia.org/articles/windowing-ct.

16.

Ronneberger

, Fischer

, Brox

U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, (2015) pp. 234–241. Springer, Cham.

17.

Tao

, Zhang

, Wang

et al., U-High resolution network (U-HRNet): cloud detection with high-resolution representations for geostationary satellite imagery, International Journal of Remote Sensing 42(9) (2021), 3511–3533.

18.

, Wang

et al., SA-Net: A scale-attention network for medical image segmentation, PloS One 16(4) (2021), e0247388.

19.

Woo

, Park

, Lee

J.Y.

, Kweon

I.S.

Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), (2018), pp. 3–19.

Hyperparameters	Methods
	U-Net (Baseline)	U-Net+Multi-scale module	U-Net+Multi-scale module+Dual-attention module	U-Net+Multi-scale module+Dual-attention module+Consistency Learning (Ours)
Optimizer	Adam Optimizer
Initial learning rate	1e-4 (VHSMC), 1e-3 (NSCLC-Radiomics)
Loss function	Binary Cross Entropy loss+Dice loss			Binary Cross Entropy loss+Dice loss+L2 loss
Epoch (patience)	100 (25)
Batch size	10