Measuring pure ground-glass nodules on computed tomography: assessing agreement between a commercially available deep learning algorithm and radiologists’ readings

Abstract

Background

Deep learning algorithms (DLAs) could enable automatic measurements of solid portions of mixed ground-glass nodules (mGGNs) in agreement with the invasive component sizes measured during pathologic examinations. However, the measurement of pure ground-glass nodules (pGGNs) based on DLAs has rarely been reported in the literature.

Purpose

To evaluate the use of a commercially available DLA for the automatic measurement of pGGNs on computed tomography (CT).

Material and Methods

In this retrospective study, we included 68 patients with 81 pGGNs. The maximum diameter of the nodules was manually measured by senior radiologists and automatically segmented and measured by the DLA. Agreement between the measurements by the radiologist and DLA was assessed using Bland–Altman plots, and correlations were analyzed using Pearson correlation. Finally, we evaluated the association between the radiologist and DLA measurements and the invasiveness of lung adenocarcinoma in patients with pGGNs on preoperative CT.

Results

The radiologist and DLA measurements exhibited good agreement with a Bland-Altman bias of 3.0%, which were clinically acceptable. The correlation between both sets of maximum diameters was also strong, with a Pearson correlation coefficient of 0.968 (P < 0.001). In addition, both sets of maximum diameters were larger in the invasive adenocarcinoma group than in the non-invasive adenocarcinoma group (P < 0.001).

Conclusion

Automatic pGGNs measurements by the DLA were comparable with those measured manually and were closely associated with the invasiveness of lung adenocarcinoma.

Keywords

Pure ground-glass nodules computed tomography deep learning algorithm radiologist measurement agreement

Introduction

With the widespread use of low-dose chest computed tomography (CT) for the screening of lung cancer, the detection rate of small pulmonary nodules and lung cancers with ground-glass nodules (GGNs) has gradually increased (1 –3). GGNs are categorized into two types based on the solid components of the nodules: mixed GGNs (mGGNs) and pure GGNs (pGGNs) (1 –3). PGGNs are generally considered radiologically non-invasive lung adenocarcinomas and are an indication for sublobar resection (4). Based on TNM staging (8th edition) for lung cancer, pGGNs <3 cm in size are classified as clinical Tis; otherwise, they are upstaged as T1a (5). However, approximately 16%–27% of pGGNs are pathologically diagnosed as invasive lung adenocarcinoma (6). Recent research found that the size of pGGNs on CT (maximum diameter) was the only radiographic parameter significantly related to lung cancer invasiveness (7).

Given the clinical importance of the maximum pGGN diameter measurement, recent recommendations for pGGN management have focused on the size of the entire lesion (8,9). Generally, the maximum diameter of pGGNs is measured manually by radiologists. However, precise and reproducible measurement of pGGNs is challenging because of the inevitable inter- and intra-observer variability (10). To avoid this, some studies have explored the feasibility of semi-automated pGGN measurements, which show less variability and better agreement with the size of the invasive components than those of manual measurements (11). However, the semi-automated methods are not sufficiently precise and frequently demand supplemental manual adjustment.

Currently, deep learning algorithms (DLAs) for lung lesion segmentation have proven to be more robust compared to conventional methods (12). Ahn et al. (13) showed that DLAs could enable automatic measurements of solid portions of mGGNs in agreement with the invasive component sizes measured during pathologic examinations. However, the measurement of pGGNs based on DLAs has rarely been reported in the literature. Thus, the aim of the present study was to evaluate the performance of a commercially available DLA for automatic measurement of the maximum diameter of surgically proven lung adenocarcinomas manifesting as pGGNs, and to further evaluate the association between DLA measurement and the invasiveness of lung adenocarcinoma.

Material and Methods

Lesion selection

This retrospective study, conducted between January 2018 and June 2021, collected the data of patients who were diagnosed with and underwent curative resection for lung adenocarcinoma, and had undergone thoracic CT within 30 days before the surgery. The inclusion criteria for the patients were as follows: (i) pathologically diagnosed with lung adenocarcinoma; (ii) availability of thin-section CT images (section thickness = 0.625–1.250 mm); (iii) pGGNs observed on CT images; and (iv) did not receive preoperative chemotherapy or chemoradiotherapy. The exclusion criteria were as follows: (i) CT scans with unqualified images (i.e. artifacts); and (ii) non-measurable lesions on CT images. This study was approved by the Ethics Committee of Affiliated Hospital of Southwest Medical University (No. KY2020147), and informed consent was obtained from all participants.

CT image acquisition

Chest CT was performed using multidetector-row CT systems (uCT550 or uCT760; United Imaging Healthcare, Shanghai, PR China). CT imaging acquisition parameters were as follows: 120 kVp and 270 mAs; field of view = 375 mm; and pitch = 1.2. Images were reconstructed using a sharp reconstruction kernel with a section thickness of 0.625–1.250 mm.

Determination of pGGNs by radiologists

All lung lesions were assessed by two expert thoracic radiologists (with >10 years of experience in thoracic radiology), who were blinded to most of the study information except the presence of pGGNs. The Fleischner Society’s definition of GGNs was used, which states that a GGN is a radiographic appearance demonstrating hazy opacity with the existence of implicit pulmonary vessels or bronchial structures on high-resolution CT, whereas a pGGN was described as a nodule without a solid composition (8,9).

All pGGNs were measured using a picture archiving and communication system (PACS); lesion size on the CT image was defined as the maximum diameter in the axial plane in accordance with the 8th edition of TNM staging (8,9). A lung window setting (1600 Hounsfield Units [HU]; window level = −600 HU) was used in the commercial PACS; the mean of manual measurements were calculated using the following formula: (radiologists1 + radiologists2)/2.

Finally, laterality, primary site, and CT attenuation values were assessed from the region of interest, covering the majority of pGGNs, excluding blood vessels and air space.

Maximum diameter measurement by the DLA

A commercially available DLA (Deepwise 20201130fix1a, Deepwise and League of PHD Technology Co., Ltd., Hangzhou, PR China) was employed in this study. This algorithm was designed to perform automated segmentation of a given pGGN, thus displaying the maximum diameter, longest perpendicular diameter, and lesion contour as an overlay in the axial plane. A recurrent neural network as the framework embedded the convolution neural network, and more details can be found in previously published papers (14,15).

The proposed algorithm was pretrained on the Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) dataset (16). The LIDC-IDRI dataset includes 1010 patients with 1018 scans and 910 nodules, which were annotated by at least four radiologists. The dataset was divided into training, validation, and testing sets, with 350, 67, and 493 nodules, respectively. The training set contained data from multiple CT manufacturers, including machines from GE, SIEMENS, Philips, Toshiba, and so on; thus this is very important to build a well-generalized model. It was expected to present a good performance for CT scans with a slice thickness ≤2 mm. This software was approved by China's National Medical Products Administration (NMPA) (www.deepwise.com).

This automatic segmentation was achieved via the radiologist loading the raw CT data (DICOM format) of each patient into the DLA and then recording the measurement data.

Pathologic evaluation

Postoperative or intraoperative frozen section pathological assessment was performed according to the International Association for the Study of Lung Cancer/American Thoracic Society/European Respiratory Society classification, encompassing atypical adenomatous hyperplasia, adenocarcinoma in situ (AIS), minimally invasive adenocarcinoma (MIA), and invasive adenocarcinoma (IAC) (17). Participants in this study were classified as non-IAC or IAC according to the pathological findings. The CT and histopathology images of non-IAC and IAC pGGNs are shown in Fig. 1.

Statistical analyses

Measurements of pGGN maximum diameter by the DLA and radiologists were compared using paired t-tests. To evaluate the agreement between both measurement types, we hypothesized that they would fall within a predefined clinically acceptable scope. Thus, the Bland–Altman method (18) was used to evaluate the 95% limits of agreement (LOA) between the DLA and radiologist measurements, with a priori acceptable clinically significant limits of agreement for diameter measurement set at ±25%, which is the threshold for clinically significant change (19). In the Bland–Altman plot, the outliers were <5%, indicating that the two methods were in good agreement with each other (18).

Correlation analysis between the DLA and radiologist measurements was performed using Pearson correlation. Inter- and intra-reader reproducibility between the DLA and radiologist measurements were assessed with the intraclass correlation coefficient (ICC): reproducibility was defined as poor (ICC < 0.400), fair to good (ICC = 0.400–0.750), or excellent (ICC > 0.750). In addition, the paired t-test was used to compare the measurement time difference between groups. Finally, numerical data between the non-IAC and IAC groups were compared using independent-sample t-tests, and categorical variables were compared using the chi-square test.

The diagnostic efficacy of the DLA and radiologist measurements were compared using the area under the curve (AUC) of receiver operating characteristic (ROC) curves. An AUC value of >0.750 was considered the threshold for good diagnostic efficacy, and the DeLong test was used to compare AUCs (20). Statistical analyses were performed using MedCalc statistical software version 15.2.2 (MedCalc Software, Ostend, Belgium). Statistical significance was set at P < 0.05.

Results

Demographic and clinical characteristics

We enrolled 68 patients (with 81 lesions) (18 men, 50 women; mean age = 52.59 ± 10.55 years; age range = 23–69 years) who had undergone curative resection of lung adenocarcinomas and had pGGN findings on preoperative CT. According to the final pathologic diagnosis, the 81 lesions comprised 32 (39.5%) IACs and 49 (60.5%) non-IACs.

Agreement between the DLA and radiologist measurements

There was excellent repeated measurement between the DLA and mean of manual measurement (ICC = 0.967). The DLA and each radiologist yielded an ICC value of 0.951 and 0.950, respectively. Meanwhile, There was a strong correlation between the DLA and mean of manual measurement (Pearson correlation coefficient = 0.968); the DLA and each radiologist had a Pearson correlation coefficient of 0.952 and 0.954, respectively. The maximum diameter of pGGNs measured by the DLA was 14.62 ± 5.81 mm, which was slightly larger than that measured by the radiologists (14.26 ± 5.83 mm). This difference was significant (P < 0.001) as shown in the violin plot quantitative analysis presented in Fig. 2. The Bland–Altman analysis showed good agreement between the mean of manual measurements and DLA measurements. Moreover, upon inspection of the Bland–Altman plot (Fig. 3), 3/81 (3.7%) outliers were noted, and as the maximum diameter of the lesion decreased, more outliers were observed. Furthermore, the Bland–Altman bias was 3.0%, with a 95% LOA of −17.8%–23.8%, falling within the predefined clinically acceptable scope. Similar results were observed at the DLA and each radiologist (Figs. 4 and 5). Bland–Altman plots for the DLA and radiologist 1 showed consistent results, with 2/81 (2.5%) outliers noted; Bland–Altman plots for DLA and radiologist 2 also showed consistent results, with 3/81 (3.7%) outliers noted.

Measurement time between the DLA and radiologist measurements

The mean measurement time per case was 72.30 ± 9.74 s for radiologist 1, 71.16 ± 8.57 s for radiologist 2, and 50.07 ± 6.20 s for the DLA. The measurement time of the DLA was significantly shorter than those of radiologists 1 and 2 (P < 0.001).

Radiographic pGGN characteristics associated with IAC

Radiographic differences between the IAC and non-IAC groups were included in our final analysis (Table 1). Both the DLA and radiologist maximum diameter measurements were significantly larger in the IAC group than in the non-IAC group (P < 0.001). There was no significant difference in CT attenuation value (P = 0.197), laterality (P = 1.000), or primary site (P = 0.064). The ROC curve analysis on the radiologist and DLA measurements is shown in Fig. 6. The radiologists’ measurements for differentiating IAC from non-IAC presented an AUC of 0.826 (95% confidence interval [CI] = 0.726–0.901), with a sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of 84.37%, 69.39%, 64.30%, and 87.20%, respectively. The measurement by the DLA did not significantly improve the AUC (0.833, 95% CI = 0.733–0.906, sensitivity = 87.5%, specificity = 69.39%, PPV = 65.10%, NPV = 89.5%, DeLong test; P = 0.647).

Table 1.

Radiographic appearance of IAC and non-IAC pGGNs.

Characteristic	Non-IAC	IAC	P value
Characteristic	(n = 49)	(n = 32)	P value
Laterality			1.000
Right	29 (59.2)	19 (59.4)
Left	20 (40.8)	13 (40.6)
Primary site			0.064
Upper lobe	32 (65.3)	28 (87.5)
Lower lobe	15 (30.6)	4 (12.5)
Middle lobe	2 (4.08)	0 (0.00)
CT attenuation value	−494.90 ± 135	−453.52 ± 143	0.197
DLA	12.0 ± 4.62	18.6 ± 5.15	<0.001
Radiologist	11.7 ± 4.63	18.2 ± 5.35	<0.001

Values are given as n (%) or mean ± SD.

CT, computed tomography; DLA, deep learning algorithm; IAC, invasive adenocarcinoma; pGGNs, pure ground-glass nodules.

Discussion

Precise and reproducible measurement of the maximum diameter on CT images is important for the assessment and management of pGGNs. However, the precision and reproducibility of manual pGGN measurements is challenging due to inevitable inter- and intra-observer variability. Earlier semi-automated methods also demonstrated restrictions because of the low rate of sufficient original segmentation and the frequent need for supplemental manual adjustment (11). In the present study, we estimated the feasibility and performance of a commercial DLA for the automatic measurement of maximum diameter of pGGNs on CT images.

Our study showed that the maximum diameter of pGGNs measured by the DLA was slightly larger than that measured by the radiologists (P < 0.001). This discrepancy may have been due to the low contrast between the margin of pGGNs and adjacent lung tissues; the radiologists may have ignored these subtle contrast differences. In the measurement of maximum diameter of pGGNs on CT images, the DLA and radiologist measurements were strongly correlated, with a Pearson correlation coefficient of 0.968 (P < 0.001). In the Bland–Altman plot, the DLA showed close agreement with the radiologists’ measurements, with 3/81 (3.7%) outliers. The Bland–Altman bias was 3.0%, with a 95% LOA of −17.8%–23.8%, which falls within the predefined clinically acceptable scope, indicating that DLAs may be an acceptable substitute for radiologist measurements. Furthermore, upon inspection of the Bland–Altman plot, we noticed more outliers as the maximum diameter of the lesion decreased, which is most likely due to measurement variability, differentially affecting smaller lesions.

In addition, we reviewed the radiographic appearance of non-IAC and IAC lesions. Larger maximum diameters estimated by the DLA and radiologists were found in the IAC group compared with the non-IAC group (P < 0.001). Previous studies have shown that only CT size (maximum diameter) is associated with the invasiveness of pGGNs, rather than other radiographic features (including CT attenuation value, shape, margin, bubble lucency, or pleural indentation) (7,21 –24). Nelson et al. found that no distinguished radiographic appearance was associated with lymphovascular invasion or lymph node metastasis, suggesting that radiographic features were not influential. Consequently, some researchers have selected CT size and solid components as the classifying standards for GGNs (23).

Based on the clinical aspect, it is essential that IAC should be distinguished from AIS/MIA, rather than MIA/IAC from AIS. Several reports have suggested that patients with pGGNs have excellent survival; thus, the time window for pGGN intervention is broad. Additionally, for MIA pGGNs, regular follow-up remains an alternative (25). Thus, we applied DLA and radiologist measurements to differentiate IAC from non-IAC pGGNs. Our results revealed that both DLA and radiologist measurements could achieve high sensitivity and NPV, but lower specificity and PPV. Both DLA and radiologist measurements had high AUCs (>0.750), which indicate good diagnostic efficacy (20).

Although the DLA did not significantly improve the AUC (0.833 vs. 0.826, DeLong test; P = 0.647), we suggest that DLA measurement may be a reliable method for estimating the invasiveness of pGGNs. Our results indicate that DLA without additional input could be comparable with the radiologists’ level of agreement and reliability in maximum diameter measurement for pGGNs. In addition, DLA measurements were also associated with the invasiveness of pGGNs. Use of automated measurements may be extended to facilitate the automated classification of lesions based on the presence and size of pGGNs and subsequently promote the efficient estimation and management planning of pulmonary lesions (i.e. automatic lesion management) with large CT datasets, such as national lung cancer screening programs.

The present study has some limitations. First, this was a single-center retrospective study, and the number of cases was relatively limited; thus, further large-scale research is required. Second, there was an unavoidable verification bias, as we only enrolled patients who underwent surgical resection for pGGNs, which were diagnosed pathologically as lung adenocarcinomas. Moreover, the ratio of IAC was relatively high. Third, both the DLA and radiologists measured only the maximum diameter of pGGNs in this study. Thus, whether another radiographic characteristic can achieve a similar performance remains uncertain.

In conclusion, automatic measurements of lung adenocarcinoma manifesting as pGGNs by a commercially available DLA were comparable to those of the two radiologists and presented a strong association with lung adenocarcinoma invasiveness. With further optimization, including training for segmentation, the DLA could have wider application in the management of pGGNs.

Footnotes

Acknowledgements

We thank Xiangtan Central Hospital and the Affiliated Hospital of Southwest Medical University for supplying the patients used in this study. We also thank WuHan No. 1 Hospital for their excellent collaboration in this project.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Wei Zhang

References

Deng

Wang

, et al. Lung adenocarcinomas manifesting as radiological part-solid nodules define a special clinical subtype. J Thorac Oncol 2019;14:617–627.

Zhang

, et al. CT features and quantitative analysis of subsolid nodule lung adenocarcinoma for pathological classification prediction. BMC Cancer 2020;20:60.

Zhang

Wen

, et al. Distinct prognostic factors in patients with stage I non-small cell lung cancer with radiologic part-solid or solid lesions. J Thorac Oncol 2019;14:2133–2142.

Asamura

Hishida

Suzuki

, et al. Radiographically determined noninvasive adenocarcinoma of the lung: survival outcomes of Japan Clinical Oncology Group 0201. J Thorac Cardiovasc Surg 2013;146:24–30.

Travis

Asamura

Bankier

, et al. The IASLCl Lung Cancer Staging Project: proposals for coding T categories for subsolid nodules and assessment of tumor size in part-solid tumors in the forthcoming eighth edition of the TNM classification of lung cancer. J Thorac Oncol 2016;11:1204–1223.

Kitami

Sano

Hayashi

, et al. Correlation between histological invasiveness and the computed tomography value in pure ground-glass nodules. Surg Today 2016;46:593–598.

Zhang

Wang

, et al. Computed tomography density is not associated with pathological tumor invasion for pure ground-glass nodules. J Thorac Cardiovasc Surg 2021;162:451–459.e3.

MacMahon

Naidich

Goo

, et al. Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 2017;284:228–243.

Martin

Kanne

Broderick

, et al. Lung-RADS: pushing the limits. Radiographics 2017;37:1975–1993.

10.

Scholten

Jacobs

van Ginneken

, et al. Detection and quantification of the solid component in pulmonary subsolid nodules by semiautomatic segmentation. Eur Radiol 2015;25:488–496.

11.

Kim

Goo

Park

. Evaluation of T categories for pure ground-glass nodules with semi-automatic volumetry: is mass a better predictor of invasive part size than other volumetric parameters? Eur Radiol 2018;28:4288–4295.

12.

Wang

Zhou

Liu

, et al. Central focused convolutional neural networks: developing a data-driven model for lung nodule segmentation. Med Image Anal 2017;40:172–183.

13.

Ahn

Lee

Noh

, et al. Use of a commercially available deep learning algorithm to measure the solid portions of lung cancer manifesting as subsolid lesions at CT: comparisons with radiologists and invasive component size at pathologic examination. Radiology 2021;299:202–210.

14.

Wang

Yang

, et al. Natural history of pathologically confirmed pulmonary subsolid nodules with deep learning–assisted nodule segmentation. Eur Radiol 2021;31:3884–3897.

15.

Tang

, et al. Long-term follow-up of persistent pulmonary pure ground-glass nodules with deep learning–assisted nodule segmentation. Eur Radiol 2020;30:744–755.

16.

Armato

3rd McLennan

Bidaut

, et al. The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys 2011;38:915–931.

17.

Travis

Brambilla

Noguchi

, et al. International association for the study of lung cancer/American thoracic society/European respiratory society international multidisciplinary classification of lung adenocarcinoma. J Thorac Oncol 2011;6:244–285.

18.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307–310.

19.

Gietema

Schaefer-Prokop

Mali

, et al. Pulmonary nodules: interscan variability of semiautomated volume measurements with multisection CT– influence of inspiration level, nodule size, and segmentation performance. Radiology 2007;245:888–894.

20.

Jones

Athanasiou

. Summary receiver operating characteristic curve analysis techniques in the evaluation of diagnostic tests. Ann Thorac Surg 2005;79:16–20.

21.

Lee

Park

, et al. Lung adenocarcinoma invasiveness risk in pure ground-glass opacity lung nodules smaller than 2 cm. Thorac Cardiovasc Surg 2019;67:321–328.

22.

Sun

Zhan

, et al. Ground glass opacities: imaging, pathology, and gene mutations. J Thorac Cardiovasc Surg 2018;156:808–813.

23.

Nelson

Godoy

MCB

Benveniste

, et al. Clinicoradiographic predictors of aggressive biology in lung cancer with ground glass components. Ann Thorac Surg 2018;106:235–241.

24.

Antonoff

. The search for reliable markers of ground glass opacity prognosis: the truth remains largely unknown. J Thorac Cardiovasc Surg 2018;156:814–815.

25.

Mao

She

Zhu

, et al. A proposal for restaging of invasive lung adenocarcinoma manifesting as pure ground glass opacity. Ann Thorac Surg 2019;107:1523–1531.