Abstract
Background
Preanalytical processes—including sample collection, fixation, and handling—significantly influence the integrity of nucleic acids, which in turn impacts the accuracy of next-generation sequencing (NGS)-based biomarker analyses. Though commonly utilized, cytological specimens remain underrepresented in studies evaluating preanalytical quality.
Methods
We retrospectively analyzed 1048 NGS tests performed between May 2019 and December 2022, encompassing cytological materials (smear slides and cell blocks), small/core biopsies, and resection specimens. Quality control (QC) was assessed using pre-sequencing cycle threshold (Ct) values and post-sequencing metrics such as fragment length, on-target unique fragments, deduplication ratio, and unique start.
Results
Overall, 22.9% of samples did not meet manufacturer Ct thresholds. The final QC failure rates were 12.4% for RNA and 3.5% for DNA. RNA QC failures were significantly more frequent in cytology and resection specimens compared to biopsies. However, post-sequencing analysis revealed that smear samples exhibited significantly higher median DNA and RNA fragment lengths compared to FFPE specimens, indicating superior nucleic acid preservation in samples that passed QC.
Conclusion
Cytological specimens are viable for NGS-based molecular testing, particularly when preanalytical variables are standardized and optimized. DNA quality from smears may even exceed that of FFPE samples, while RNA integrity remains more sensitive to preparatory conditions. These findings support the broader use of cytological materials in molecular diagnostics and highlight the need for tailored preanalytical workflows.
Keywords
Introduction
Next-generation sequencing (NGS) has brought about a new era of cancer research, enabling scientists to explore the complex genomic and transcriptomic changes associated with the disease. Although NGS provides a great deal of diagnostic data often aimed at identifying clinically actionable biomarkers, its effectiveness depends on the quality of the nucleic acids extracted from patient samples.
Preanalytical processes, such as sample collection and storage, are critical in determining the success of NGS-based molecular analysis.1–7 Any misstep during these crucial steps can have cascading effects, leading to degraded and fragmented nucleic acids, unwanted contaminants, or inconsistencies. These alterations can significantly compromise the accuracy of biomarker detection, potentially leading to false positives or negatives. Consistent preanalytical procedures are critical for making NGS-based biomarker analysis robust and reproducible. After considering all these factors, including specimen collection methods, the type of fixative used, fixation duration, tissue processing protocols, and nucleic acid extraction kits and methods, laboratories will be better equipped to generate high-quality and reproducible nucleic acid samples. Preanalytical processes include type of specimen collected (eg, blood or tissue), type of fixatives and fixation duration, tissue processing protocols chosen, and nucleic acid extraction methods/kits/protocols used/followed.8–10 It is very well known that these procedures are not standardized and vary among labs, since all labs adjust these manufacturers-defined protocols according to their needs and ease of use. In this case, a better understanding and control of these variables can improve the accuracy of molecular testing.
In pathology laboratories, various sample types—including aspiration smears, cell blocks, formalin-fixed paraffin-embedded (FFPE) tissues, and, less commonly, fresh frozen tissue—are routinely used for molecular testing, each with varying success rates. It is well established that fresh frozen tissue samples yield the highest quality DNA and RNA.11–13 However, FFPE samples are robust and widely used as the sample of choice for molecular testing across most pathology departments. Cytologic samples, such as smears and cell blocks, also hold significant potential, mainly as fine needle aspiration biopsy is increasingly employed. Utilizing these samples for downstream molecular applications should be a logical step forward. However, aspiration cytology often results in limited amounts of material.
Among the various metrics available to evaluate nucleic acid integrity, we selected cycle threshold (Ct) values as the primary pre-sequencing quality control (QC) parameter, given their routine use, accessibility, and clinical applicability. While specimens with necrosis, poor fixation, or insufficient nucleic acid concentration were excluded from the analysis, in addition, sequencing-derived parameters (such as fragment length, on-target unique fragments / deduplication ratio / coverage metrics) were also incorporated to further evaluate nucleic acid quality. Our study aims to evaluate how different specimen types and their pre-analytical handling impact the quality of nucleic acids and, consequently, the success of NGS-based biomarker analysis.
Methods
This retrospective study included the data of NGS analyses performed at the Department of Pathology between May 2019 and December 2022. DNA- and RNA-based panels were evaluated during this period. After excluding bone marrow samples, ctDNA (liquid biopsy) samples, sarcoma panels, homologous recombination deficiency (HRD) panels, and samples with unknown specimen types (eg, unspecified biopsy material or cytological material), a total of 1048 specimens were included in the analysis.
The exclusion of sarcoma panels and liquid biopsies was done to achieve a more homogenous dataset. Additionally, bone marrow samples were excluded due to their distinct handling requirements, specifically variable acid decalcification/treatment, which differ significantly from solid tissue or cytology specimens.
Preparation of Samples
All specimens (resection specimens, small/core biopsies, cell blocks, and smear slides) were routinely evaluated by expert pathologists and/or cytopathologists for the eligibility of NGS testing (for the cellularity and tumor cell density of the sample). Only specimens with a tumor presence of at least 20% and more than 500 tumor cells were considered for inclusion. In exceptional circumstances, such as clinical urgency, a small number of specimens with 10% tumor content were also included. Tumor harboring smear slides, that would be scraped, were digitally scanned and archived using a digital slide scanner (Philips IntelliSite UltraFast scanner, Koninklijke Philips, Amsterdam, Netherlands) before nucleic acid isolation. For cell blocks and FFPE samples, at least 6 sections of 5 µm thickness were taken into an Eppendorf tube from each block. If macro-dissection was indicated to enrich tumor cell density, sections were taken onto glass slides, and tumor regions were meticulously scraped into the tube.
For smears, coverslip detachment was adapted to the type of sealant employed. Film-sealed slides had coverslips removed after immersing them in acetone for 15 to 30 minutes, whereas entellan-sealed slides required a longer xylene immersion, sometimes for days depending on the age of the smear slide. Stained smear slides were scraped to collect cells, cautiously avoiding benign epithelial, stromal, or inflammatory cell contamination. RNA or DNA extraction from each patient utilized 1 to 3 smears as necessary, with a cumulative minimum of 500 tumor cells for each nucleic acid type.
Extraction and Sequencing of Nucleic Acids
RNA and DNA extraction and purification were performed using the Qiagen AllPrep Kit (Qiagen, Germany), and nucleic acid concentration was measured with a Qubit Fluorometer (Thermo Fisher Scientific, Waltham, MA). Sequencing libraries were prepared using the ArcherDx targeted panel kits (FusionPlex Comprehensive Thyroid and Lung Panel, VariantPlex Solid Tumor Panel).
These library preparation kits use a technology named anchored multiplex PCR (AMP), where the nucleic acid fragments are ligated to adapters with molecular barcodes and sample indices, followed by enrichment with unidirectional gene-specific primers (GSPs) that capture both known and unknown variants. An analysis pipeline is also provided (Archer™ Analysis 7.4, ArcherDx). The core AMP technology utilizes primer extension to enable efficient and specific amplification of DNA, including both genomic DNA and cDNA.
QC metrics for RNA included standard PCR reactions using housekeeping gene primer sets. According to the kit manufacturers’ specifications, a sample is considered eligible (or passed QC) if the cycle threshold (Ct) values were below 27 for cDNA (complementary DNA) and below 11 for gDNA (genomic DNA). Sample type, type of the nucleic acid isolated, and Ct value of the QC PCR were recorded.
Post-Sequencing QC Metrics
The following sequencing-derived QC metrics were used to assess the quality and integrity of extracted nucleic acids across different specimen types:
Statistical Analysis
Descriptive statistics were used to summarize the data. Categorical variables were compared using the Chi-square test or Fisher's exact test, as appropriate. Continuous variables were analyzed using a 2-tailed Student's t-test, based on assumptions of normal distribution.
For sequencing-derived QC parameters—on-target unique fragment percentage, deduplication ratio
A P-value of <.05 was considered statistically significant. Given the predefined hypothesis-driven approach rather than a high-dimensional exploratory analysis, the use of P-values was deemed appropriate. All statistical analyses were conducted using SPSS software
Results
Overview
Our comprehensive analysis encompassed 1048 studies, categorized by sample type as follows: 207 consisted of cytological materials (84 smear slides and 123 cell blocks), 494 were small/core biopsies (eg, tru-cut biopsies), and 347 were resection specimens. Among these, 550 samples were derived from primary tumors, while 498 were from metastatic lesions.
Nucleic Acid Quality Assessment
Upon examining nucleic acid quality, we observed that 240 studies did not meet the manufacturer's recommended QC criteria based on Ct thresholds. Among these, 138 samples were further processed despite failing the threshold, primarily due to clinical urgency or specific requests from treating physicians; 63 revealed detectable mutations, while 75 displayed no mutations but successfully passed sequencing QC. Among the Ct-failed samples that were nevertheless processed due to clinical necessity or lack of additional material, most had values only slightly above the manufacturer's thresholds (28-30 for RNA, 11-12 for DNA). In our institutional workflow, a slightly more flexible Ct criterion is occasionally applied in such instances; however, when no pathogenic mutation is detected, repeat sequencing on a newly obtained specimen is routinely recommended.
However, a total of 102 studies could not proceed to sequencing due to QC failure, underscoring the critical influence of preanalytical variables on sample suitability for molecular analysis. Overall, RNA (cDNA) samples demonstrated a failure rate of 12.4% (91/732), while DNA (gDNA) samples had a lower failure rate of 3.5% (11/316) (Figure 1).

Pass/fail rates of quality check PCR by nucleic acid type.
A more in-depth analysis of these 102 QC-failed studies revealed distinct patterns. Sixty specimens were tested only for RNA and failed, 25 exhibited RNA QC failure despite passing gDNA QC, 12 showed concurrent RNA and gDNA failure (corresponding to six patients), three were tested only for gDNA and failed, and two demonstrated gDNA QC failure despite passing RNA QC. These findings confirm that RNA degradation remains the predominant cause of QC failure, whereas gDNA failure is relatively low.
Among the RNA-failed samples (n = 91), 18 were smear slides, 15 were cell blocks, 23 were small/core biopsies, and 35 were resection specimens. Among the DNA-failed specimens (n = 11), 1 was a smear, 1 was a cell block, 5 were small/core biopsies, and 4 were resections. The failure rates for RNA were 25.0% (18/72) in smears, 17.0% (15/88) in cell blocks, 6.8% (23/336) in biopsies, and 14.8% (35/236) in resection specimens. Stratification of the RNA failures revealed significant variability across specimen types (P < .001), with higher failure rates in cytological and resection materials compared to biopsies. In contrast, gDNA failures showed no significant variability across specimen types (P > .3). The respective failure rates for DNA were 4.8% (1/21) in smears, 2.9% (1/35) in cell blocks, 3.4% (5/149) in biopsies, and 3.6% (4/111) in resections (Table 1).
Distribution of RNA and DNA QC Failures Across Sample Types.
Among QC-failed samples, the estimated mean tumor fraction was 44% in smears, 44% in cell blocks, 38% in small/core biopsies, and 52% in resection specimens for RNA-based assays, and 80%, 60%, 26%, and 24%, respectively, for DNA-based assays. RNA failures were therefore not limited to low-cellularity specimens, while DNA failures occurred even in high-tumor-fraction specimens, underscoring the impact of fixation and processing variables on nucleic acid quality.
These results highlight the combined effects of input quantity and fixation-related degradation on QC performance, supporting the need for tailored preanalytical optimization for both cytology and resection specimens.
Comparison of Internal and External Sample Quality
The QC failure rate for all (cytology & biopsy & resection) in-house samples (collected at KUH) was 10.1% (68 out of 674 samples). External (consult) samples, on the other hand, had a QC failure rate of 10.3% (34 out of 374). This slight difference did not reach statistical significance, indicating comparable nucleic acid quality between external and in-house samples.
Sequencing-Based QC Metrics (Table 2)
Post-sequencing QC analysis was restricted to 892 samples (607 of 641 RNA and 285 of 305 DNA) due to the unavailability of retrospective raw data files for 54 specimens.
Summary of Sequencing-Based QC Metrics.
Fragment Length Analysis
Overall, smear cytology consistently provided the highest-quality nucleic acid fragments, whereas resection specimens yielded the lowest fragment lengths across both RNA and DNA.
On-Target Unique Fragment Percentage
Deduplication Ratio
Average Unique Start Sites (per GSP2)
Discussion
The critical role of preanalytical processes extends beyond simple contamination or degradation. 14 Our findings support the opening statement of this section by highlighting how key factors—such as specimen type, preanalytical handling protocols, and RNA/DNA-specific challenges—directly influence nucleic acid quality and subsequent molecular analysis success rates. For example, the superior gDNA and cDNA fragment length observed in smear samples emphasize the benefits of avoiding formalin fixation. In contrast, these samples’ higher RNA QC failure rates underscore the need for tailored protocols to address their inherent variability. Together, these observations demonstrate the profound impact of preanalytical processes on ensuring the reliability and reproducibility of NGS-based workflows.
The integrity and quantity of the extracted nucleic acids are crucial in determining the sensitivity and specificity of NGS analysis. Although Ct values are surrogate markers rather than absolute indicators of nucleic acid quality, they represent practical and widely adopted measures in clinical workflows. In this study, their use was particularly relevant since pre-analytical exclusions (eg, necrotic, decalcified, or inadequately fixed specimens) had already been applied, leaving Ct thresholds as the most reliable indicator of input suitability.
A significant factor where preanalytical processes can vary is the type of sample. For example, large resection specimens may be overfixed or underfixed, while cytological samples are often fixed with ethanol or air-dried.15,16
We demonstrated that cytological samples (smears and cell blocks) and FFPE samples are suitable for downstream molecular applications, achieving acceptable success rates (∼90% for all), consistent with the literature.17,18 Genomic DNA-based studies showed a higher success rate than RNA-based (cDNA) assays, reflecting the inherent fragility of RNA and its susceptibility to degradation, particularly during formalin fixation and paraffin processing. These differences can in part be explained by fixation-related variables: biopsy specimens, with their smaller size, are generally fixed under more optimal and uniform conditions, whereas large resections are at risk of delayed formalin penetration, suboptimal fixation, or autolysis. By contrast, cytology samples are not affected by formalin in the same way but may fail due to limited tumor cellularity or material insufficiency. RNA degradation is a known challenge that significantly impacts molecular testing performance.19,20
QC assessment in this study combined both pre-sequencing and sequencing-derived parameters. Ct thresholds served as a pre-sequencing indicator, guiding decisions on whether samples were suitable for library preparation. In contrast, fragment length, on-target unique fragment percentage, deduplication ratio, and average unique start sites are all sequencing-derived metrics that reflect different aspects of nucleic acid integrity and library performance. Fragment length indicates the degree of nucleic acid fragmentation, on-target unique fragments reflect specificity and usable reads, deduplication ratio represents library complexity and amplification bias, and unique start sites denote overall sequence diversity.
Together, these complementary metrics demonstrated that DNA is relatively robust across all specimen types, whereas RNA displayed significant performance variations depending on the metric analyzed. Notably, RNA fragment length analysis revealed statistically significant differences, with smear and core biopsy samples exhibiting higher median lengths compared to resection specimens. However, greater fragment length did not necessarily correlate with higher specificity; resection samples, despite having shorter fragments, demonstrated significantly higher on-target unique fragment percentages compared to other groups (P < .05).
This contrast highlights that fragment length alone may not fully reflect RNA functional quality or library success. Therefore, the inclusion of additional metrics such as on-target unique fragment percentage and unique start site counts proved highly informative in distinguishing technically acceptable but biologically variable samples. These complementary metrics help interpret borderline pre-sequencing results and guide decisions regarding sample acceptance in clinical practice.
While the overall pass rates were acceptable, RNA (12.4%) showed a higher failure rate than DNA (3.5%), reflecting RNA's inherent fragility and susceptibility to preanalytical variables. Failures in RNA-based analyses were most prevalent in smear samples, likely due to factors such as low cellularity, variability in staining methods, or preparation techniques. In contrast, DNA failures appeared more evenly distributed across sample types, emphasizing its robustness compared to RNA. These observations underscore the importance of fully addressing RNA-specific challenges in cytological samples to leverage their diagnostic potential while specifically accounting for pre-analytical variables such as storage conditions or prep media.
Importantly, our findings 21 challenge the perception that smears are inherently inferior for molecular applications. In fact, imprint smears have shown excellent performance in our cohort, particularly in diagnostically difficult scenarios such as decalcified bone lesions or suboptimally preserved resections, where traditional tissue processing often leads to inadequate quality.
In smear samples, specific preanalytical factors such as staining methodologies (eg, May-Grünwald-Giemsa [MGG], Papanicolaou [PAP], Diff-Quik [DQ]) or coverslip techniques (eg, lam or film coverslip) may play a critical role in RNA quality. For instance, our recent study 22 demonstrated that, contrary to previous literature,23,24 MGG-stained smears demonstrate superior RNA quality compared to other techniques. Similarly, this study did not specifically analyze differences in nucleic acid quality between film-sealed and entellan-sealed smear slides. These findings highlight the complexity of preanalytical variables and their considerable impact on molecular testing outcomes. A deeper exploration of QC failures and their underlying causes, focusing on smear preparation techniques, tissue preservation methods, and sample-specific variables, is essential to minimize failure rates and optimize molecular diagnostics workflows in future research and clinical applications.
Our investigation into RNA integrity revealed that smear samples bypass formalin fixation and paraffin embedding and achieved cDNA fragment lengths superior to FFPE samples. DNA fragment length also serves as an indicator of DNA integrity. 25 Our analysis revealed significant differences among sample types. Smear samples exhibited superior gDNA quality, with a median fragment length of 151.5 bp, which was significantly higher than both core biopsies and resection specimens (P < .05). Interestingly, there were no significant differences between biopsies and resection specimens, yielding fragment lengths of approximately 123 bp. These findings highlight the impact of formalin fixation and paraffin embedding on DNA integrity, which smears avoid entirely.26–28
Among the sample types evaluated, biopsies emerged as the most reliable in terms of nucleic acid quality, demonstrating both high RNA and DNA integrity with low failure rates. This superiority is likely due to the consistent fixation and processing of biopsy specimens compared to the variability seen in other sample types. Another critical consideration is using uniform isolation kits and protocols across all specimen types. While the protocols are similar, the inherent differences between materials—such as smears that have never been exposed to formalin but are still processed using FFPE-specific kits29,30 —underscore the need for tailored workflows. Laboratories should focus on optimizing protocols for each sample type to ensure the highest quality nucleic acids for NGS-based analyses.
In the era of precision and personalized medicine, the handling of tissue samples significantly influences genetic testing outcomes. Standardized workflows ensure consistency, particularly for underutilized but high-quality samples like cytologic smears. Our results underscore the need for harmonized protocols across laboratories to maximize the reproducibility and reliability of molecular testing.
In conclusion, cytological and FFPE samples can be effectively utilized for NGS-based molecular applications. Pathologists and cytopathologists are responsible for preserving molecular integrity during preanalytical processing, ensuring the full diagnostic potential of these samples is realized.21,31,32
Previous Presentation
This study was presented as a platform presentation at the 32nd National Pathology Congress in Antalya, Turkey, October 25–29, 2023.
Footnotes
Acknowledgments
The authors would like to thank Arzu Baygül for her assistance with the statistical evaluation and Merve Say for her support in bioinformatic analysis.
This manuscript was prepared with the assistance of natural language processing tools driven by artificial intelligence (AI) for language refinement and formatting. These tools were used solely to enhance clarity and structure and were not involved in the generation or analysis of data.
Ethics Approval
This study was performed in accordance with the Declaration of Helsinki and was approved by the Koç University Institutional Review Board (IRB) (Approval number: 2025.019.IRB2.015).
Patient Consent Statement
Patient consent was not required due to the retrospective and anonymized nature of the data.
Author Contributions
Concept, design, and critical review were done by CAM and IK. Data collection or processing was provided by ZÇA, GÇ, CAM, İK. Analysis or interpretation was done by İK and ÇAM. Literature search and drafting were done by CAM.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Permission to Reproduce Material
Not applicable.
Clinical Trial Registration
Not applicable.
