Abstract
Virtual staining of unstained tissue for histologic assessment is the subject of burgeoning research and has been approached using various methodologies. This technology has the potential to reduce laboratory turnaround time, reduce consumption of chemicals and water, and improve occupational health and safety for laboratory personnel. In addition, the technology presents the alluring prospect of non-destructive hematoxylin and eosin histologic examination, allowing unlimited multiplexing on the same section, and improved image analysis techniques that are unimpeded by inter- and intra-laboratory stain variation. Recent advancements in this field and projections of applicability to nonclinical pharmaceutical development and discovery pathology settings warrant a brief review. Virtual staining has been applied most widely to unlabeled (unstained) tissue but has also been used in stain-to-stain transformation. Specimen input varies from conventional formalin-fixed paraffin-embedded tissue to partially processed or intact tissue. Imaging is commonly traditional brightfield or fluorescence, although other modalities are available. Depending on the imaging modality, computational methods such as deep learning neural networks are used to infer the virtual stain that is ultimately viewed as a digitized histologic image. Current barriers to applicability include qualification, histologic quality, generative artificial intelligence concerns, training material acquisition, and infrastructure.
Keywords
*This is an opinion article submitted to the Toxicologic Pathology Forum. It represents the views of the author. It does not constitute an official position of the Society of Toxicologic Pathology, British Society of Toxicological Pathology, or European Society of Toxicologic Pathology, and the views expressed might not reflect the best practices recommended by these Societies. This article should not be construed to represent the policies, positions, or opinions of their respective organizations, employers, or regulatory agencies. The Toxicologic Pathology Forum is designed to stimulate discussion of topics relevant to regulatory issues in toxicologic pathology. Readers of Toxicologic Pathology are encouraged to send their thoughts on TPF opinion articles or ideas for new discussion topics to the Editor.
Introduction
Virtual staining is generally defined as the computational reproduction of a histochemical or immunohistochemical (IHC) stain of a tissue, which often leverages deep learning neural networks and various digital microscopy techniques beyond conventional brightfield scanning. Although this technology has been applied to routine, chemically stained hematoxylin and eosin (H&E) brightfield scanned whole-slide images (WSIs) to produce virtual special histochemical stains, increased attention has been focused on the use of this technology on unlabeled (unstained) tissues and outputs that include various histochemical stains as well as advanced multiplexing. 8 The vast variation in methodologies and outputs makes this a complex field to summarize, as is the research overlap between virtual staining, virtual biopsy (which crosses over with direct-tissue imaging), and three-dimensional pathology techniques. The scope of this opinion piece will be limited to a concise overview of different virtual staining methodologies, applications relevant to toxicologic pathology, perceived benefits and disadvantages of this technology, and musings on the current challenges and barriers to implementation (Figure 1).
Virtual staining workflow example.
What Is Virtual Staining?
Tissue Form Input
Virtual staining techniques have largely been focused on unlabeled, cover-slipped slides, derived from routine formalin-fixed paraffin-embedded (FFPE) processing and standard microtomy.7,15 This technique mostly retains the conventional histology laboratory workflow while removing the chemical staining process, and therefore could be integrated with relative ease. Conventionally processed and stained H&E slides have also been used in histochemical and IHC stain-to-stain transformation methods. 18 Stain-to-stain transformation refers to the use of a conventionally stained (labeled) slide that is then digitized and computationally processed to infer a virtual stain. FFPE tissue sections are not the only tissue form input amenable to virtual staining techniques. Other specimen inputs that have achieved measures of success include unlabeled tissue on slides in various stages of processing and varying thicknesses (3 and 20 µm); in paraffin (unprocessed); deparaffinized and non-cover slipped; frozen sections (7 µm thick); fresh (hand-trimmed) tissue (2 mm thick); and fresh chemically cleared tissue (1 mm thick).7,5,17
Imaging Approaches
Convincing examples of virtual histochemical and IHC have been achieved based on unlabeled and H&E-labeled WSIs obtained from standard brightfield imaging technology used in day-to-day digital pathology workflows.7,11 Unlabeled fluorescence WSIs have similarly been used to produce high-quality virtual histochemical and IHC stains.11,15 Theoretically, fluorescence scanners may utilize multiple fluorescent channels and filters to capture the autofluorescent signature of tissues in order to render a microscopic image. In practice, most autofluorescence virtual staining techniques use DAPI (4′,6-diamidino-2-phenylindole) excitation and emission filter sets.11,15 DAPI is a blue-fluorescent DNA stain that exhibits a many-fold enhancement of fluorescence on binding to AT regions (where the replication complex is formed) of double-stranded DNA. More niche imaging modalities that have demonstrated success include ultraviolet photoacoustic microscopy, laser-scanning multiphoton microscopy, synchrotron radiation micro-computed tomography, and Fourier transform infrared spectroscopic imaging.3,5,10,16,17 Although these specialized imaging modalities appear to offer high-quality virtual stains, in addition to intriguing applications such as 3D histology and fresh tissue/intraoperative histology, they require significant investment in technology, and their applicability may be most relevant to clinical rather than nonclinical settings.
Training and Computation of Virtual Stains
GAN deep learning training
Using the tissue form input and imaging approach of choice, a virtual stain is typically developed using a deep learning technique known as a generative adversarial network (GAN) model.5-7,11,15,18 In order to train a GAN model, most virtual stain methodologies require large amounts of paired training data composed of unlabeled WSIs with pixel-level registration to corresponding labeled WSIs. The GAN uses pixel-level discrepancies between the unlabeled and labeled WSIs to discern features that relate to the desired stain (whether that be histochemical or IHC). This data is then fed into a convolutional neural network (CNN) that ultimately becomes capable of inferring the desired stain directly from unseen, unlabeled sections.
Due to the large amount of training data required in this framework and the inherent technical challenges in pixel-level registration of unlabeled and labeled WSIs, research efforts have focused on increasing GAN model complexity, and using unsupervised training modalities.3,6,7 One example is the use of serial histologic sections obtained via a 3D scanner, followed by supervised optimization with pixel-level registration of WSI data. 7 Further work has examined less-intensive, unsupervised deep learning approaches using “Unsupervised content-preserving Transformation for Optical Microscopy” (UTOM). 9 This training method purports to learn the mapping between two image domains without requiring paired training data, while avoiding distortions of the image content. If proven to be successful in real world scenarios, it would eliminate the rate limiting step for pixel-level registration and supervised approaches, enabling virtual staining and other applications of artificial intelligence in biomedical imaging to proliferate.
Other computational methods
One method that does not rely on a GAN model, but a comparatively simple artificial neural network (ANN), is Fourier transform infrared (FT-IR) spectroscopic imaging and computation. Here, spectral metrics are mapped to red-green-blue (RGB) color values on corresponding, conventionally stained H&E WSIs during training to produce stainless, computed histopathology from infrared spectroscopic imaging data. 10
Synchrotron radiation micro-computed tomography SRµCT has also been investigated for virtual histology of soft tissue samples. Using propagation-based phase contrast imaging computation, virtual stains of unlabeled paraffin-embedded tissues are produced, without damaging or physically sectioning the tissue block. 16
Advances in chemical clearing of tissues and laser-scanning multiphoton microscopy have also allowed mathematical color conversion from fluorescence to H&E, using eosin and DAPI, the latter as a nuclear dye analogue. Using this technique, chemically-cleared biopsies of 1 mm thickness are serially imaged at varied depths to produce multiple levels of a biopsy. After applying the patented process “Clearing Histology with MultiPhoton microscopy” (CHiMP), images are acquired without the need for routine histologic processing or physical sectioning. This technology allows digital evaluation within hours of tissue sampling.4,13,17
Advantages of Virtual Staining
Many of the virtual staining methods described have, at minimum, the potential to vastly improve H&E stain inter- and intra-laboratory consistency. This would aid human evaluation and likely improve generalizability of future downstream machine learning model performance. In addition, utilization of non-destructive techniques provides potentially limitless opportunities for multiplexing virtual stains (both from unstained and H&E-stained brightfield scanned tissue sections). Currently, virtual stains exist for H&E, Jones, Masson’s trichrome, Periodic-acid Schiff, fluoro-jade B, PanCK, CD45, human epidermal growth factor receptor 2, and various lymphocyte marker IHCs, all of which may be applied to the same original WSI at the click of a button.1,2,14 Highly variable among the virtual staining methodologies is the degree to which laboratory turnaround time is improved. Some techniques largely retain the conventional histology laboratory workflow but remove chemical staining processes, thus removing just a single step. However, with the pathologist’s potential need for follow-up diagnostic stains, the overall turnaround time may be reduced from days to minutes. Other approaches, such as CHiMP, would represent an overhaul of conventional histology in favor of markedly shortened processing times and serially imaged tissues. 4
Implementation of virtual staining methods could have numerous positive impacts on the current conventional histology laboratory workflows, especially in environmental and occupational safety. At present, water usage is a key environmental sustainability challenge in histology laboratories. Key considerations related to occupational safety consist of chemical exposure in conventional stains including (but not limited to) heavy metals, xanthene (of eosin staining), picric acid, fuchsin, and paraldehyde. Any reductions that could be made to reduced handling of breakage-prone glass would likely also be welcomed by laboratory personnel, as well as the occasionally clumsy pathologist.
Challenges to Virtual Stain Adoption
Scanners and Infrastructure Investment
Depending on the virtual stain tissue form input and imaging approach, significant change and infrastructure investment may be required for implementation. At present, conventional brightfield scanners may present the easiest adoption solution; however, most machines are currently optimized for high-contrast (stained) samples, which makes image acquisition of unstained deparaffinized glass slide samples challenging. Techniques that utilize fluorescence scanners (eg, Leica Aperio FL 120, Hamamatsu NanoZoomer S60, Zeiss Axioscan 7) appear to circumvent this issue; however, storage of large fluorescence image files, scanner throughput, scanner cost, and glass coverslips (rather than plastic, which will autofluoresce) are other pertinent, practical considerations. It is likely that many of the challenges relevant to brightfield and fluorescence scanners will resolve as technological advances are made. More specialized imaging techniques (such as light-sheet, Raman spectroscopy, laser multiphoton, and ultraviolet photoacoustic microscopy) often require considerable investment in equipment, and many are not yet suited to high-throughput usage. 8 As these techniques become more cost-effective, their application may widen beyond research and clinical applications.
Training Material
As described above, virtual staining methods that depend on GAN models require large quantities of training material. Acquiring this data is a laborious process and involves accessing relevant tissue sets, cutting and scanning cover-slipped unlabeled slides, then manually removing the coverslip, conventionally staining, cover-slipping, and rescanning the slide again. In the case of toxicologic pathology, producing training material that contains a balanced representation of lesions is critical, given the general preponderance of normal tissues, in addition to the typical considerations given to tissues and species. Large-scale initiatives, such as BIGPICTURE, which aims to create an extensive, freely available repository of digitized pathology slides to advance artificial intelligence tool development, may help provide the diverse whole-slide image (WSI) content required to accelerate progress in this field. 12
Generative Adversarial Network Models
As with any deep learning method, concerns exist around the reliability of the output. Relevant to GAN models are their propensity to produce hallucinations, which may manifest as false hypernucleation in baseline models. 6 These inaccuracies have a tendency to resolve as the network complexity is increased, and the dataset is diversified. Ultimately, qualification by a toxicologic pathologist with respect to the intended use is required to determine at what point of network training we consider the model sufficiently trained, and what level of prospective and retrospective testing we need to satisfy nonclinical pathology applications.
Qualification
The plethora of virtual staining modalities and a lack of described standards in virtual staining development present challenges in qualification and wider implementation. 8 The development of standards will assist in the comparison of virtual stain performance and may facilitate categorization of virtual stain products and their appropriate use-cases. Extensive retrospective and prospective testing are required, in which both histological and computational assessments are performed. When considering computational assessments that assess image similarity at the pixel-level, it is important to emphasize that these approaches do not consider histologically impactful features and their representation. It is only histologic evaluation by subject-matter experts that can assess whether the virtual stain is suited for its intended use. Therefore qualification processes for virtual staining in nonclinical toxicologic pathology would require side-by-side virtual and conventional H&E stain WSI evaluation by a toxicologic pathologist, in relation to standards that may include (but are not limited to) consistency of intraobserver findings (eg, severity grade, findings of interest, observation), nuclear and cytoplasmic detail, stroma detail, color balance, and the presence of hallucinations.6,8 Hallucinations in virtual staining describe the generation of visually convincing but biologically inaccurate histological features by computational models, which are not present in the source tissue and may mislead pathological interpretation. The author suggests that international collaboration between pharmaceutical companies, contract research organizations, and artificial intelligence providers, with input from regulators, would be necessary to define virtual staining standards relevant to nonclinical pathology.
Histologic Quality
Generally, published examples of histologic quality from various virtual staining modalities show remarkable accuracy (Figure 2). 2 Reported shortfalls include the previously mentioned GAN-associated hallucinations, which the author observed in the form of false hypernucleation of hepatic portal veins and false nuclei within intravascular pale eosinophilic material (serum) in virtual H&E stains. Other research groups have noted discrepancies in nuclei shape and stroma, as well as hallucinations where erythrocytes and lymphocytes were clustered. 6 The author noted one more persistent issue: false hyponucleation within areas of smooth muscle, particularly the tunica media of arterioles. In all instances, these issues have been resolved with increased neural complexity and broadening of the training dataset.
Brain: Virtual HE stains (Pictor Labs) on the left and the corresponding conventionally stained HE on the right. High magnification examples inset.
Technology Maturation
Qualification of nascent technology can be arduous; however, the prompt development of standards in virtual stain development should be prioritized to better understand the strengths and limitations of these various modalities. Simultaneously, improvements in GAN model development, and the use of unsupervised models could help broaden training datasets while reducing the expense and labor of producing high-quality training material. It goes without saying that the likely emergence of new, superior, deep learning techniques may render these concerns moot in this rapidly paced world of artificial intelligence. Finally, the continual improvement of scanner technology and affordability will address some of the more practical implementation concerns. Overcoming these challenges will allow the focus to shift to histological assessment and optimization, and the “business case” for virtual staining. Should we see progress in these areas, it is not unreasonable to expect virtual staining workflows to enter mainstream toxicologic pathology within the coming years.
A Note on Immunohistochemical Virtual Staining
Both published and unpublished reports show virtual IHC stains that are largely indistinguishable from their conventionally stained counterparts.1,18 Given that IHC is defined as antibody-based recognition of a specific protein or protein isoform, it is questionable whether virtual mimicry of this technique may be regarded as a sufficient substitute. However, one could argue that virtual staining based on the fluorescent signature of a structural protein representative of the target antigen is a superior technique. Regardless, utilization of virtual IHCs within the use-case species and tissue type likely holds great promise, particularly within research and multiplexing applications.
Conclusions
Virtual staining is an emerging field of pathology that encompasses a wide range of modalities and has the potential to impact nonclinical safety assessment and toxicologic pathology. Buoying this enthusiasm are prospects of improved workplace sustainability due to reduced water and chemical consumption benefiting the environment, reduced technician reagent exposure, and savings in laboratory processing times. Additional scientific advantages include non-destructive histology with multiplexing, serial non-physical sectioning of samples, and reduced inter- and intra-laboratory staining variability for improved image analysis generalizability. Challenges exist around infrastructure investment and histology workflow disruption. Potential change management of this scale, which will likely involve generative artificial intelligence techniques, intuitively calls for great caution and meticulous qualification. At present, we lack adequate training material for the current deep learning methods, standards for the development and qualification of virtual staining, and scanner technology with appropriate throughput and other logistical requirements for implementation in toxicologic pathology. Despite this, financial investment in these technologies has been strong, and virtual staining is expected to enter the clinical realm in the near future. Given the momentum behind these modalities, awareness and readiness among toxicologic pathologists, histotechnologists, and managers is prudent.
Footnotes
Acknowledgements
The author thanks collaborators at Pictor Labs and Applikate for their roles in the development of this emerging technology, as well as Jimmy Tran, Lise Bertrand, Daniel Rudmann, Daniel Patrick, Kevin Keane, and Erio Barale-Thomas for their assistance in reviewing this opinion piece. The author also gratefully acknowledges the training material contributions made by Blueprint Medicines, Eli Lilly and Company, and Johnson & Johnson Innovative Medicine to Charles River Laboratories virtual staining research efforts.
Author Contributions
The author contributed to Conceptualization; Writing – original draft; and Writing – review & editing (EEVC).
Declaration of Conflicting Interests
The author declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article. The author was engaged in research collaborations with Pictor Labs and Applikate.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
