Abstract
Digitalization of pathology workflows has undergone a rapid evolution and has been widely established in the diagnostic field but remains a challenge in the nonclinical safety context due to lack of regulatory guidance and validation experience for good laboratory practice (GLP) use. One means to demonstrate that digital slides are fit for purpose, that is, provide sufficient quality for pathologists to reach a diagnosis, is conduction of comparison studies, which have been published both, for veterinary and human diagnostic pathology, but not for toxicologic pathology. Here, we present an approach that uses study material from nonclinical safety studies and that allows for the statistical comparison of concordance rates for glass and digital slide evaluation while minimizing time and effort for the involved personnel. Using a benchmark study design, we demonstrate that evaluation of digital slides fits the purpose of nonclinical safety evaluation. These results add to reports of successful workflow validations and support the full adaptation of digital pathology in the regulatory field.
Introduction
Since the first launches of remote pathology systems in the early 1990s, the field of digital pathology has experienced a rapid evolution.11,13 Whereas this technology has readily been applied for diagnostic purposes, both, in human and in veterinary medicine,2,9,11 its implementation in the field of nonclinical drug development is still in progress, despite obvious benefits like easier and faster slide transfer, knowledge exchange, and workflow flexibility.
The main impediment to full use of digital pathology in toxicologic pathology is uncertainty about regulatory requirements, 12 especially in terms of validation of digital pathology systems and workflows. One major point of concern is, whether pathologists can make their diagnosis on the digital slide with the same level of accuracy as they do on the glass slide as the current established standard method.
In this context, readouts often used to describe and measure method comparisons are:
Concordance. Agreement between two diagnoses (i.e., light microscopy on glass-slide diagnosis and digital whole-slide image [WSI] diagnosis) by the same observer.
Equivalency. Comparison of, for example, the concordance rates of two methods to show that both methods are “similar” to a predefined degree.
Noninferiority. Comparison of, for example, the concordance rates of two methods to show that one is not significantly lower than the other (within a predefined margin)
Numerous studies comparing conventional brightfield microscopy of histopathological glass slides with the digital evaluation of WSIs for diagnostic purposes have been published, both for human and veterinary medicine.6–8,11 However, while diagnostic pathology aims at an accurate diagnosis in a small tissue sample with clinical background information, toxicologic pathology has the goal to detect very subtle changes in a large variety of tissues. Thus, although relying on the same technical laboratory procedures and a similar educational background of the investigating pathologists, validation results cannot readily be transferred from diagnostic to toxicopathology workflows.
Different approaches for such comparisons have been explored and extensively described by Bertram et al. 3 These include (1) simple modality comparisons (degree of concordance between two methods at one investigation time point each), (2) ground truth or gold standard comparisons (comparison of consistency of two methods with the gold standard; while the gold standard should be generated independently of the tested methods), and (3) benchmark studies (comparison of concordance rates of two time points of a standard modality to a new modality, necessitating a minimum of three examination time points).
Of importance, irrespective of the selected study design, the diagnostic intraobserver concordance rate between the two viewing modalities, which is influenced by several variables, has been identified to be the most important outcome measure of such studies. 3 However, besides selecting a method and study setting that results in a reliable statement on diagnostic concordance, it is important that the selected study design is a representation of the “real-life” application of the method and fits with the capacities (personnel and time) of the group in which the study is conducted. In the context of toxicologic pathology it has been proposed to differentiate between concordance testing (blinded comparison of two methods with statistical evaluation) and equivalency testing (unblinded confirmation of a finding/lesion with the digital method). 4 The goal of our concordance study was to define and use a minimal approach to formally compare conventional light microscopy examination of glass slides with examination of WSIs in the setting of nonclinical drug safety evaluation. This concordance study aims to be a proof of concept to demonstrate that digital examination of WSIs leads to a similar degree of intraobserver agreement as examination of glass slides while ensuring the same diagnostic quality level for future use in primary study evaluation and/or peer review.
Material and Methods
Selection of Cases
Cases were selected by a toxicologic pathologist who did not participate in the study. To validate digital pathology for a wide field of applications in nonclinical toxicopathology, glass slides and corresponding WSIs for this study comprised typical findings observed in studies performed at the test facility. The test cases included spontaneous background as well as induced changes in mice, rats, and dogs and were diagnosed according to INHAND (International Harmonization of Nomenclature and Diagnostic Criteria) nomenclature. All studies were conducted in an AAALAC (Association for Assessment and Accreditation of Laboratory Animal Care International) accredited laboratory in full accordance with German animal welfare legislation and Boehringer Ingelheim principles for animal use. In addition to common findings, selected “threshold” features of histological samples considered critical for recognition on the digital slides were included if available in study material. Cases were considered “critical,” if, for example, the diagnosis depends on recognition of small structures (e.g., intracellular pigment) or subtle, tinctorial differences (e.g., eosinophilic granules 12 ). Efforts were made to include a broad spectrum of organs and different animal species. An overview of included cases is shown in Supplemental Table A1.
The number of cases (n = 76) was determined with the aim to closely follow previous publications on concordance or noninferiority studies.3,11
Production of Histological Slides, WSIs, and Presentation of Cases
Slides were stained with hematoxylin and eosin (H&E); for Kidneys, a Periodic acid-Schiff (PAS)-stained slide was included. To mimic the situation of study evaluation, each slide was labeled with the animal species and treatment group (Control, Low, Mid, or High Dose) and a random two-digit number. To avoid recognition of recurring slide numbers by the participants, digital slides received a different number. The slides were scanned using a Hamamatsu Nano Zoomer S360 at 0.23 µm/pixel resolution.
For the glass slides, standard light microscopes for study evaluation were used. For the digital slides, Patholytix Preclinical Study Browser V. 2.1.0.17 (Deciphex) was employed as slide viewing program. The digital workstation was composed (as a minimum) of a laptop, a conventional monitor, and a curved monitor as well as a 3D-mouse for navigation. It fulfilled or exceeded minimum requirements as defined by the viewing software vendor at the time of study conduction.
The entire set of test cases was divided into three subsets of similar size (A, B, and C; see Table 1 and Supplemental Table A2). The content of one subset of glass slides corresponded to the content of the same subset for digital slides. Cases were presented to the observer after stratified randomized assignment to one of the three subsets, based on organ systems. This was to ensure a heterogeneity of cases and questions throughout all subsets.
Example distribution of subsets over time points.
Participants
A total of eight toxicopathologists completed the study, comprising pathologists of various experience levels. A training session of 1 hour (minimum) on the digital workstation (technical training) was provided to each participant. Participants were advised not to communicate about the study/results during the conduct of the study, to avoid any influence among each other between the examination time points. Pathologists were pseudonymized by numbering them 1 to 7 and 9 (due to failure to complete in time, pathologist 8 was excluded from study evaluation).
Study Design/Schedule
The glass slides were evaluated twice, the digital slides once, thus falling in the “Benchmark study design” category described by Bertram et al in 2021.
The time schedule for conduct of the study was set as follows: Three examination time points were defined, each followed by a washout period between 3 and 6 weeks. This was determined following recommendations from literature, where a minimum of 2 weeks is described to be favorable for the study outcome.3,11 Excessively long washout periods may result in changes in the individual pathologist’s experience and thus diagnostic criteria.
3
Defining a time frame of 3-6 weeks allowed for a flexible planning and better feasibility for all participating pathologists. During each examination time point, three slide subsets (2 glass subsets and one digital subset) were evaluated (Table 1).
One time point usually comprised three working days. No time limit for examination of a single slide subset during these three days was set.
Questionnaires and Survey
With each slide, five questions, each of which could be answered by Yes or No only were asked.
The complete questionnaires are shown in Supplemental Table A2.
To note: An additional survey about the user experience on the digital workstation was conducted after the third examination time point, comprising questions about ease of handling, ergonomic aspects and the subjective perception of the workstation and digital slide viewing (out of scope of the present publication).
Statistical Analysis
The statistical evaluations were carried out using the software package SAS Version 9.4. The primary goal was to determine the concordance rates (number of congruent diagnosis pairs divided by all diagnosis pairs evaluated) per pathologist between glass/glass and glass/digital evaluations to verify whether a digital slide evaluation is comparable or noninferior to a glass-slide evaluation.
Initially, data of all questionnaires were analyzed using simple descriptive statistics (e.g., sample sizes, proportions, estimated location, and dispersion parameters), and results are presented numerically and graphically (Supplemental Tables A3-A7 and Figure 1A-C). This was done for glass/glass and glass/digital comparisons, different time points and different levels (observer, organ system, and species). For each pathologist, concordance rates (in percent) and kappa coefficients (with values between –1 and 1) were calculated as a measure of agreement between
glass evaluation at two different time points (denoted as “glass1” and “glass2”),
Glass1 and digital evaluation,
Glass2 and digital evaluation.
Concordance rates describe the percentage of similar answers at two time points which were analyzed in three categories: (1) same answer at both time points, (2) different answers at some time points, and (3) no comparison due to missing values.
In general, kappa coefficients are interpreted as follows 8 :
< 0: insufficient agreement,
[0, 0.2]: lacking agreement,
[0.2, 0.4]: small agreement,
[0.4, 0.6]: moderate agreement,
[0.6, 0.8]: substantial agreement,
> 0.8: nearly perfect agreement
between methods.
Because the kappa coefficient additionally accounts for the possibility of an agreement occurring by chance, it is generally more robust than the simple concordance rate.
In the present study, the sequence of assessments was determined to be at random and the washout period between two analysis time points was assumed to be sufficiently large, such that, both evaluation methods were assumed to be independent from a chronological perspective (different sequences between examination methods).
Moreover, to calculate concordance rates and kappa coefficients to quantify the level of agreement between both evaluation methods, questions are assumed to be independent (one question does not have an influence on the response to another question).
On the other hand, comparisons of different methods depend on the pathologist, such that, concordance rates and kappa coefficients, respectively, depend on the observer level.
Different inferential statistical methods were applied to verify significance, equivalence or non-inferiority between the above specified concordance rates or kappa coefficients, respectively.
Equivalence tests (two one-sided t tests, TOST) are used to verify the similarity of two paired samples, for example, containing the observed concordance rates or kappa coefficients, respectively, involving predefined equivalence limits (5%). More precisely, differences of concordance rates (kappa coefficients) are compared with an interval equal to [–5%, 5%] ([–0.05, 0.05]).
The noninferiority analysis was examined using an artificial gold standard (defined as consistent results at both glass-slide examination time points). In doing so, we set the noninferiority margin to 90% for the concordance rates and to 0.8 for kappa coefficients, reflecting the involvement of agreement by chance.
In addition, a generalized linear mixed model was used to analyze proportions of concordant rates and kappa coefficients, respectively, to detect statistical differences between glass/glass and glass/digital. In general, this can be done using a two-sided binominal model or with an approximate normal distribution which would be consistent to the normality assumption in the TOST-approach in the present applications. More precisely, degrees of freedom for the latter mentioned comparisons were calculated using a method according to Kenward and Roger. 7 Observers were modeled as a random factor and repeated measures were involved for the different comparisons (glass/glass, glass1/digital, and glass2/digital). In addition, heteroscedasticity was considered for both inferential comparisons with glass/glass.
Results
Descriptive Statistics
The concordance rates between glass/glass and glass/digital evaluations on observer level revealed slightly higher concordance rates between the two glass examination time points than between the second glass examination and digital examination.
The results are summarized using simple descriptive statistics (arithmetic mean, standard deviation) and are presented in Table 2. Individual concordance rates, ranging from 76.41% to 97.18%, are visualized in Figure 1 and are given in Supplemental Table A3; individual kappa coefficients are listed in Table A3. 8 Further descriptive results on species and organ level are shown in Supplemental Tables A10-A14 and A17-A23.
Descriptive statistics of concordance rates on observer level.
Arithmetic mean and standard deviation of concordance rates in percent.
Comparison of concordance rates; observer level. (A) The similarity of concordance rates of the two investigation time points on glass slides versus the first examination time point on glass and the investigation of digital slides. (B) The similarity of concordance rates of the two investigation time points on glass slides versus the second examination time point on glass and the investigation of digital slides.
In addition, identical glass evaluations were treated as an “artificial gold standard” and the concordance rates/kappa coefficients between this “gold standard” and the digital evaluations were calculated. In general, these comparisons led to higher than the original concordance rates/kappa coefficients (Supplemental Tables A15 and A16).
Inferential Statistics
Tests on equivalence between glass and digital assessments using concordance rates and kappa coefficients, respectively, are shown in Table 3. Briefly, it was demonstrated that concordance rates of both glass time points and glass versus digital examination are equivalent to the 5% margin. A similar result can be obtained for kappa coefficients using lass/glass versus glass time point 2/digital.
Equivalence tests for concordance rates (in percent) and kappa coefficients: difference (glass1 vs glass2)—(glass vs digital).
TP = time point of examination; N = number of examinations; mean = mean of paired differences; SD = standard deviation; the overall P value of the TOST: the largest P value of the two one-sided tests; LCL = lower 90% confidence limit; UCL = upper 90% confidence limit; LB = lower bound; UB = upper bound; Equi. = equivalent.
To further investigate the non-equivalent result (kappa coefficients of both glass time points, and glass1 and digital), one-sided t-Tests were used to compare the kappa coefficients of glass1 versus glass2 and glass1 versus digital. As shown in Supplemental Table A5, the paired difference of kappa coefficients (glass1 vs glass2)—(glass1 vs digital) had a mean of 0.0185, which was not significantly larger than zero (P value of .2957).
The results of the generalized linear mixed model to compare glass and digital assessments based on concordance rates and kappa coefficients, respectively, are given in the Supplemental Tables A6 and A7. This method shows no statistically significant differences between the kappa coefficients of glass and digital evaluations. Moreover, the comparison of concordance rates between glass/glass and glass1/digital does not yield a significant difference; only the comparison between glass/glass and glass2/digital leads to a significant outcome to the 5% level (P value = .0454). Here it becomes apparent, that the difference between means is increased whilst the corresponding standard deviation is decreased compared with glass/glass versus glass1/digital (Table 2).
A noninferiority test of concordance rates (kappa coefficients) based on a data set reflecting an “artificial gold standard” compared with a limit of 90% (0.80) is presented in Supplemental Table (A9) stating statistically significant results. More precisely, 90% (0.80) was chosen as the noninferiority limit since our gold standard approach does not consider concordant glass/digital examinations if glass/glass observations are discordant.
Survey (out of scope)
The user feedback was overall positive or very positive. The use of the viewing platform (login, slide selection, slide sorting, and viewing) was rated easy or very easy by all users. Single users reported that the use of some 3D-mouse functionalities and the slide viewing on the large screen would require some adaptation; however, users were confident that insecurities and subjectively perceived technical difficulties would improve significantly with more training and over time.
Discussion
Study Design
This study aimed at an approach that is feasible to conduct in an operational pathology group (i.e., with a minimal effort that does not significantly disrupt routine work), while delivering a solid result on the equivalency of methods of microscopic examination (glass vs digital). The chosen benchmark study design approach best combines these properties while providing results demonstrating the validity of digital slide examination based on different statistical approaches.
The technical specifications were defined based on vendor recommendations and standards 12 at the time of the study planning. It can be assumed that results generated with the described technical setup will remain stable or can even improve as technology is evolving. Thus, this study aimed at a proof of concept that can serve as a reference to prove the validity of the method, provided the technical standard is maintained.
It was shown that the intraobserver differences between two examination time points on glass and one examination point of WSI allowed for a reliable evaluation of the concordance between two time points (glass vs glass slides and glass vs digital slides) while saving time for the participants.
Observer Level
The concordance rates and kappa coefficients of both glass examinations were slightly higher than the corresponding glass/digital observations. However, the difference between both concordance rates (kappa coefficients) were determined through an equivalence test (two-sided t test) involving predefined 5% margins (0.05 margins for kappa coefficients).
Concordance rates were shown to be equivalent. Kappa coefficients were demonstrated to be equivalent comparing digital to the second, but not to the first glass time point.
The differences between both glass time points and the first glass time versus digital were minimal and not statistically significant to the 5% level.
These results were supported by the outcome of the generalized linear mixed model. This method showed no statistically significant differences between the kappa coefficients of glass and digital evaluations. Only the comparison between concordance rates of glass/glass and glass2/digital yielded a significant difference, although with a P value of.0454. This result is a consequence of an increased effect, and also of a substantially decreased standard deviation.
Twice the same result on glass was considered the “artificial gold standard.” Using this approach, we aimed at reducing a potential bias due to a high interobserver variability for certain cases (i.e., difficult or unclear cases). When comparing the digital results with this “artificial gold standard,” concordance within a 10% interval (i.e., noninferiority to a limit of 90%) was demonstrated. It must be noted, that in this case, the average concordance rate of glass/digital was even higher than the original setting (comparison of time points among both methods). For kappa coefficients, noninferiority to a limit of 0.80 could be demonstrated statistically significant. This corresponds to nearly perfect agreement in the interpretation of kappa coefficients. This may indicate that most discordant results have been caused by a generally more difficult diagnosis and thus overall higher interobserver variability. In addition to that, the minimal differences may be caused by the fact that the pathologists were more experienced in the use of glass slides compared with WSIs.
Similarly, the concordance rates were not substantially different for investigated organ system or species (Supplemental Tables A10-A14).
Taken together, it can be concluded that the number of concordant results on observer level is comparable between glass and digital, and glass and glass-slide evaluations.
Advantages and Limitation of the Study Design
In addition to a study design that was fit for purpose for an operational pathology group, the setup of this study particularly allowed not only for mere participation of all pathologists but also to build trust and improve performance on the digital method as well as for method evaluation irrespective of the individual pathologist’s skills and experience level.
Consequently, this resulted in limitations with regards to the statistical design: The classical gold standard method is described to be the most accurate.,
3
but is not always feasible due to constraints in time and resources: A gold standard could have been generated by a consensus diagnosis of a minimum of two experienced toxicologic pathologists on a glass slide. However, these two pathologists would then be biased and could not participate in the study or only after a substantial time period (washout), thus significantly increasing the overall study time. Alternatively, these two pathologists would be excluded from participation, resulting in a critically low number of observers for statistical evaluation. In addition, the individual performance of each participating pathologist would have had an influence on the results, leading to an evaluation of both, the method, and the diagnostic skills of a pathologist.
Potential influence of individual diagnostic skills was further attenuated by the questionnaire design with Yes/No questions only. Short answer questions can be more variable and more easily influenced by the individual experience level.
This study was conducted on H&E-stained and PAS-stained slides from one histopathological laboratory. It proves that the method is suitable to provide the pathologist with a digital image that allows for histopathological assessment. Before investigating other histochemical and immunohistochemical stains, a thorough comparison (i.e., judgment of equivalency) between the digital and the glass slide would be advisable although not require a full, blinded concordance study with statistical evaluation.
With respect to the spectrum of species and organs, based on practical considerations, the study was conducted on the spectrum of available tissues from ongoing non-GLP studies and did not cover the entirety of potential species and tissues that can occur in nonclinical studies. Literature data show concordance, comparability or noninferiority of glass slide and WSI-investigation in a broad range of species, organ systems, and disease entities. 3 Thus, it is trusted that expansion to other tissues or species will not have an impact on the concordance of glass/glass and glass/digital evaluations.
Considerations for Digital Pathology Implementation
One factor to consider when planning the implementation of a digital pathology workflow is the selection of building block for the image chain (scanner, servers, viewing hardware, and software etc.). Based on the image chain paradigm, 12 it is trusted that exchange of a single image chain building block will not influence the overall image quality for the pathologist; however, such changes need to be conducted in the framework of a change control process.
Some slide scanners have been qualified as medical devices and concordance studies specifically for those devices have been published 10 and viewing software platforms have achieved GLP certification. In addition, the first publication on successful validation of dedicated viewing and slide management software programs for primary slide evaluation and/or GLP peer review can help increase trust in the system as a whole. 6
Besides the image chain building blocks, workflow implications and company-specific use cases need to be considered when defining the need for and extent of any kind of comparison study for verification and/or GLP validation. Considering the dynamics in this topic and the heterogeneity of companies’ workflows, there cannot be one general recommendation on the one suitable method to prove concordance, equivalency, or noninferiority. 5 It should be emphasized that the comparison method, spectrum of test cases and means of documentation depend on the intended application and must be chosen on an individual basis. For instance, for organizations that will only conduct contemporaneous peer review on the digital slide, it will very likely be sufficient to confirm that the peer review pathologist is able to identify the same lesion on a WSI as the study pathologist on a glass slide. 4
Formal method comparisons, irrespective of the study design, can be integrated in the process of workflow validation (e.g., for GLP use) by documenting user requirements fulfillment, image chain reliability, and user training. Of importance, to conduct such a study is also a means to train the participants in using the digital viewing system and to build trust as demonstrated with the user survey accompanying this study.
Conclusion
We present a minimal approach comparison study design that has demonstrated that digital histopathology evaluation is fit for the purpose of primary toxicopathologic assessment by showing that the number of concordant results is comparable between glass and digital, and glass and glass-slide evaluations. This study complements published studies that have demonstrated concordance/noninferiority of digital pathology as a method per se. It thus serves both, as a general proof of concept and as a basis for confident use of digital microscopy in the highly regulated preclinical safety field, ultimately leading to general regulatory acceptance of digital pathology evaluation.
Supplemental Material
sj-docx-10-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-10-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-11-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-11-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-12-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-12-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-13-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-13-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-14-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-14-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-15-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-15-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-16-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-16-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-17-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-17-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-18-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-18-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-19-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-19-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-2-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-2-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-20-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-20-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-21-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-21-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-22-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-22-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-23-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-23-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-24-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-24-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-3-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-3-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-4-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-4-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-5-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-5-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-6-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-6-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-7-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-7-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-8-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-8-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-docx-9-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-docx-9-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Supplemental Material
sj-tif-1-tpx-10.1177_01926233241255125 – Supplemental material for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology
Supplemental material, sj-tif-1-tpx-10.1177_01926233241255125 for A Minimal Approach to Demonstrate Concordance of Digital and Conventional Microscopy in Toxicologic Pathology by Charlotte Lempp, Stefanie Arms, Christof Albert Bertram, Robert Klopfleisch, Bernd-Wolfgang Igl, Leonie Hezler, Thomas Nolte and Gabriele Pohlmeyer-Esch in Toxicologic Pathology
Footnotes
Acknowledgements
The authors would like to thank all pathologists and technical staff of the pathology group at Nonclinical Drug Safety, Boehringer Ingelheim Pharma GmbH&Co KG for their contribution to the study, as well as Monika Brüning for data analysis support.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: CL, SA, B-WI, LH, TN, and GP-E are employed by Boehringer Ingelheim Pharma GmbH & Co. KG. The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
