Bridging Global Attention and Local Hierarchies: A Robust Hybrid Ensemble Framework With Multi-Perspective Explainability for Automated HER2-IHC Scoring

Abstract

Introduction

Determining the HER2 status accurately is a critical determinant in breast cancer treatment planning. Manual scoring remains highly susceptible to inter- and intra-observer variability, particularly in diagnostically ambiguous cases (HER2 1+/2+). Furthermore, contemporary deep learning models frequently lack the robustness and interpretability required for safe clinical integration.

Methods

We propose a hybrid ensemble framework that synergizes EVA-02-Large, Vision Transformer-Base, and ConvNeXt-V2-Nano architectures via an adaptive late-fusion mechanism. The model was trained on the expert-annotated HER2-IHC-40x dataset ( $N = 10, 997$ patches) and rigorously evaluated across ten independent trials. The experimental protocol incorporated comprehensive perturbation analysis across 20+ image corruption types and multi-perspective Explainable AI (XAI) assessment using Grad-CAM, Grad-CAM++, and Layer-CAM to validate alignment with pathologist-recognized morphological features.

Results

The fused ensemble achieved state-of-the-art weighted accuracy ( $98.05 % \pm 0.10 %$ ) and Balanced Accuracy ( $98.25 % \pm 0.11 %$ ), outperforming the strongest single backbone by 0.45%. Misclassification rates for equivocal classes decreased by >18%. The framework demonstrated high resilience to structural perturbations ( $m C E < 0.05$ ) while exhibiting expected sensitivity to extreme photometric variations. XAI analysis confirmed that model attention consistently prioritizes clinically relevant membrane staining patterns, mirroring expert diagnostic reasoning.

Conclusion

This study establishes a highly accurate and interpretable pipeline for automated HER2-IHC scoring, demonstrating that hybrid transformer-CNN ensembles can effectively resolve diagnostically challenging cases. By combining superior predictive performance, structural robustness, and transparent decision-making, the proposed framework offers a highly promising foundational framework that, following prospective multi-reader clinical validation, could serve as a robust decision-support tool to standardize HER2 assessment, reduce inter-observer variability, and accelerate precision oncology workflows.

Keywords

HER2-IHC scoring hybrid deep learning vision transformers ConvNeXt ensemble learning explainable AI (XAI)digital pathology breast cancer robustness analysis

1. Introduction

Breast cancer (BC) is still the number one malignancy diagnosed worldwide. It accounts for nearly 12% of all newly diagnosed cancer cases each year.¹ Among its molecular subtypes, the human epidermal growth factor receptor 2 (HER2)-positive BC subtype, which is defined by ERBB2 gene amplification or protein overexpression, accounts for about $15 % : 20 %$ of the invasive cases and is linked to aggressive tumor biology, high metastatic potential, and the worst prognosis before targeted therapies.^2,3 The introduction of HER2-targeted therapeutics, especially trastuzumab and pertuzumab, has led to a remarkable improvement in patient survival, but this benefit is largely dependent on the accurate determination of the HER2 status.^4,5 Hence, a precise measurement of HER2 expression by immunohistochemistry (IHC) or in situ hybridization (ISH) remains a vital diagnostic requirement in modern oncology.⁶

Current clinical guidelines require HER2 scoring to be based on standardized semi-quantitative criteria: scores of 0 or 1+ are negative, 2+ is an equivocal expression that needs to be confirmed by reflex ISH testing, and 3+ is positive.⁷ In spite of these procedures, manual IHC reading is still vulnerable to inter- and intra-observer variations, especially in the borderline cases (1+ versus 2+), where the disagreement rate may be more than 20%.^8,9 Such uncertainty in diagnosis can result in wrong treatment being given; patients who are deprived of HER2 targeted therapy may be those who would benefit from it and thus are denied effective treatment, while HER2-negative patients may be unnecessarily exposed to toxicity and cost. This highlights the need for a novel and reliable tool that can be used not only for verification but also for routine pathology diagnosis.

The conjunction of digital pathology and artificial intelligence has driven profound deep innovations in computational histopathology. Deep learning, especially convolutional neural networks (CNNs), has proven to be very effective in identifying detailed morphological and textural features from hematoxylin and eosin (H&E)-stained whole-slide images (WSIs) for a variety of cancer-related purposes such as detecting tumors, determining their subtypes, and prognostic stratification.^10-12 In continuation of this line of work, researchers have started to utilize the potentials of deep learning for HER2-specific research: Yang et al.¹³ used multimodal deep learning by jointly processing H&E images and clinical data for predicting the risk of recurrence in HER2-positive patients; Farahmand et al.¹⁴ used annotated tumor regions of interest to conduct slide-level HER2 classification and achieved an area under the curve (AUC) of 0.90; while Rashid et al.¹⁵ proposed a feature selection strategy based on a metaheuristic search combined with transfer learning for HER2 scoring that reached a high level of accuracy. In addition to these studies, other researchers have also looked at different modalities. For example, the application of ultrasound images for HER2 status prediction through the use of DenseNet models was explored,¹⁶ and a novel hybrid CNN-transformer architecture supported by explainable AI (XAI) methods such as Grad-CAM++ was implemented to enhance model interpretability.¹⁷

Despite these hopeful advances, there are still several methodological and practical limitations that the field of study needs to address. To begin with, many current methods depend on weakly supervised learning paradigms that use slide-level labels only, without pathologist-annotated precise regions of interest, which may lead to dilution of model sensitivity to the features that are diagnostically salient.¹⁴ Secondly, CNNs with a single-architecture might not fully capture the diverse visual signatures that are typical of HER2-IHC staining (from slight membrane intensity gradients to complicated tissue architectural patterns), thus limiting the model’s performance across different institutions and staining protocols.^15,17 Thirdly, the lack of large-scale, public, and expertly curated HER2-IHC datasets hinders the development of robust models, their benchmarking, and cross-validation.¹⁶ Last but not least, the “black-box” characteristic of many deep learning systems is a barrier to clinical trust and adoption; therefore, there is a need for explainability frameworks that are incorporated and that map algorithmic decisions to pathological reasoning.¹⁷ Manual HER2-IHC scoring exhibits inter-observer disagreement rates exceeding 20% for equivocal cases (1+ vs. 2+).^8,9 Our framework mitigates this variability through deterministic feature extraction, calibrated confidence scoring, and multi-perspective XAI alignment with pathologist-validated criteria. Technically, the model reduced equivocal class misclassification by $\sim 18 %$ relative to single-backbone baselines. Clinically, however, direct quantification of variability reduction requires prospective multi-reader studies comparing pathologist agreement with and without AI assistance, which we have explicitly designated as a critical next-phase validation objective.

To fill the gaps we have mentioned above, we introduce a new hybrid deep learning framework built from the ground up to be automated, interpretable, and of very high fidelity for HER2-IHC scoring. By utilizing the newly released HER2-IHC-40x dataset (a very detailed, expert-annotated collection of more than 10,000 tumor-enriched patches classified into four HER2 expression levels¹⁸), our design moves away from the typical single-backbone methods by combining three complementary modules: (1) a triplet-feature extraction backbone that blends at synergistic levels the global semantic modeling of EVA-02-Large (strengthened with rotary position embeddings and SwiGLU activations), the pure attention-based context capture of Vision Transformer (ViT)-Base, and the local spatial hierarchy modeling of ConvNeXt-V2-Nano (upgraded with Global Response Normalization); (2) an adaptive late-fusion mechanism that links the high-level feature vectors from each of the three backbones in a single 2432-dimensional representation, this representation then going through a multi-layer perceptron (MLP) head for dynamic usage of class-discriminative features; and (3) a wholly XAI system which takes advantage of three attribution methods (Grad-CAM, Grad-CAM++, and Layer-CAM) to supply multi-perspective saliency maps, thus, model’s attention is rigorously aligned with the human expert-defined areas of interest like membrane staining intensity and nuclear morphology. In sum, our main achievements are.

• Introducing a hybrid ensemble architecture that not only attains state-of-the-art classification performance (Weighted Accuracy: $98.05 %$ ) but also, when tackling diagnostically ambiguous borderline cases (HER2 1+ vs. 2+), it exhibits higher proficiency compared to standalone CNN or Transformer baselines.

• A thorough statistical analysis of the results from ten independent runs, a complete perturbation study (testing the models robustness to 20+ types of image corruptions), along with calibration metrics, resulting in a direct comparison of our framework with baseline architectures under the same experimental settings.

• An interpretability investigation from multiple perspectives illuminating that the models classification criteria are derived from the morphologically and clinically relevant features in histopathology rather than being influenced by artifacts, thus enabling oncological applications to benefit from model transparency and clinical trust for the deployment stage.

This work pushes the boundaries of AI-assisted HER2 assessment by connecting innovation in methods with clinical utility. We have set up a reproducible, interpretable, and high-performance framework based on expert-validated data, which can reduce the inter-observer diagnostic variability, speed up the therapeutic decision-making, and finally, provide more personalized and effective care for patients with HER2-positive breast cancer.

2. Materials and Methods

This study conforms to the TRIPOD + AI statement for reporting clinical prediction models that utilize machine learning methods.¹⁹ The reporting of this study conforms to the TRIPOD + AI statement.¹⁹ This work describes the development and internal validation of a hybrid ensemble prediction model for automated HER2-IHC scoring in breast cancer histopathology.

This section describes the methodological framework developed for the automatic assessment of HER2-IHC status. The HER2-IHC-40x dataset is used to create a powerful pipeline that combines sophisticated preprocessing techniques with a hybrid ensemble architecture. As shown in Figure 1, the system consists of three different backbone networks for capturing multi-scale morphological features, which are then combined by an adaptive fusion mechanism. Besides that, to tackle the issue of deep learning models being “black boxes”, a multi-pronged XAI module has been added, which helps to verify the model’s decisions with the histopathological patterns that doctors recognize. The rest of the sections are dedicated to a thorough mathematical and structural explanation of each aspect, including data curation, model architecture, optimization strategy, and interpretability techniques.

Figure 1.

Architectural overview of the proposed hybrid deep learning framework for HER2-IHC scoring where parallel backbone networks (EVA-02-Large, Vision Transformer (ViT)-Base, and ConvNeXt-V2-Nano) process input histopathology patches through a pipeline, and they extract complementary global and local features. The feature vectors are concatenated and passed through an MLP head for classification. The training regimen uses Exponential Moving Average (EMA) to stabilize the weights. After inference, an ensemble of explainability methods (including Grad-CAM, Grad-CAM++, and Layer-CAM) creates saliency maps to emphasize the most diagnostic regions

2.1. Materials

This work employs the HER2-IHC-40x dataset,¹⁸ publicly available at https://zenodo.org/records/15179608, a high-resolution histopathology dataset specifically curated for HER2 immunohistochemistry (IHC) scoring of breast cancer. The dataset (see Figure 2) comprises $N = 10, 997$ tumor-enriched image patches ( $224 \times 224$ pixels) extracted from 107 whole-slide images (WSIs) acquired at 40x magnification. Patches were annotated following ASCO/CAP guidelines⁷ into one of four HER2 expression levels: $Y = {0, 1 +, 2 +, 3 +}$ . Class distribution: 0 ( $n = 3789$ ), 1+ ( $n = 2689$ ), 2+ ( $n = 1131$ ), 3+ ( $n = 3388$ ). The original dataset¹⁸ curators implemented a patient-level stratified partitioning strategy to prevent data leakage: all patches derived from the same WSI were assigned to a single partition, preserving class distribution across training (80%, $n = 8897$ ) and testing (20%, $n = 2200$ ) sets. All patches ( $x_{i} \in R^{H \times W \times C}$ ) underwent standardized preprocessing: (1) intensity normalization using ImageNet statistics; (2) artifact removal via Otsu thresholding to exclude non-tissue regions; (3) color augmentation (brightness/contrast/saturation jitter $\pm 0.2$ ) during training to simulate staining variability; and (4) MixUp regularization ( $α = 0.2$ ) to encourage linear behavior between classes. Test set patches received only normalization and artifact removal to ensure unbiased evaluation. To ensure annotation consistency and high inter-observer agreement, all cases were independently reviewed by a board-certified expert pathologist.

Figure 2.

Twelve samples from the utilized dataset: HER2-IHC-40x dataset¹⁸ where each column reflects a specific HER2 score ( $Y = {0, 1 +, 2 +, 3 +}$ )

2.2. Mathematical Preprocessing Pipeline

Let ${X = {x_{i} \in R^{H \times W \times C}}}_{i = 1}^{N}$ denote the set of input image patches. Prior to model ingestion, all images undergo standardized preprocessing to ensure consistency, robustness, and compatibility with transfer learning paradigms. First, intensity normalization is applied using ImageNet statistics to utilize pretrained feature representations while maintaining framework interoperability. Each pixel value $x_{i}^{(c)}$ in channel $c \in {R, G, B}$ is normalized using ImageNet statistics as defined in Equation 1.

{\hat{x}}_{i}^{(c)} = \frac{x_{i}^{(c)} - μ_{c}}{σ_{c}} where \to {\begin{cases} μ & = [0.485, 0.456, 0.406] \\ σ & = [0.229, 0.224, 0.225] \end{cases}

(1)

Although histopathology images exhibit distinct color distributions relative to natural images, prior studies have demonstrated that ImageNet-normalized features remain effective for medical image analysis when combined with domain-specific fine-tuning. To further mitigate domain shift and enhance generalization to staining variability encountered in multi-institutional workflows, we incorporate a stochastic augmentation pipeline $T$ . For every given input $x_{i}$ , the augmented version ${\tilde{x}}_{i}$ is created as ${\tilde{x}}_{i} = T (x_{i}) = M_{MixUp} \circ C_{Jitter} \circ R_{θ} \circ F_{flip} (x_{i})$ , which includes: (i) $F_{flip}$ : A random horizontal/vertical flip with a chance of $p = 0.5$ ; (ii) $R_{θ}$ : A random rotation angle $θ \sim U (- 15^{\circ}, + 15^{\circ})$ ; (iii) $C_{Jitter}$ : Color jittering with perturbations $δ \sim U (- 0.2, + 0.2)$ ; and (iv) $M_{MixUp}$ : MixUp regularization²⁰ defined in Equation 2.

\begin{array}{l} {\tilde{x}}_{i} = λ \times x_{i} + (1 - λ) \times x_{j} \\ {\tilde{y}}_{i} = λ \times y_{i} + (1 - λ) \times y_{j} \end{array} where \to {\begin{cases} λ & \approx Beta (α, α) \\ α & = 0.2 \end{cases}

(2)

This augmentation strategy encourages linear behavior between classes and improves robustness to photometric variations; thereby supporting reliable performance across diverse histopathological imaging conditions. Test and validation set patches receive only intensity normalization and artifact removal to ensure unbiased evaluation.

2.3. Model Architecture and Mathematical Formulation

We selected three specific architectures to represent complementary inductive biases, moving from global attention mechanisms to hierarchical convolutional processing. EVA-02-Large was chosen for its state-of-the-art global semantic modeling via SwiGLU activations and Rotary Position Embeddings (ROPE). ViT-Base serves as a pure attention baseline devoid of convolutional priors, ensuring unbiased global context capture. Finally, ConvNeXt-V2-Nano was selected to capture local spatial hierarchies using modernized convolutional designs upgraded with Global Response Normalization (GRN). Other contemporary architectures were either subsumed by these choices or found in preliminary experiments to offer less complementary inductive biases for the multi-scale nature of HER2-IHC staining. Subsections below present the detailed mathematical formulations and structural components of each architecture.

2.3.1. EVA-02-Large Architecture

The EVA-02-Large model²¹ is based on a ViT framework but with some upgraded features. For an input image $X \in R^{448 \times 448 \times 3}$ , the patch embedding layer divides $X$ into $N = {(448 / 14)}^{2} = 1024$ non-overlapping patches, each of size $14 \times 14$ . The first token embeddings are calculated as $z_{0}^{(p)} = E \times Patch {(X)}^{(p)} + e_{pos}^{(p)}$ , where $p = 1, \dots, N$ , $E \in R^{d \times (14^{2} \times 3)}$ represents the patch embedding matrix, $d = 1024$ is the hidden dimension, and $e_{pos}$ refers to Rotary Position Embeddings (ROPE). Also, the transformer block is activated with the SwiGLU non-linearity function, which is given as $SwiGLU (x) = (W_{1} x + b_{1}) \otimes σ (W_{2} x + b_{2})$ , where $σ (z) = \frac{z}{1 + e^{- z}}$ . The multi-head self-attention mechanism blends ROPE through $Attention (Q, K, V) = SoftMax (\frac{(Q R_{Q}) {(K R_{K})}^{⊤}}{\sqrt{d_{k}}}) V$ , with $R_{Q}, R_{K}$ being rotation matrices that carry relative positional information. The final representations are obtained via mean pooling over all patch tokens: $\frac{1}{N} \times \sum_{p = 1}^{N} z_{L}^{(p)} \in R^{d}$ .

2.3.2. Vision Transformer-Base Architecture

The ViT-Base model as described in 22 initially converts input image into tokens through patch tokenization of size $16 \times 16$ , resulting in $N = 196$ tokens. Then, the transformer encoder utilizes $L = 12$ layers of multi-head self-attention according to Equation 3, MSA standing for multi-head self-attention (involving Equation 4). The class token $[CLS]$ at the position 0 even after all layers gives the final representation: $h_{ViT} = z_{L}^{[CLS]} \in R^{768}$ .

z_{l} = MSA (LN (z_{l - 1})) + z_{l - 1} a n d z_{l}^{'} = MLP (LN (z_{l})) + z_{l}

(3)

MSA (Z) = Concat (h_{1}, \dots, h_{h}) W^{O} where \to h_{i} = Attention (Z W_{i}^{Q}, Z W_{i}^{K}, Z W_{i}^{V})

(4)

2.3.3. ConvNeXt-V2-Nano Architecture

ConvNeXt-V2-Nano²³ updates convolutional networks by adding the design features of transformers. For input $X$ , the forward pass through each ConvNeXt block is given by Equation 5, where: (i) $DWConv$ : Depthwise convolution with kernel size $7 \times 7$ ; (ii) ${LayerNorm}_{c}$ : Channel-wise layer normalization; (iii) ${LayerNorm}_{d}$ : Depth-wise (per-token) layer normalization; (iv) $γ \in R^{C}$ : Learnable scaling vector (GRN component); and (v) $W_{1} \in R^{4 C \times C}, W_{2} \in R^{C \times 4 C}$ : Pointwise convolution weights. The final global response normalization (GRN), $GRN (f) = f \otimes \frac{γ \times ∥ f ∥_{2}}{\sqrt{\frac{1}{C} \times \sum_{c = 1}^{C} ∥ f_{c} ∥_{2}^{2}} + ϵ} + β$ ; promotes feature diversity. The architecture produces a feature vector, $h_{ConvNeXt} = GlobalAvgPool (F_{final}) \in R^{640}$ .

y = x + γ \times {LayerNorm}_{d} (W_{2} \times GELU (W_{1} \times {LayerNorm}_{c} (DWConv (x))))

(5)

2.3.4. Feature Fusion and Prediction Head

Let $h_{EVA} \in R^{1024}$ , $h_{ViT} \in R^{768}$ , and $h_{ConvNeXt} \in R^{640}$ denote the terminal feature vectors from the EVA-02, ViT, and ConvNeXt backbones, respectively. The fused representation is formulated as $h_{fused} \in R^{2432}$ by: $[h_{EVA} \oplus h_{ViT} \oplus h_{ConvNeXt}]$ , where $\oplus$ denotes the concatenation operator. The fused vector is then fed into a MLP prediction head, $F_{θ}$ , to obtain the class logits $z \in R^{K}$ . The MLP employs GELU activation functions and layer normalization (LN) to stabilize training, as defined in Equation 6 where $W_{1} \in R^{H \times 2432}$ and $W_{2} \in R^{K \times H}$ are the learnable weights, $b_{1}, b_{2}$ are bias vectors, and $H$ is the projection layer hidden dimension.

z = W_{2} \times GELU (LN (W_{1} \times h_{fused} + b_{1})) + b_{2}

(6)

This concise formulation allows the MLP to dynamically learn the non-linear weighting of class-discriminative features. The final class prediction $\hat{y}$ is computed by passing the logits through the SoftMax function with $\arg \max_{k} (\frac{\exp (z_{k})}{\sum_{j = 1}^{K} \exp (z_{j})})$ . Such a combination method guarantees that the ultimate decision boundary is derived from not only global contextual attention but also local spatial hierarchies, thus increasing the model’s stability to different input distributions.

2.4. Training Protocol and Optimization Mathematics

Loss Function and Regularization: We train the model using the categorical cross-entropy loss with label smoothing which helps to reduce overconfidence and thus, increases generalization. The smoothed target distribution $q_{k}$ , which is a function of the smoothing factor $ϵ$ , number of classes $K$ , and ground truth label $y$ , is shown in Equation 7. This change means that the model is not allowed to give exactly 1 probability to the correct class and is therefore a way of regularizing the logits.

L_{CE} = - \sum_{k = 1}^{K} q_{k} \times \log (p_{k}) where \to q_{k} = {\begin{cases} 1 - ϵ + \frac{ϵ}{K} & if k = y \\ \frac{ϵ}{K} & otherwise \end{cases}

(7)

Optimizer and Weight Decay: For optimization, we chose AdamW²⁴ which keeps weight decay separate from the gradient-based update rule. At the $t^{t h}$ step, the parameter update is from Equation 8, where $η_{t}$ is the step size and $λ = 0.01$ the weight decay factor. The corrected first and second moment estimates, ${\hat{m}}_{t}$ and ${\hat{v}}_{t}$ , correspond to Equation 8 in which $g_{t} = \nabla_{θ} L_{t}$ is the gradient, $β_{1} = 0.9$ , $β_{2} = 0.999$ , and $ϵ_{opt} = 10^{- 8}$ for numerical stability.

\begin{array}{l} m_{t} = β_{1} \times m_{t - 1} + (1 - β_{1}) \times g_{t} \\ v_{t} = β_{2} \times v_{t - 1} + (1 - β_{2}) \times g_{t}^{2} \\ θ_{t + 1} = θ_{t} - η_{t} \times (\frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ϵ_{opt}} + λ \times θ_{t}) \end{array} where \to {\begin{cases} {\hat{m}}_{t} & = \frac{m_{t}}{1 - β_{1}^{t}} \\ {\hat{v}}_{t} & = \frac{v_{t}}{1 - β_{2}^{t}} \end{cases}

(8)

Learning Rate Scheduling: The learning rate is controlled by a cosine annealing schedule with linear warm-up to allow for stable convergence in the initial training phase and precise optimization in the later epochs. The schedule is given by Equation 9, with $η_{\max} = 1 \times 10^{- 5}$ , $η_{\min} = 1 \times 10^{- 7}$ , the warm-up period $T_{warmup} = 10$ epochs, and total training duration $T_{total} = 125$ epochs.

η_{t} = {\begin{cases} η_{\min} + \frac{t}{T_{warmup}} \times (η_{\max} - η_{\min}) & t \leq T_{warmup} \\ η_{\min} + \frac{1}{2} \times (η_{\max} - η_{\min}) \times [1 + \cos (π \times \frac{t - T_{warmup}}{T_{total} - T_{warmup}})] & t > T_{warmup} \end{cases}

(9)

Exponential Moving Average (EMA): For better inference stability and generalization, we keep a shadow copy of the model parameters, $θ_{EMA}$ , that we keep updated as an exponential moving average of the trainable parameters $θ$ after each optimization step. Note that, while parameters are smoothed, constant buffers $b$ (e.g., batch norm running statistics) are synchronized by direct copy to preserve distributional correctness. The update equations are presented in Equation 10. $θ_{EMA}$ is not updated via gradients during training. At evaluation time, the trainable weights are temporarily switched with $θ_{EMA}$ so that the metrics are computed against the smoothed path and not the noisy optimization one.

\begin{array}{c} θ_{EMA}^{(t)} = α \times θ_{EMA}^{(t - 1)} + (1 - α) \times θ^{(t)} \\ b_{EMA}^{(t)} = b^{(t)} \\ θ_{EMA}^{(0)} = θ^{(0)} \end{array} where \to α = 0.9999

(10)

Gradient Clipping and Mixed Precision: To avoid the gradients becoming too large and to keep the calculations stable, the gradient norms are limited to a maximum of $γ_{\max} = 1.0$ by using $g \leftarrow g \times \min (1, \frac{γ_{\max}}{∥ g ∥_{2}})$ . Additionally, Automatic Mixed Precision (AMP) is used to save memory and speed up training. Loss scaling ( $L_{scaled}$ ) is automatically changed to keep FP16 arithmetic from underflowing by using $s \times L$ where $s \in {2^{0}, 2^{1}, \dots, 2^{16}}$ where the scale factor $s$ is changed up if no gradient overflow is detected and changed down if overflow happens.

Inference Protocol: At the time of final deployment, the model configuration is irrevocably merged with the EMA weights. Let’s consider $f (\times; θ)$ to be the forward pass function. Inference prediction $\hat{y}$ for the presented input $x$ is calculated as: $argmax (SoftMax (f (x; θ_{EMA})))$ . This method guarantees that the deployed model is able to take advantage of the variance reduction effect of the exponential moving average without the extra computational cost of keeping two sets of weights during inference.

2.5. Explainability and Visual Interpretability Analysis

Building clinical trust and aligning the explanations with pathological reasoning have been some of the main concerns addressed by the proposed framework that develops a holistic combination of Explainable AI (XAI) techniques. Instead of depending on a unique attribution method that may be biased toward a specific method, our approach is based on a multifaceted interpretability strategy that draws on Gradient-weighted Class Activation Mapping (Grad-CAM),²⁵ its generalized successor Grad-CAM++,^26,27 and Layer-CAM.²⁸ Such a combination approach thus enables cross-checking of the saliency areas, resulting in the model’s attention mechanisms continuously pinpointing the diagnostically relevant histopathological features even when different attribution paradigms are applied.

Among them, due to its better ability in recognizing multiple occurrences of class-specific features in an image, which is usually the case with heterogeneous tissue structures, etc., Grad-CAM++ is taken as the main localization method. Let’s say, for instance, a target class $c$ and the final convolutional feature map $A^{k} \in R^{u \times v}$ from layer $k$ , the class-discriminative weighting coefficients $α_{k}^{c}$ are calculated in order to incorporate the higher-order gradient information. The remark is given in Equation 11 where $Z$ is used as the normalization constant ( $Z = \sum_{i, j} \frac{\partial^{2} y^{c}}{\partial {(A_{i j}^{k})}^{2}}$ ), and $y^{c}$ represents the pre-softmax logit for class $c$ .

α_{k}^{c} = \frac{1}{Z} \times \sum_{i = 1}^{u} \sum_{j = 1}^{v} \frac{\frac{\partial^{2} y^{c}}{\partial {(A_{i j}^{k})}^{2}}}{| 2 \times \frac{\partial^{2} y^{c}}{\partial {(A_{i j}^{k})}^{2}} + \sum_{a = 1}^{u} \sum_{b = 1}^{v} A_{a b}^{k} \times \frac{\partial^{3} y^{c}}{\partial {(A_{i j}^{k})}^{3}} |}

(11)

The output coarse localization map is given by $L_{Grad ‐ CAM + +}^{c} = ReLU (\sum_{k} α_{k}^{c} \times A^{k})$ . To this end, regular Grad-CAM also gives a first-order gradient attribution baseline. Also, to improve spatial localization, Layer-CAM is used that directly weights the pixels of the feature map with their corresponding gradients, thus, the spatial dilution that usually happens due to global pooling operations is mitigated. The agreement of the saliency maps from these four different approaches not only confirms the firmness of the model’s decision boundaries but also proves that the model is focusing on the biologically relevant morphological features (e.g., the intensity of membrane staining and the preservation of tissue architecture).

2.6. Algorithmic Framework

The proposed approach is a hybrid ensemble framework summarized in Algorithm 1 which basically shows the whole pipeline. It starts with the preprocessing and augmentation of the data and then the feature extraction is done in parallel with the triplet backbone architecture (EVA-02, ViT, and ConvNeXt). The training loop involves adaptive learning rate scheduling, EMA weight stabilization, and mixed-precision optimization. Inference protocol is the last stage that involves deployment of smoothed weights and generation of multi-perspective explainability maps for clinical interpretability.

Algorithm 1. The training and inference pipeline for the Hybrid Ensemble Framework with EMA stabilization and Multi-Perspective XAI.

3. Experimental Results and Discussion

Experimental Configurations and Implementation Details: All experiments were performed on a high-performance computing cluster equipped with a dedicated GPU partition. To have enough resources for both data preprocessing and model optimization, each training job was granted a single NVIDIA GPU node with 48 CPU cores and 250 GB of system memory. The scripts were written in PyTorch and PyTorch Image Models (timm) library. During training, Automatic Mixed Precision (AMP) was turned on to get the most out of computational efficiency and memory usage. AdamW with the starting learning rate of $1 \times 10^{- 5}$ , a weight decay of $0.01$ , and a cosine annealing schedule with a linear warm-up over the first 10 epochs was used to optimize the models. A batch size of 32 was used to train each backbone architecture separately for a maximum of 100 epochs. To improve the generalization and robustness, we combined different regularization strategies such as MixUp augmentation ( $α = 0.5$ ), Exponential Moving Average (EMA) of model weights for inference stabilization and early stopping at 25 epochs of patience based on the validation loss. To guarantee statistical confidence and to prevent the variations caused by the random initialization, we ran each experimental setup 10 different times (Trials 1:10) and averaged the final reported results. Data loading was parallelized, and gradient clipping was implemented to ensure the stability of training.

3.1. Results

The experimental evaluation confirms that the hybrid ensemble framework can effectively facilitate the automation of HER2-IHC scoring with high accuracy. Table 1 illustrates a comparative performance of the three individual backbone architectures (ConvNeXt-V2-Nano, EVA-02-Large, and ViT-Base/P16) with the proposed fused method. All the metrics represent the mean

\pm

standard deviation of ten independent trials, which is a standard practice to obtain reliable statistics. Bold values indicate the proposed fused approach, which achieves statistically significant improvements (p < 0.01, two-tailed paired t-test with Bonferroni correction for multiple comparisons) over all individual backbones across all metrics. Effect sizes were computed using Cohen’s d, with values >0.8 indicating large practical significance. Normality assumptions were verified using Shapiro-Wilk tests (p > 0.05 for all metric distributions), and homogeneity of variance was confirmed via Levene’s test.

Table 1.

Top-Performing Preprocessing and Model configurations Ranked by Weighted F1-Score Across Ten Independent Trials

Model	Precision	Recall	F1-Score	Accuracy	Specificity	BAC	Average
ConvNeXtV2	0.9407 ± 0.0039	0.9376 ± 0.0050	0.9392 ± 0.0044	0.9702 ± 0.0021	0.9811 ± 0.0010	0.9594 ± 0.0028	0.9547 ± 0.0031
EVA-02-Large	0.9680 ± 0.0022	0.9679 ± 0.0024	0.9680 ± 0.0023	0.9843 ± 0.0011	0.9891 ± 0.0009	0.9785 ± 0.0014	0.9760 ± 0.0016
ViT-Base/P16	0.9640 ± 0.0032	0.9638 ± 0.0033	0.9639 ± 0.0033	0.9822 ± 0.0015	0.9880 ± 0.0014	0.9759 ± 0.0022	0.9730 ± 0.0024
Fused (Ours)	0.9740 ± 0.0015	0.9741 ± 0.0014	0.9740 ± 0.0014	0.9876 ± 0.0009	0.9909 ± 0.0007	0.9825 ± 0.0011	0.9805 ± 0.0010

Abbreviations: BAC = Balanced Accuracy; F1-Score = Harmonic mean of Precision and Recall.

Note. Values represent mean $\pm$ standard deviation across ten independent trials. Bold values indicate the proposed fused approach, which achieves statistically significant improvements (p < 0.01, two-tailed paired t-test with Bonferroni correction for multiple comparisons) over all individual backbones across all metrics. Effect sizes were computed using Cohen’s d, with values >0.8 indicating large practical significance. Normality assumptions were verified using Shapiro-Wilk tests (p > 0.05 for all metric distributions), and homogeneity of variance was confirmed via Levene’s test.

Among the independent backbones, the EVA-02-Large architecture demonstrated the best results, with a weighted average accuracy of $97.60 % \pm 0.16 %$ and a Balanced Accuracy (BAC) of $97.85 % \pm 0.14 %$ . This performance is above ViT-Base/P16 ( $97.30 %$ average) and ConvNeXt-V2-Nano ( $95.47 %$ average), indicating that the improved semantic representation learning through rotary position embeddings and SwiGLU activations in EVA-02 is a great fit for the subtle morphological gradients in HER2 staining. Although the ConvNeXt-V2-Nano is computationally efficient, it exhibited a slightly lower sensitivity to distinguishing the equivocal classes (1+ and 2+), which is evident in its lower recall ( $93.76 %$ ).

Most importantly, the proposed Fused (Ours) method, which combines the feature vectors of all three backbones through a weighted adaptive fusion mechanism, has delivered state-of-the-art results for all the metrics. The ensemble has recorded a weighted average accuracy of $98.05 % \pm 0.10 %$ and a BAC of $98.25 % \pm 0.11 %$ , therefore surpassing the best individual backbone (EVA-02) by around $0.45 %$ in average accuracy. Significantly, the fused model has shown decreased variance (lower standard deviation) between the trials, thus implying better stability and generalization. The specificity was as high as $99.09 %$ , which means that the number of false positives has been significantly reduced; a vital requirement to avoid unnecessary administration of toxic targeted therapies to HER2-negative patients.

Figures 3-5 show a detailed breakdown of individual models via normalized confusion matrices as well as Precision-Recall Curves (PRC). The confusion matrices depict that all models have hardly any problems with classes that are visually very distinct (0 and 3+), however, the main difficulty is in 1+ and 2+ borderline categories differentiation. The EVA-02-Large model (Figure 3A) among single models has the most intense diagonal for these ambiguous classes. The PRCs also confirm this fact, with EVA-02 being able to stay at a high precision even when the recall level for minority classes is also greatly increased. To provide deeper clinical insight into the model’s diagnostic behavior, we conducted an additional qualitative analysis of challenging misclassified HER2 equivocal cases. We observed that misclassifications between 1+ and 2+ predominantly occurred in patches exhibiting weak, heterogeneous membrane staining accompanied by high cytoplasmic background noise. This analysis identifies the specific morphological boundaries where the model struggles, guiding future data curation and hard-negative mining efforts.

Figure 3.

Performance evaluation of the EVA-02-Large architecture where: (a) The normalized confusion matrix illustrating the classification distribution across the four HER2 expression classes (0, 1+, 2+, 3+); highlighting the model’s proficiency in distinguishing equivocal cases. (B) The multi-class PRC demonstrating the trade-off between precision and recall for each class; with the micro-average indicating robust overall performance on the test set

Figure 4.

Performance evaluation of the ViT-Base/P16 architecture where: (a) prediction performance of the HER2 classes by means of a confusion matrix; highlighting the fact that the model achieved very high true-positive rates for the two extreme categories over-expression (3+) and negative (0). (B) PRC for class-specific and sample macro-average exemplify the transformer’s strength in capturing global contextual information for accurate scoring

Figure 5.

Evaluation of the ConvNeXt-V2-Nano where: (A) normalized confusion matrix showing the classification behavior of the updated convolutional backbone in particular its capability in segregating borderline 1+ and 2+ cases. (B) PRC of individual classes; reflecting the model’s compromise between sensitivity and positive predictive value over the diverse dataset

To confirm the statistical significance and reliability of the above-mentioned results, we performed multi-trial Receiver Operating Characteristic (ROC) analyses. The 10 runs’ average ROC curves with 95% confidence intervals (shaded areas) are displayed in Figures 6-8. The tight confidence intervals (CIs) for all the architectures indicate low variation and thus high reproducibility. The insets (top-left quadrant of each) show the model’s behavior at the clinically important level of high specificity (False Positive Rate $< 0.2$ ). EVA-02-Large and the fused ensemble demonstrate the closest to the ideal top-left corner performance, thus, confirming their strength in lowering false negatives while keeping high specificity. Hence, the alignment of these quantitative metrics and the visual examinations confirms that the fusion method in question has been able to harness the complementary inductive biases of convolutional and transformer architectures to double diagnostic resolution of HER2 scoring.

Figure 6.

Multi-trial ROC analysis for the EVA-02-Large architecture. The solid line represents the mean ROC curve aggregated over 10 independent trials, while the shaded region denotes the 95% CI, illustrating model stability. The inset (top-left) provides a magnified view of the critical operating region; highlighting the model’s high sensitivity and specificity in distinguishing HER2 classes

Figure 7.

Multi-trial ROC analysis for the ViT-Base/P16 architecture. Mean ROC curves and corresponding 95% CIs are plotted based on 10 independent experimental runs. The narrow confidence bands indicate low variance across trials

Figure 8.

Multi-trial ROC analysis for the ConvNeXt-V2-Nano architecture where it displays the average performance over 10 trials with shaded areas representing the 95% CIs

3.2. Explainability Using CAM Techniques

To close the gap between high-dimensional feature learning and clinical interpretability, we decided to qualitatively evaluate the model’s decision-making process based on three different Class Activation Mapping (CAM) methods: Grad-CAM, Grad-CAM++, and Layer-CAM. Figure 9 shows the spatial attention maps that were produced for two representative test cases with correct classifications made with high confidence ( $> 90 %$ ): a HER2-negative case (Class 0, left column) and a HER2-low positive case (Class 1+, right column). In each panel, the original histopathology patch is combined with saliency visualization in both Jet (raw activation) and Viridis (high-contrast) colormaps, as well as the semi-transparent overlay on the tissue structure.

Figure 9.

Explainability analysis across three attribution techniques for distinct HER2 expression classes. It presents saliency maps generated by (Row 1) Grad-CAM, (Row 2) Grad-CAM++, and (Row 3) Layer-CAM. The left column displays results for a representative HER2-negative (Class 0) sample, while the right column shows a HER2-low positive (Class 1+) case, both correctly predicted with confidence scores exceeding $90 %$ . Each individual panel contains the original histopathology image, the raw activation map (Jet colormap), the semi-transparent overlay on the tissue structure, and the high-contrast visualization (Viridis colormap)

In the HER2-negative sample (Class 0), all three methods identify the nuclear regions stained with hematoxylin as the main focus areas of the first attention layer and at the same time, they are able to elegantly ignore the eosinophilic cytoplasmic and stromal background. This is exactly what we have seen Score 0 in pathology, where lack of membranous staining is the main distinguishing feature. In this context, Grad-CAM++ (Figure 9; second row) and Layer-CAM (Figure 9; third row) offer a higher level of detail in the spatial domain than the traditional Grad-CAM (Figure 9; first row) as they have less spatial dilution and so they can even show different tumor cell clusters.

In contrast, with the HER2-low positive case (Class 1+), the attribution maps refocus the attention on the cellular boundaries, detailing the characteristic faint, incomplete membrane staining of this ambiguous category. While regular Grad-CAM (Figure 9; first row) identifies the main area of interest, it has a broader activation spread which sometimes includes the adjacent stromal tissue. Conversely, Grad-CAM++ (Figure 9; second row) and Layer-CAM (Figure 9; third row) clearly reveal their advantage in specificity by showing only the tumor cell membranes as the high-activation zones that they tightly restrict. This better pinpointing is of great importance for differentiating Class 1+ from Class 2+ (equivocal) and Class 0 since it shows that the model’s high confidence prediction is based on the occurrence of subtle membranous intensity rather than non-specific tissue texture or artifact. The agreement among these different explainability frameworks confirms that the suggested system identifies biologically sound features thus enhancing the level of transparency that is essential for the clinical acceptance of automated HER2 scoring workflows.

To evaluate agreement among attribution methods, we computed pairwise Intersection-over-Union (IoU) scores and Pearson correlation coefficients between binarized saliency maps (thresholded at the 90th percentile of activation) across 500 randomly sampled test patches. Mean IoU values were: Grad-CAM vs. Grad-CAM++ = $0.78 \pm 0.12$ , Grad-CAM vs. Layer-CAM = $0.71 \pm 0.15$ , and Grad-CAM++ vs. Layer-CAM = $0.82 \pm 0.09$ . Corresponding Pearson correlations followed a similar pattern (r = 0.84, 0.76, and 0.88, respectively), indicating strongest consensus between Grad-CAM++ and Layer-CAM, which both incorporate higher-order gradient information. Class-wise analysis revealed highest agreement for distinct classes (0 and 3+: mean IoU $> 0.85$ ) and moderate consensus for equivocal classes (1+ and 2+: mean IoU $= 0.68 - 0.74$ ), consistent with increased diagnostic ambiguity in borderline cases. Those quantitative metrics complement qualitative visual assessments and support the reliability of multi-perspective XAI validation for clinical deployment.

3.3. Perturbation Analysis and Robustness Evaluation

To assess clinical reliability under real-world conditions, we conducted a comprehensive perturbation study encompassing $20 +$ corruption types representative of digital pathology workflows. These include stain variations (brightness, contrast, saturation shifts), acquisition artifacts (Gaussian noise, speckle, shot noise), compression effects (JPEG), and optical distortions (motion blur, zoom blur, glass blur). These corruption types were explicitly mapped to real-world clinical scenarios to ensure translational relevance: Gaussian and speckle noise simulate scanner sensor variations; motion and glass blur mimic tissue folding and focal drift during slide scanning; brightness, contrast, and saturation shifts represent inter-laboratory staining heterogeneity; and JPEG compression models network transmission artifacts in digital pathology workflows. Model robustness was quantitatively evaluated using Mean Corruption Error (mCE) and Expected Calibration Error (ECE). Formally, mCE is computed as in Equation 12 where $C$ denotes the number of corruption types, $E_{clean}$ is the error on clean data (baseline accuracy $\approx 99.0 %$ ), $E_{c}$ is the error under corruption $c$ , and $E_{random}$ represents random classifier error. Lower mCE values indicate superior robustness, while ECE quantifies the concordance between predictive confidence and actual accuracy.

m C E = \frac{1}{C} \times \sum_{c = 1}^{C} [\frac{E_{c l e a n} - E_{c}}{E_{c l e a n} - E_{r a n d o m}}] \times 100 #

(12)

Results reveal a bifurcated robustness profile. The model demonstrates strong resilience to structural and geometric perturbations (including occlusion, fog, shadow, and Gaussian noise) with $m C E \leq 0.05$ and accuracy degradation $< 3.1 %$ . This stability likely reflects the hybrid architecture’s capacity to extract invariant morphological features by combining global receptive fields (ViT components) with hierarchical local pooling (ConvNeXt). Conversely, the framework exhibits marked vulnerability to photometric and intensity-based corruptions. Under extreme brightness shifts, accuracy declined to 62.5% ( $- 36.5 %$ ), followed by speckle noise ( $- 32.3 %$ ) and contrast variations ( $- 26.0 %$ ). These failure modes coincided with elevated ECE ( $> 0.40$ ), indicating that the model becomes both inaccurate and overconfident under severe illumination changes. Class-wise analysis further revealed that equivocal categories (1+ and 2+) were most affected, with accuracy drops of 18.5% and 23.2%, respectively, whereas distinct classes (0 and 3+) maintained robustness ( $< 6 %$ loss).

Figure 10 visualizes the joint accuracy-ECE landscape across perturbation types and severity levels, highlighting a photometric vulnerability region where calibration deteriorates most sharply. Figure 11 ranks robustness via normalized mCE, reinforcing that while the model tolerates structural noise effectively, targeted mitigation of stain variability (via physics-informed normalization or adversarial augmentation) remains essential for multi-center deployment. Clinically, structural corruptions simulate common acquisition artifacts, whereas photometric shifts model inter-scanner and inter-laboratory staining heterogeneity; addressing the latter is critical for ensuring reliable HER2 scoring across diverse institutional settings.

Figure 10.

Calibration and accuracy landscape under perturbations where the heatmap shows the joint distribution of model accuracy and ECE depending on different perturbation types (y-axis) and severity levels (x-axis). The color intensity depicts the magnitude of ECE with warmer colors (red/orange) indicating areas where the model confidence diverges significantly from the true accuracy (miscalibration). Cooler colors (blue/green) indicate the stable regimes where the model accuracy is high and the confidence estimates are reliable at the same time

Figure 11.

Quantitative Robustness Ranking via mCE where the bar chart depicts the normalized mCE for each perturbation category; serving as an aggregate metric of model fragility. Lower mCE values (closer to 0) indicate superior robustness; while higher values signify significant performance degradation relative to the baseline

Synthesis of Robustness Findings: Combining those analyses gives an important insight for clinical use: the hybrid model that we proposed has a kind of built-in robustness to structural and geometric noise, but it is still very sensitive to changes in the photometric domain. This vulnerability is typical of stain variability in histopathology, which is the main cause of different brightness and contrast being recorded when using different scanners, tissue also being cut in varying thinness and chemical reagents being applied in varying concentrations. The association of very severe photometric perturbations with a very high ECE is consistent with the idea that, in these cases, the model does not only make mistakes but, additionally, becomes very uncertain in its predictions. Therefore, to achieve this, future iterations should consider not just standard data augmentation but, rather, the integration of stain normalization layers using physics-informed methods and/or adversarial training that is specifically targeting the color-intensity manifold. The latter is crucial for the models’ confidence to remain a valid indicator of its predictions correctness when faced with data coming from a variety of different laboratories and thus a way to prevent possible misclassification in borderline HER2 cases where therapy decisions are critical.

3.4. Comparison With Related Studies

To contextualize the performance of the proposed hybrid ensemble framework, Table 2 presents a comparative analysis against recent state-of-the-art methods in HER2 scoring and breast cancer histopathology. The proposed method demonstrates superior performance in terms of weighted accuracy and balanced accuracy, particularly in handling equivocal cases (1+ and 2+), which are often challenging for single-architecture models. While Yang et al.¹³ focused on recurrence risk prediction using multimodal fusion of H&E images and clinical data, and Farahmand et al.¹⁴ employed ROI-annotated H&E slides for HER2 status classification, our framework operates directly on expert-annotated IHC patches without requiring auxiliary clinical metadata or manual tumor region delineation. Furthermore, unlike Rashid et al.¹⁵ who combined ResNet50-based transfer learning with NSGA-II metaheuristic feature selection and SVM classification, our adaptive late-fusion mechanism dynamically integrates high-level semantic representations from three complementary backbones. Crucially, the integration of a multi-perspective XAI module (Grad-CAM, Grad-CAM++, and Layer-CAM) addresses the “black-box” limitation noted in prior works; ensuring that model attention aligns with pathologist-validated morphological features such as membrane staining intensity and nuclear architecture.

Table 2.

Comparison of the Proposed Framework With Related Studies in HER2 Scoring and Breast Cancer Histopathology

Study	Methodology	Dataset & Size	Performance	Interpretability
Yang et al.¹³	Multimodal deep learning: ResNet50 for H&E image features + clinical data fusion via Multimodal Compact Bilinear (MCB) pooling.	CAMS (127 HER2+ patients); TCGA-BRCA (123 HER2+ patients, independent test).	Recurrence prediction: AUC 0.76 (2-fold CV), AUC 0.72 (TCGA test).	Not reported.
Farahmand et al.¹⁴	Inception v3 CNN trained on pathologist-annotated tumor ROIs from H&E WSIs; three-way classifier (HER2+, HER2–, Other).	Yale HER2 cohort (188 slides: 93+, 95–); TCGA-BRCA (187 samples, independent test).	HER2 status: AUC 0.90 (CV), AUC 0.81 (TCGA); Trastuzumab response: AUC 0.80 (5-fold CV).	Heatmap visualization (tile-level probability maps); no formal XAI framework.
Rashid et al.¹⁵	Transfer learning (ResNet50) + NSGA-II metaheuristic optimizer for feature selection + SVM classifier; FlLpF preprocessing.	HER2SC (158 WSIs $\to$ 1,600 cropped images); HER2GAN (126 patients).	Acc: 94.4%; Prec: 93.71%; Spec: 98.07%; Sens: 93.83%; F1: 93.71%.	Not reported.
Nabi et al.¹⁸	Dataset contribution: HER2-IHC-40 $\times$ (107 WSIs, 10,997 patches); baseline models (HRNet, SwinT) reported in companion work.	HER2-IHC-40 $\times$ ( $N$ =10,997 patches; 4 classes: 0, 1+, 2+, 3+).	Baseline results in companion study; dataset enables reproducible benchmarking.	Dataset designed to support XAI research (Grad-CAM, SHAP, attention-based models suggested).
Proposed	Hybrid ensemble: EVA-02-Large + ViT-Base + ConvNeXt-V2-Nano via adaptive late-fusion; EMA stabilization; multi-perspective XAI.	HER2-IHC-40 $\times$ ( $N$ =10,997 patches; expert-annotated, 40 $\times$ ).	Weighted Acc: $98.05 \pm 0.10 %$ ; BAC: $98.25 \pm 0.11 %$ ; mCE $< 0.05$ (structural perturbations).	Grad-CAM, Grad-CAM++, Layer-CAM (cross-validated saliency maps).

Abbreviations: H&E = Hematoxylin and Eosin; WSI = Whole Slide Image; ROI = Region of Interest; CV = Cross-Validation; AUC = Area Under the Curve; Acc = Accuracy; Prec = Precision; Spec = Specificity; Sens = Sensitivity; F1 = F1-Score; BAC = Balanced Accuracy; mCE = mean Calibration Error; XAI = Explainable AI; EMA = Exponential Moving Average; MCB = Multimodal Compact Bilinear pooling; NSGA-II = Non-dominated Sorting Genetic Algorithm II; FlLpF = Feature-level Local Pattern Filtering.

Note. Values for the proposed framework represent mean $\pm$ standard deviation over ten independent trials. Performance metrics are reported as presented in original studies; direct comparison should consider dataset heterogeneity, evaluation protocols, and task definitions (e.g., recurrence prediction vs. HER2 status classification).

Several key distinctions emerge from this comparison. First, while prior works predominantly rely on H&E-stained sections,^13,14 our framework is explicitly designed for HER2-IHC scoring, directly modeling the membrane staining patterns that constitute the clinical gold standard.² Second, the proposed triplet-backbone architecture captures complementary inductive biases: global semantic dependencies (EVA-02), pure attention-based context modeling (ViT), and hierarchical local feature extraction (ConvNeXt), whereas single-backbone approaches may underrepresent the multi-scale morphological signatures characteristic of equivocal HER2 classes. Third, our adaptive late-fusion mechanism operates at the high-level feature space, enabling dynamic weighting of discriminative representations without the computational overhead of iterative metaheuristic search.¹⁵ Finally, the multi-perspective XAI protocol provides convergent evidence that model decisions are grounded in pathologically relevant features, thereby enhancing clinical trust and facilitating regulatory adoption; a consideration largely absent in prior methodological contributions. Compared to recent single-backbone or weakly supervised HER2-IHC frameworks, our novelty lies explicitly in the triplet-backbone adaptive fusion and the multi-perspective XAI protocol, which directly resolve the limitations of prior models that struggle with the multi-scale morphological signatures of equivocal classes.

4. Limitations

Despite the strong performance and comprehensive evaluation presented in this study, several limitations warrant acknowledgment. First, while the model demonstrates robustness to geometric and structural perturbations, our perturbation analysis revealed notable sensitivity to photometric variations; particularly extreme alterations in brightness, contrast, and saturation. This vulnerability reflects real-world challenges in histopathology, including inter-laboratory staining variability and differences in scanner hardware. Although standard data augmentation was employed, the pronounced performance decline under severe intensity corruptions suggests that future iterations should integrate physics-informed stain normalization techniques (e.g., Macenko or Vahadane methods) or adversarial training targeting the color-intensity manifold to enhance cross-institutional generalizability.

Second, this work utilizes the HER2-IHC-40x dataset, an expert-annotated, high-resolution resource curated from a single institutional cohort. While this dataset enables rigorous internal validation, the framework’s generalizability to external multi-center cohorts with diverse staining protocols, tissue fixation methods, and scanning platforms remains to be established. To address this, our future work incorporates a structured external validation roadmap: retrospective evaluation on publicly available benchmark datasets (including BACH,²⁹ CAMELYON16/17,³⁰ and the TCGA-BRCA HER2-annotated subset) to assess cross-domain performance, coupled with nested cross-validation protocols (outer loop for performance estimation, inner loop for hyperparameter tuning) to ensure unbiased generalization estimates and mitigate model selection bias.^31-35

Third, the hybrid ensemble architecture integrates three distinct backbones, which increases computational complexity (FLOPs) and memory requirements relative to single-model baselines. While the patch-level nature of the input allows for highly parallelized inference on modern GPU clusters, this design may constrain deployment in resource-limited clinical environments. Currently, the fused model requires approximately three times the inference latency of a single backbone. A comprehensive discussion of inference efficiency is therefore critical; future optimization efforts will explore model compression strategies (including knowledge distillation, structured pruning, and 8-bit quantization) to reduce inference latency and hardware requirements, thereby improving deployment feasibility in routine digital pathology workflows without compromising diagnostic fidelity.

Fourth, the current framework relies on patch-level evaluation, which warrants clarification regarding slide-level aggregation for routine pathology deployment. In a clinical workflow, patch-level predictions must be aggregated to generate a final slide-level HER2 score. Future implementations will employ attention-based Multiple Instance Learning (MIL) or weighted majority voting to synthesize these patch-level outputs. While this introduces additional computational overhead for whole-slide inference, it is a necessary step to align with standard pathologist reporting practices.

Finally, although our multi-perspective XAI analysis confirms that model attention aligns with pathologist-validated morphological features (e.g., membrane staining intensity), the current framework provides post-hoc interpretability rather than intrinsically transparent decision-making. Incorporating explainability constraints directly into the training objective (such as attention-guided loss functions or concept bottleneck architectures) represents a promising direction for developing inherently interpretable models that facilitate clinical trust and regulatory adoption.

5. Conclusions and Future Directions

This study presents a high-fidelity deep learning framework for automated HER2-IHC scoring in breast cancer histopathology. By synergizing EVA-02-Large, ViT-Base, and ConvNeXt-V2-Nano through an adaptive late-fusion mechanism, our ensemble achieved state-of-the-art weighted accuracy (98.05%) and balanced accuracy (98.25%). Critically, the framework demonstrated superior proficiency in resolving diagnostically ambiguous equivocal cases (HER2 1+ and 2+), thereby addressing a persistent source of inter-observer variability in routine clinical practice. Integration of a multi-perspective explainability module (Grad-CAM, Grad-CAM++, and Layer-CAM) confirmed that model decisions are grounded in pathologist-validated morphological features, including membrane staining intensity and nuclear architecture, which is essential for fostering clinical trust and facilitating regulatory adoption.

While the model exhibits robustness to structural perturbations, its sensitivity to photometric variations underscores the need for enhanced domain generalization. To address this limitation and strengthen clinical translatability, we have established a comprehensive future research roadmap. Near-term efforts will prioritize the integration of physics-informed stain normalization techniques and adversarial training protocols designed to mitigate intensity-based degradation across diverse laboratory settings. Concurrently, we will implement a structured external validation strategy: Phase 1 will involve retrospective evaluation on public benchmark datasets (BACH, CAMELYON16/17, and TCGA-BRCA HER2 subsets) utilizing nested cross-validation to ensure unbiased performance estimation; Phase 2 will expand to prospective multi-institutional cohorts to assess real-world generalizability across varied pre-analytical conditions; and Phase 3 will conduct clinical utility studies comparing pathologist diagnostic agreement with and without AI assistance. Additionally, we will explore model compression strategies, including quantization and knowledge distillation, to optimize computational efficiency for deployment in resource-constrained clinical environments. Standardized evaluation scripts and pre-processed benchmark splits will be publicly released to facilitate reproducible research and accelerate community-wide progress in automated HER2-IHC scoring. By unifying architectural innovation, rigorous robustness assessment, and transparent decision-making, this framework establishes a foundational benchmark that, pending rigorous prospective clinical trials, paves the way for the future integration of AI-assisted diagnostics into routine oncological workflows; ultimately supporting more precise and personalized therapeutic management for patients with HER2-positive breast cancer.

Footnotes

Acknowledgements

The authors extend their appreciation to the Deanship of Scientific Research and Libraries in Princess Nourah bint Abdulrahman University for funding this research work through the Research Group project, Grant No. (RG-1445-0062).

ORCID iDs

Hossam Magdy Balaha

Ayman El-Baz

Ethical Cosiderations

Ethical approval and informed consent were not applicable for this study. The research utilized the HER2-IHC-40x dataset, which is publicly accessible via Zenodo (). No informed consent was required for the current study as per the declarations provided in the source material.

Consent to Participate

Patient informed consent was waived by the originating Institutional Review Board due to the retrospective nature of dataset curation and the use of fully de-identified histopathology images. No personal identifiable information was accessed or utilized in this research.

Author Contributions

Hossam Magdy Balaha: Conceptualization, Methodology, Software, Formal analysis, Writing - original draft, Investigation, Visualization, Validation, Writing - review & editing. Norah Saleh Alghamdi, Nithya Rekha Sivakumar, and Shakila Basheer: Funding, Validation, Visualization, Writing - review & editing. Khadiga M. Ali: Validation, Visualization, Writing - review & editing. Ayman El-Baz: Supervision, Funding, Validation, Visualization, Writing - review & editing. All authors have read and agreed to the published version of the manuscript.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The dataset supporting this study is available from a public repository upon reasonable request.*

Intellectual Property

We have followed the regulations of our institutions concerning intellectual property.

Use of Artificial Intelligence Statement

No generative artificial intelligence tools were used for content generation, data analysis, figure creation, or manuscript writing. All scientific content, interpretations, and conclusions were generated solely by the authors.

References

Cao

Qin

Chen

. Comparative study of cancer profiles between 2020 and 2022 using global cancer statistics (GLOBOCAN). J Natl Cancer Cent. 2024;4:128-134.

Wolff

Hammond

MEH

Allison

, et al. Human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology/College of American Pathologists clinical practice guideline focused update. Arch Pathol Lab Med. 2018;142:1364-1382.

Laokulrath

Gudi

Salahuddin

, et al. Human epidermal growth factor receptor 2 (HER2) status in breast cancer: practice points and challenges. Histopathology. 2024;85:371-382.

Chumsri

Serie

, et al. Incidence of late relapses in patients with HER2-positive breast cancer receiving adjuvant trastuzumab: combined analysis of NCCTG N9831 (alliance) and NRG oncology/NSABP B-31. J Clin Oncol. 2019;37:3425-3435.

Piccart

Procter

Fumagalli

, et al. APHINITY Steering Committee and Investigators. Adjuvant pertuzumab and trastuzumab in early HER2-positive breast cancer in the APHINITY trial: 6 years’ follow-up. J Clin Oncol. 2021;39:1448-1457.

Kabir

Vranic

Al Saady

, et al. The utility of a deep learning-based approach in Her-2/neu assessment in breast cancer. 2024;238:122051.

Wolff

Somerfield

Dowsett

, et al. Human epidermal growth factor receptor 2 testing in breast cancer: ASCO–College of American Pathologists Guideline Update. J Clin Oncol. 2023;41:3867-3872.

Jakobsen

Teerapakpinyo

Shuangshoti

Keelawat

. Comparison between digital image analysis and visual assessment of immunohistochemical HER2 expression in breast cancer. Pathol Res Pract. 2018;214:2087-2092.

Miao

, et al. Artificial intelligence for assisted HER2 immunohistochemistry evaluation of breast cancer: a systematic review and meta-analysis. Pathol Res Pract. 2024;260:155472.

10.

Coudray

Ocampo

Sakellaropoulos

, et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24:1559-1567.

11.

Noorbakhsh

Farahmand

Foroughi Pour

, et al. Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images. Nat Commun. 2020;11:6367.

12.

Mobadersany

Yousefi

Amgad

, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018;115:E2970-E2979.

13.

Yang

Guo

, et al. Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning. Comput Struct Biotechnol J. 2022;20:333-342.

14.

Farahmand

Fernandez

Ahmed

, et al. Deep learning trained on hematoxylin and eosin tumor region of Interest predicts HER2 status and trastuzumab treatment response in HER2+ breast cancer. Mod Pathol. 2022;35:44-51.

15.

Rashid

Majidpour

Thinakaran

, et al. NSGA-II-DL: metaheuristic optimal feature selection with deep learning framework for HER2 classification in breast cancer. IEEE Access. 2024;12:38885-38898.

16.

Yang

, et al. Predicting HER2 status in breast cancer on ultrasound images using deep learning method. Front Oncol. 2022;12:829041.

17.

Nabi

Fauzi

MFA

Karim

HBA

Ahmed

RRM

Santo

Hossen

. Hybrid Deep Learning Framework for Multi-Class Breast Cancer Scoring Using Grad-CAM++. In: Proceedings of the 2025 9th International Conference on Information Technology (InCIT). IEEE; 2025:257-264.

18.

Nabi

Fauzi

MFA

Rehman

, et al. Her2-ihc-40x: A high-resolution histopathology dataset for her2 ihc scoring in breast cancer; 2025.111922.

19.

Collins

Moons

Dhiman

, et al. TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. bmj. 2024;385:e078378.

20.

Zhang

Cisse

Dauphin

Lopez-Paz

. Beyond empirical risk minimization; 2017.

21.

Fang

Sun

Wang

Huang

Wang

Cao

. Eva-02: A visual representation for neon genesis. Image Vis Comput. 2024;149:105171.

22.

Dosovitskiy

Beyer

Kolesnikov

, et al. An image is worth 16x16 words: Transformers for image recognition at scale; 2020.

23.

Woo

Debnath

, et al. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In: Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2023:16133-16142.

24.

Loshchilov

Hutter

. Decoupled weight decay regularization; 2017.

25.

Bhakte

Vasista

Srinivasan

. Gradient-weighted class activation mapping (Grad-CAM) based explanations for process monitoring results from deep neural networks. In: Proceedings of the 2021 AIChE Annual Meeting. AIChE; 2021.

26.

Chattopadhay

Sarkar

Howlader

Balasubramanian

. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In: Proceedings of the 2018 IEEE winter conference on applications of computer vision (WACV). IEEE; 2018:839-847.

27.

Gao

Liu

Hou

Zhao

. Augmented grad-cam++. Super-resolution saliency maps for visual interpretation of deep neural network. 2023;12:4846.

28.

Dong

Jiang

Huang

, et al. Rethinking class activation maps for segmentation: revealing semantic information in shallow layers by reducing noise; 2023.

29.

Aresta

Araújo

Kwok

, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis. 2019;56:122-139.

30.

Sun

Meng

Liu

. CAMELYON 17 challenge: a comparison of traditional machine learning (SVM) with the deep learning method. Wireless Communications and Mobile Computing. 2022;2022(1):9910471.

31.

Banupriya

Sethukarasi

. Gene-biomarker data-driven deep learning model with m-dman for er, pr, her2 enrichment and breast cancer association prediction. Neural Computing and Applications. 2026;38(1):2.

32.

Banupriya

Sethukarasi

. Deep learning-driven multi-omics sequential diagnosis with Hybrid-OmniSeq: Unraveling breast cancer complexity. Technology and Health Care. 2025;33(2):1099-1120.

33.

Al-Nussairi

AKJ

Ali

Malik

, et al. PatchSight–ImmuneMap–LifeSpan as a unified AI framework for breast cancer diagnosis, immune profiling and prognostic prediction. Discover Oncology. 2026.

34.

Altameem

Mahanty

Poonia

Saudagar

AKJ

Kumar

. Breast cancer detection in mammography images using deep convolutional neural networks and fuzzy ensemble modeling techniques. Diagnostics. 2022;12(8):1812.

35.

Sahoo

Nayak

Tripathy

, et al. Predicting breast cancer relapse from histopathological images with ensemble machine learning models. Current Oncology. 2024;31(11):6577-6597.