Multimodal Coupling Prompt Learning for Image Classification Tasks

2.1. VLP Models

Given the success of pretrained models in the fields of CV and NLP, numerous studies have attempted to pretrain large-scale models on the joint modality of vision and language. These pretrained models are known as VLP models. The rise of VLP paradigms began with the transfer of BERT (Devlin et al., 2018) to cross-modal representation learning, followed by a series of studies Lu et al. (2019) introducing BERT into multimodal pretraining. Recently, the encoder–decoder framework’s multimodal pretraining paradigms have gained attention, with many encoder–decoder models achieving state-of-the-art performance in cross-modal understanding and generation tasks (Bao et al., 2022).

Another research trend in the VLP field is contrastive learning. The most typical contrastive learning-based VLP models is CLIP, which uses vision transformer (ViT) (Dosovitskiy et al., 2020) or ResNet (He et al., 2016) as the image encoder, transformer (Vaswani et al., 2017) as the text encoder, and a contrastive loss objective to jointly train the two encoders. CLIP’s pretraining dataset is extensive, comprising approximately 400 million image–text pairs. Following CLIP, a series of studies have demonstrated that successful pretraining with contrastive learning methods on large-scale data is not accidental (Jia et al., 2021).

2.2. Prompt Learning

The motivation behind prompt learning is to leverage the prior knowledge learned by large-scale pretrained models to perform various downstream tasks. Prompt learning can be summarized as “pretraining, prompting, predicting” reorganizing downstream tasks into forms similar to pretraining tasks, thereby better guiding models to utilize pretrained knowledge to complete specific downstream tasks.

Schick and Schütze (2020) proposed PET (Pattern-Exploiting Training), transforming the input of text classification tasks into cloze questions to convert the reasoning process into a text generation task, fully utilizing the text generation capabilities of language models. Petroni et al. (2019) proposed LAMA (LAnguage Model Analysis), modifying the relation extraction task into cloze questions without altering the pretrained language model, achieving better relation extraction performance than knowledge bases. Compared to fine-tuning, prompt learning freezes most of the pretrained model’s parameters, adjusting the pretrained model with minimal parameter amounts (e.g., 1%). Recent advancements indicate that prompt learning can help pretrained models achieve performance comparable to fine-tuning across different NLP downstream tasks, including natural language understanding and generation (Liu et al., 2021).

2.3. Prompt Learning in VLP Models

Recent studies in the vision-language learning field have also demonstrated the effectiveness of prompt learning. Khattak et al. (2023) showed that visual prompt learning could surpass fine-tuning in a series of tasks, with significant advantages in training efficiency. In cross-modal representation learning, prompt learning for CLIP has become a major focus. CLIP is a contrastive learning-based multimodal pretraining model. By using handcrafted prompt templates to convert labels into text descriptions, CLIP achieves remarkable performance in zero-shot image classification tasks. To further enhance performance, CLIP also proposed prompt ensemble methods by manually creating multiple prompt templates. Creating fixed prompts can be a labor-intensive process, which has led to the adoption of continuous prompts or the integration of adapters (Adapter) for CLIP. In addition to CLIP, another area of research explores the use of visual prompts with pretrained language models for multimodal representation learning. These studies demonstrate that even when large-scale pretrained language models are frozen during downstream transfer, they can still effectively adapt to few-shot learning scenarios in multimodal tasks. Similarly, the combination of adapters with CLIP has been shown to yield promising performance. CoOp Zhou et al. (2022b) enhances CLIP for few-shot transfer by fine-tuning a continuous set of prompt vectors within its language branch. On the other hand, CoCoOp (Zhou et al., 2022a) addresses the generalization challenges of CoOp, particularly its underperformance in novel classes. This is achieved by explicitly conditioning the prompts on specific image instances, thus improving its adaptability to previously unseen data. MaPLe (Khattak et al., 2023) adds learnable context tokens in the language branch and conditions vision prompts on language prompts through a coupling function to enable interaction. Inspired by Maple, we add self-attention to the text encoder and project features to the visual encoder, enriching text and guiding visual learning. Cross-modal interaction between text and visual outputs generates prompt features that guide contrastive learning.

3.1. Problem Description and Analysis for CLIP’s Feature Extraction Bias

Based on RA-CLIP, we believe that CLIP’s visual encoder also suffers from feature extraction bias between the pretraining and fine-tuning stages in image classification tasks. The primary cause is the uneven quantity and distribution of image features learned during the pretraining stage. This can be further summarized as an inherent bias in feature extraction for differently distributed samples during the pretraining phase.

For example, the VLP models may predominantly learn features of common flowers such as “roses,” “carnations,” and “tulips” during the pretraining stage. These common flowers form the VLP models’ prior knowledge and recognition capability for the concept of “flowers.” However, when the VLP models are deployed on a downstream image classification dataset that includes a wide variety of flower species, including many rare ones that the VLP models did not encounter during the pretraining stage, such as “ghost orchids,” “blue cat’s face orchids,” and so on, it struggles to recognize and correctly classify these rare flower species. This visual feature recognition bias due to the uneven image distribution between pretraining and fine-tuning stages is particularly detrimental to fine-grained image classification datasets such as Flower101 (flower species dataset), StanfordCars (vehicle dataset), and FGVC Aircraft (aircraft dataset).

In summary, the model tends to more effectively recognize categories frequently appearing in the pretraining dataset, whereas its feature extraction and recognition abilities significantly diminish for less frequent categories due to insufficient sample support. This learning imbalance directly reflects the model’s prior knowledge bias, where the knowledge base formed during the pretraining stage is not comprehensive but rather overly optimized for certain categories and lacks necessary adaptability for others. When these pretrained models are deployed on specific downstream image classification datasets, the aforementioned feature extraction bias further impacts the model’s recognition accuracy on those datasets. This impact is typically caused by distribution differences between datasets, that is, the visual feature space distance between the pretraining dataset and the downstream task dataset. To narrow this distance, it is essential for the model to capture a broader and more balanced range of visual features during the pretraining stage, thereby enhancing its adaptability to various categories and scenarios.

3.2. Multimodal Coupling Prompt Learning CLIP (MCPL-CLIP)

3.2.1. THIPL Method

As shown in Figure 1, this section proposes the THIPL method, which optimizes both the visual and text encoders of CLIP by creating learnable prompt embeddings in the text modality and injecting them into the corresponding prompt embeddings of the visual modality. The specific implementation of THIPL involves adding learnable prompt vectors to several layers of the text encoder’s transformer structure, which are then introduced into the corresponding layers of the visual encoder’s transformer structure through a bottleneck mapping structure. This method retains the prior knowledge learned by the visual encoder during the pretraining stage and allows it to more easily capture the classification features of new image samples during the generalization process.

Essentially, THIPL is based on the principle of complementarity in cross-modal learning, achieving deeper semantic alignment between images and text to better understand and express the intrinsic features of image data. Text-semantic hierarchy injection prompt learning includes the following components: Text Prompt Features. To learn the contextual features of the text encoder, we first introduce b learnable embedding vectors { P i ∈ R d } i = 1 b and add them to the input layer transformer structure of CLIP’s text encoder T ( ⋅ ) . Self-Attention Layer. Considering the instability in a semantic expression of the embedding vectors { P i ∈ R d } i = 1 b due to random initialization parameters in the early stages of training, we introduce a self-attention layer (self-attention layer) to extract semantic relevance features and high-dimensional abstract features from the text descriptions, enhancing the semantic diversity and training stability of the embedding vectors { P i ∈ R d } i = 1 b .

OPEN IN VIEWER

Figure 1.

THIPL, where “add” denotes element-wise addition, Word Embed represents the word embedding method, Patch Embed represents the image patch embedding method, and F ϕ ( ⋅ ) represents the PMS. P 1 , P 2 , … , P b and P ~ 1 , P ~ 2 , … , P ~ b denote text prompt vectors and visual prompt vectors, respectively. W 0 represents the word embedding of the text, and W 0 ′ is the vector obtained from W 0 after passing through the self-attention layer. P 1 ′ , P 2 ′ , … , P b ′ are the results of element-wise addition between W 0 ′ and P 1 , P 2 , … , P b . P 1 1 , P 1 2 , … , P 1 b denote the learnable text embedding vectors created at each layer of the text encoder, while P ~ 1 1 , P ~ 1 2 , … , P ~ 1 b represent the learnable visual embedding vectors created at each layer of the visual encoder. The superscript indicates the layer number, and the subscript indicates the specific prompt at that layer. Note. THIPL = text-semantic hierarchy injection prompting learning; PMS = prompt mapping structure.

Let SA λ ( ⋅ ) represent the self-attention layer defined by training parameters λ . Then the text prompt vectors { P i ∈ R d } i = 1 b , formed by adding the self-attention output features and embedding vectors { P i ∈ R d } i = 1 b , can be represented by equation (1):

{ P i ∈ R d } i = 1 b = [ P 1 , P 2 , … , P b ] = { P i ∈ R d } i = 1 b + SA λ ( [ w 1 , w 2 , … , w N ] ) ,

(1)

where [ w 1 , w 2 , … , w N ] represent the concatenated fixed text features of the input layer. The plus sign represents element-wise addition. Considering that the CLIP uses “A photo of a {CLASS},” for comparison purposes, we have also used this fixed text feature. In this context, the concatenated fixed text features refer specifically to the phrase “A photo of a {CLASS},” which is incorporated into the model to provide a consistent textual input for comparison.

Thus, the input embedding features in CLIP’s text encoder T ( ⋅ ) can be expressed as equation (2):

S input = { P i ∈ R d } i = 1 b ∪ W 0 = [ P 1 , P 2 , … , P b , w 1 , w 2 , … , w N ] ,

(2)

where W 0 = [ w 1 , w 2 , … , w N ] .

We introduce this structure into the first K layers of the CLIP text encoder T ( ⋅ ) ’s transformer structure { L j ( ⋅ ) } j = 1 K , where K is a hyperparameter representing the depth of prompt embedding. Assuming the text encoder T ( ⋅ ) has M transformer layers, the features transmitted in the first K layers of the transformer structure can be expressed by equation (3):

S i = T i ( [ P i − 1 , W i − 1 ] ) , i = 1 , 2 , … , K ,

(3)

where [ ⋅ , ⋅ ] denotes concatenation. When K = 1 , S i = S input .

The features transmitted in the rear M − K layers of the text encoder T M ( ⋅ ) ’s transformer structure can be expressed by equation (4):

S j = T j ( [ _ , W j − 1 ] ) , j = K + 1 , K + 2 , … , M .

(4)

Therefore, the final output features of the text encoder Z text = T M ( W M − 1 ) can be expressed by equation (5):

Z text = T M ( W M − 1 ) .

(5)

PMS. To construct a mapping mechanism from text prompts to visual prompts, we introduce a PMS defined by training parameters ϕ , which maps text prompt vectors { P i ∈ R d } i = 1 b into visual prompt vectors { P ~ i ∈ R d } i = 1 b and adds them to the corresponding layers of the visual encoder’s transformer structure. Assuming the PMS { P ~ i ∈ R d } i = 1 b is defined by training parameters ϕ , it can be expressed by equation (6):

F ϕ ( ⋅ ) = Bottleneck ( MLP ϕ de-dimension ( ⋅ ) , ReLU ( ⋅ ) , MLP ϕ up-dimension ( ⋅ ) , AvgPooling ( ⋅ ) ) ,

(6)

where MLP denotes a multilayer perceptron, de-dimension denotes dimensionality reduction, up-dimension denotes dimensionality increase, AvgPooling( ⋅ ) denotes average pooling, and ReLU denotes rectified linear unit, respectively.

Based on the definition of PMS, visual prompt vectors { P ~ i ∈ R d } i = 1 b can be expressed by equation (7):

{ P ~ i ∈ R d } i = 1 b = F ( { P i ∈ R d } i = 1 b ) .

(7)

Assuming CLIP’s visual encoder is represented by V ( ⋅ ) , the input embedding features I in the visual encoder V ( ⋅ ) can now be expressed by equation (8):

I input = [ P ~ 1 , P ~ 2 , … , P ~ b , X 0 ] , P ~ n = F ϕ ( P n ) , n = 1 , 2 , … , b ,

(8)

where X 0 denotes the image input features and n represents any one of the b prompts in a certain layer.

Similar to the text encoder T ( ⋅ ) , we introduce this structure into the front K layers of the visual encoder. Assuming the visual encoder V M ( ⋅ ) has M transformer layers, the features transmitted in the front K layers of the visual encoder’s transformer structure can be expressed by equation (9):

I i = V i ( [ P ~ i − 1 , X i − 1 ] ) , i = 1 , 2 , … , K ,

(9)

where [ ⋅ , ⋅ ] denotes concatenation. When K = 1 , I i = I input .

The features transmitted in the rear M − K layers of the visual encoder V M ( ⋅ ) ’s transformer structure can be expressed by equation (10):

I j = V j ( [ _ , X j − 1 ] ) , j = K + 1 , K + 2 , … , M .

(10)

Therefore, the final output features of the visual encoder Z image can be expressed by equation (11):

Z image = V M ( X M − 1 ) .

(11)

3.2.2. VAIPL Method

Optimizing and improving text descriptions in the text encoder is a crucial research direction for CLIP models, as it significantly impacts the model’s generalization performance. In the past, during generalization from source to target datasets, it was often difficult to capture concrete visual features that express key semantic information. This is because the text encoder cannot provide a text description that adapts to the new data distribution, making it difficult for the vision-language model to activate its pretrained knowledge circuits for vision-language alignment.

Building on THIPL, we further propose the VAIPL method, as shown in Figure 2. VAIPL method introduces information from the visual encoder under specific data distributions into the text encoder, using the visual information under the specific data distribution to construct instance-level enhanced text features, thereby reducing or compensating for the feature expression differences between modalities in cross-data distribution generalization scenarios.

OPEN IN VIEWER

Figure 2.

Structure of visual-semantic attention interaction prompt learning (VAIPL). It uses the text embeddings as query vectors after max-pooling, and the visual encoder’s output features as key and value vectors. The first multiplication sign represents the product of text features (Q) and image features (K), while the second multiplication sign represents the product of the attention map (the product of Q and K) and image features (V).

The implementation of the VAIPL method is to apply cross-attention between instance-level image features from the visual encoder and text input embeddings and generate instance-level image-weighted text features through a nonlinear projection structure, referred to as semantic interaction prompt features. VAIPL helps the vision-language model learn patterns that identify and extract features beneficial for cross-modal alignment from specific data distributions. VAIPL ensures that semantic interaction prompt features can be integrated into text features in a language-compatible form, thereby enhancing the vision-language model’s modality alignment capability in generalization scenarios.

Additionally, we use CLIP’s handcrafted template “a photo of a CLASS” to nest the annotations of different datasets at the text embedding input layer. This method alleviates text feature mismatches and training instability issues during the domain generalization process.

VAIPL consists of multiple transformer decoder layers, a max-pooling function, and a nonlinear projection structure. Assuming T F μ ( ⋅ ) represents the cross-attention structure defined by training parameters μ , and G ϵ ( ⋅ ) represents the nonlinear projection structure defined by training parameters ϵ , it can be expressed by equation (12):

G ϵ ( ⋅ ) = [ MLP ϵ up-dimension ( ⋅ ) , ReLU ( ⋅ ) , MLP ϵ de-dimension ( ⋅ ) ] ,

(12)

where MLP denotes a multilayer perceptron, de-dimension denotes dimensionality reduction, up-dimension denotes dimensionality increase, and [ ; ] denotes concatenation.

VAIPL can be defined by equation (13):

VAIPL μ , ϵ = [ Maxpooling ( ⋅ ) , TF μ ( ⋅ ) , G ϵ ( ⋅ ) ] ,

(13)

where Maxpooling denotes the max-pooling function.

Assuming { c i } i = 1 L are the class word embeddings in the dataset D , where L is the number of classes, the sentence embeddings { c i t m p } i = 1 L after manual template nesting can be expressed by equation (14):

{ c i tmp } i = 1 L = WordEmb CLIP ( [ a photo of , { c i } i = 1 L ] ) ,

(14)

where WordEmb CLIP ( ⋅ ) denotes CLIP’s word embedding method, and [ ; ] denotes concatenation.

{ P i } i = 1 L = VAIPL μ , ϵ ( { c i tmp } i = 1 L , Z image ) = G ϵ ( TF μ ( Maxpooling ( { c i tmp } i = 1 L ) , Z image ) )

(15)

where Z image = V M ( X M − 1 ) .

3.2.3. Multimodal Coupling Prompt Learning CLIP (MCPL-CLIP)

We propose a two-stage MCPL-CLIP model to enhance CLIP’s performance in image classification tasks. The MCPL-CLIP structure, shown in Figure 3, includes two stages: the THIPL method and the VAIPL method. The THIPL method introduces learnable text prompt vectors into the text encoder’s various transformer layers and constructs a PMS to inject text prompt features into the visual encoder’s corresponding transformer layers, leveraging the semantic information of the text encoder to build visual prompt features. On this basis, the VAIPL method further integrates text embeddings with the visual encoder output of the text-semantic hierarchy injection prompt framework through cross-attention, constructing text prompt features for each image.

MCPL-CLIP essentially embodies a bidirectional information flow prompt design strategy. It injects semantic features from the text encoder into the visual encoder to construct visual prompts and incorporates feedback from the visual encoder to generate text prompts, achieving bidirectional mapping and joint optimization of text and visual prompts. This ultimately improves MCPL-CLIP’s classification accuracy and robustness across various fine-grained image classification scenarios, such as cross-dataset transfer, domain generalization, and base-to-novel class scenarios.

OPEN IN VIEWER

Figure 3.

Structure of MCPL-CLIP. MCPL-CLIP includes THIPL with text prompt vectors, visual prompt vectors, PMSs, self-attention layers, and VAIPL with transformer decoder structures, max-pooling functions, and nonlinear projection structures. Point-wise adding represents the word embedding method, and Patch Embed represents the image patch embedding method. Note. MCPL-CLIP = multimodal coupling prompt learning contrastive language-image pretraining; THIPL=text-semantic hierarchical injection prompt learning; VAIPL = visual-semantic attention interactive prompt learning; PMS = prompt mapping structure.

MCPL-CLIP comprises the following parts:

VAIPL Method. The VAIPL method can be defined by equation (16):

VAIPL μ , ϵ = [ Maxpooling ( ⋅ ) , TF μ ( ⋅ , ⋅ ) , G ϵ ( ⋅ ) ] ,

(16)

where TF θ ( ⋅ , ⋅ ) represents the cross-attention structure defined by training parameters θ , and G η ( ⋅ ) represents the projection structure defined by training parameters η . The cross-attention structure in VAIPL consists of six transformer decoder blocks, each with eight attention heads. The nonlinear projector structure comprises two multilayer perceptron structures linear-ReLU-linear with a hidden layer of 1024 dimensions.

PMS. The PMS includes multilayer perceptrons with ReLU activation functions as the PMS F ϕ ( ⋅ ) , defined by equation (17):

F ϕ ( ⋅ ) = Bottleneck ( MLP ϕ de-dimension ( ⋅ ) , ReLU ( ⋅ ) , MLP ϕ up-dimension ( ⋅ ) ) .

(17)

PMS consists of two bottleneck multilayer perceptron structures (linear-ReLU-linear) with a hidden layer of 128 dimensions.

Text Prompt Vectors & Visual Prompt Vectors. Text prompt vectors { P i ∈ R d } i = 1 b are transformed into image-semantic-compatible visual prompt vectors through the PMS F ϕ ( ⋅ ) , specifically expressed by equation (18):

{ P ~ i ∈ R d } i = 1 b = F ϕ ( { P i ∈ R d } i = 1 b ) .

(18)

The main difference between element-wise addition and multiplication is in how the information is integrated. While multiplication strengthens the interaction between text features and prompts, it may distort the original semantics of the text. In contrast, element-wise addition smoothly combines the prompt and text features, preserving their original meaning. Therefore, we use element-wise addition.

Coupling Visual Encoder. Based on the original CLIP visual encoder, we introduce visual prompt vectors { P ′ i ∈ R d } i = 1 b = F ( { P i ∈ R d } i = 1 b ) through the PMS F ϕ ( ⋅ ) into several transformer layers. Assuming the original CLIP visual encoder is denoted by V origin ( ⋅ ) , the modified coupled visual encoder V ( ⋅ ) can be expressed by equation (19):

V ( ⋅ ) = [ V i ( ⋅ ) , V j origin ( ⋅ ) ] , i = 1 , 2 , … , K , j = K + 1 , K + 2 , … , M ,

(19)

where V i ( ⋅ ) , i = 1 , 2 , … , K denotes the modified visual encoder’s first K layers, and [ ⋅ , ⋅ ] denotes concatenation. Assuming x is an image sample in the dataset D , the final image feature output Z image can be expressed by equation (20):

Z image = V M ( X M − 1 ) ,

(20)

where X i , i = 1 , 2 , … , M − 1 denotes the image features of sample x after passing through each layer of the visual encoder V ( ⋅ ) .

Semantic Interaction Prompt Features. Semantic interaction prompt features { P i ′ } i = 1 L can be expressed by equation (21):

{ P i ′ } i = 1 L = VAIPL μ , ϵ ( { c i tmp } i = 1 L , Z image ) = G ϵ ( T F μ ( Maxpooling ( { c i tmp } i = 1 L ) , Z image ) ) ,

(21)

where Z image = V M ( X M − 1 ) .

Coupling Text Encoder. According to Section 4.2.1, we modify several transformer layers of the original CLIP text encoder to introduce learnable text prompt vectors { P i ∈ R d } i = 1 b for learning the contextual features of the text encoder. Assuming the original CLIP text encoder is denoted by T origin ( ⋅ ) , the modified coupled text encoder T ( ⋅ ) can be expressed by equation (22):

T ( ⋅ ) = [ T i ( ⋅ ) , T j origin ( ⋅ ) ] , i = 1 , 2 , … , K , j = K + 1 , K + 2 , … , M ,

(22)

where [ ⋅ , ⋅ ] denotes concatenation. Finally, the semantic interaction prompt features { P i ′ } i = 1 L and the text encoder’s output features Z text are summed to form the final text features Z ~ text . Z ~ text can be expressed by equation (23):

Z ~ text = { P i ′ } i = 1 L + Z text ,

(23)

where Z text = T M ( W M − 1 ) .

Loss Function. The optimization objective of MCPL-CLIP is to minimize the standard classification loss function, using cross-entropy loss as the loss function to maximize the score from the true label. Assuming the dataset D has L categories, x is an image sample in the dataset D , and y is the true label, the loss function Loss ( y ∣ x ) can be expressed by equation (24):

Loss ( y ∣ x ) = exp ( sim ( V ( x ) , Z ~ text ( i = y ) ) ) ∑ i = 1 L exp ( sim ( V ( x ) , Z ~ text ) ) ,

(24)

where sim ( ⋅ , ⋅ ) denotes the semantic similarity function. During training, gradients will backpropagate and update VAIPL μ , ϵ , text prompt vectors { P i ∈ R d } i = 1 b , and the PMS F ϕ ( ⋅ ) , while the text encoder T ( ⋅ ) and visual encoder V ( ⋅ ) remain frozen.

4.1. Baselines and Datasets

The baselines include CLIP, CoOp, CoCoOp, and MaPLe. CLIP uses manually designed templates as text prompts; CoOp optimizes only learnable context vectors as text prompt features; CoCoOp uses a lightweight neural network to learn an instance-level image input token and adds it to the text encoder’s prompt embeddings; MaPLe introduces learnable context vectors in the text encoder and maps them to the visual encoder.

We use a total of 15 image classification datasets (11 image classification datasets and four ImageNet variant datasets) for evaluation. For base-to-new class generalization and cross-dataset transfer test scenarios, we evaluate the proposed method on 11 image classification datasets. These datasets cover a wide range of recognition tasks, including two general object datasets, ImageNet and Caltech101; five fine-grained datasets, Oxford Pets, Stanford Cars, Flowers102, Food101, and FGVC Aircraft; scene recognition dataset SUN397; action recognition dataset UCF101; texture dataset DTD; and satellite image dataset EuroSAT.

For the cross-dataset transfer experiment and domain generalization experiment, MCPL-CLIP is trained on the ImageNet dataset and tested on 10 image classification datasets and four ImageNet variant datasets. Detailed dataset statistics are shown in Table 1.

For the visual encoder, we use ViT-B/16 as the backbone structure, and for the text encoder, we use the original CLIP text encoder. During training, only the training parameters of VAIPL μ , ϵ , text prompt vectors { P i ∈ R d b } i = 1 L , and PMS F φ ( ⋅ ) are updated, while the text encoder T ( ⋅ ) and visual encoder V ( ⋅ ) are frozen.

All experiments use a few-shot training strategy, randomly selecting 16 samples per category. The prompt embedding depth K is set to 8, and the lengths of text prompt vectors { P i ∈ R d b } i = 1 L and visual prompt vectors { P ~ i ∈ R d b } i = 1 L are set to 2. Due to graphics processing unit (GPU) memory limitations, the batch size is set to 1.

Both MCPL-CLIP and benchmark models are trained on a single NVIDIA 3090 GPU for two epochs. The training process is optimized using the stochastic gradient descent optimizer with a learning rate of 0.0035. The training and test datasets are run three times, and the average result is taken as the final result.

4.3. Cross-Dataset Transfer Experiment

This experiment focuses on verifying the image classification performance of the model on new data distributions different from the training data distribution. This section evaluates the robustness of the proposed method on out-of-distribution datasets. The emphasis is on the model’s ability to transfer from one dataset to another, highlighting the model’s adaptability to different data sources and potential data variations. Consistent with CoCoOp (Zhou et al., 2022a), the proposed MCPL-CLIP is trained in a few-shot manner on all 1000 classes of the ImageNet dataset.

Table 2(a) shows the experimental results of MCPL-CLIP transferring to 10 datasets. It can be seen that MCPL-CLIP outperforms other benchmark models on multiple datasets. Overall, MCPL-CLIP achieves the highest average accuracy of 67.23%, surpassing other baseline models.

MCPL-CLIP can capture image-region-to-text correlation features that adapt to new datasets during the transfer to new datasets, which is crucial for handling complex image classification tasks. For example, in fine-grained image classification datasets such as StanfordCars and Flowers102, where the model needs to understand and distinguish highly similar categories, MCPL-CLIP effectively utilizes its cross-modal prompts to capture these nuances. Additionally, VAIPL further enhances MCPL-CLIP’s cross-modal alignment capability. By applying cross-attention between instance-level image features from the visual encoder and text input embeddings, and generating weighted text features through nonlinear projection, VAIPL allows MCPL-CLIP to identify and extract features beneficial for cross-modal alignment when learning from specific data distributions. This might explain MCPL-CLIP’s excellent performance on datasets such as EuroSAT (satellite image dataset) and UCF101 (action recognition dataset), as these datasets require the model to identify complex, task-relevant patterns from visual features.

However, in some datasets such as Aircraft, although MCPL-CLIP’s performance has improved, the improvement is relatively small. This might be due to the high similarity in visual features of aircraft in the Aircraft dataset, posing a greater challenge to the model’s discrimination ability. While MCPL-CLIP helps the model identify features beneficial for cross-modal alignment, distinguishing such highly similar visual features may require more refined model adjustments or specially designed feature extraction strategies. The Caltech 101 dataset contains 101 different object categories, with each category having several dozen images. The images exhibit variations in background and shooting angles. CoCoOp may be better at extracting stable features from images of objects with multiple categories and angles, which gives it an advantage on datasets such as Caltech 101. In contrast, MCPL-CLIP might not achieve optimal performance when handling these features.

The design concept of MCPL-CLIP is to inject textual features into the visual encoder and enhance the text prompts through a feedback mechanism. While this may help improve performance on specific datasets, this feedback mechanism could introduce additional noise in cross-dataset transfer tasks, thereby affecting generalization performance. Moreover, the complex prompt coupling in the MCPL-CLIP model may not be as stable and effective across different datasets as MaPLe and CoCoOp, as the complex feedback mechanism may be inefficient when handling domain shifts.

This experiment focuses on verifying the model’s ability to transfer from one dataset to another, highlighting the model’s adaptability to different data sources and potential data variations. The performance of MCPL-CLIP is tested on four ImageNet variant datasets.

Table 3 shows the experimental results of MCPL-CLIP in the domain generalization evaluation scenario. MCPL-CLIP performs comparably to other benchmark models on the ImageNet training set, showing good baseline performance. Overall, MCPL-CLIP achieves the best performance with an average accuracy of 61.06% across the four domain generalization datasets.

On the ImageNetV2 dataset, which has a slight distribution shift, MCPL-CLIP achieves the highest accuracy of 65.18% and a Macro F1 score of 0.689, indicating that it can better maintain its performance when dealing with slight distribution changes. ImageNet-S (Sketch) contains sketch images with styles significantly different from the original ImageNet images. Although MCPL-CLIP performs slightly lower than MaPLe on this dataset, it still outperforms other models, demonstrating its robustness to significant style differences in data.

ImageNet-A (Adversarial) contains difficult-to-classify image samples from natural images. MCPL-CLIP achieves the best accuracy of 51.97% and a Macro F1 score of 0.526, showing its strong adaptability and generalization ability to challenging samples. ImageNet-R (Rendition) includes artistic renditions of various images. The experimental results of MCPL-CLIP again lead, further proving the effectiveness of its cross-modal prompt and interaction mechanism in handling highly diverse image style data samples.

From the provided experimental results, it can be seen that MCPL-CLIP shows varying performance in domain generalization tasks compared to other baseline models. MCPL-CLIP’s multimodal coupling mechanism may cause feature combinations from the original data to lose effectiveness on stylized data. For stylized images, simpler feature extraction (e.g., lines and shapes) may be more effective than complex bidirectional information flow. This is why MCPL-CLIP underperforms MaPLe on the ImageNet-S dataset in domain generalization tasks, where the model needs strong feature abstraction and recognition despite domain shifts and reduced details. The complexity of MCPL-CLIP may not always be beneficial in such cases, suggesting that optimizations and adaptations are needed for stylized datasets to improve their domain generalization and robustness.

Table 4 shows the experimental results of MCPL-CLIP in base-to-new class generalization within the same data distribution. It can be seen that MCPL-CLIP maintains good generalization ability for unseen classes while ensuring the accuracy of base classes, with the harmonic mean (HM) score outperforming other benchmark models on most datasets. This indicates that MCPL-CLIP can capture and utilize cross-modal features to construct more effective and enriched cross-modal prompt features, providing the model with the ability to understand and adapt to unseen classes.

OPEN IN VIEWER

Table 4.

Comparison of MCPL-CLIP and Baselines in Base-to-new Generalization Settings.

(a) Average results of MCPL-CLIP and baselines on 11 datasets
Model	Base Acc	New Acc.	HM
CLIP	69.34	74.22	71.70
CoOp	82.69	63.22	71.66
CoCoOp	80.47	71.69	75.83
MaPLe	82.28	75.14	78.55
MCPL-CLIP	83.40	75.98	79.25
(b) ImageNet
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	72.43	–	68.14	–	70.22
CoOp	76.47	0.752	67.88	0.663	71.92
CoCoOp	75.98	0.743	70.43	0.691	73.10
MaPLe	76.66	0.761	70.54	0.702	73.47
MCPL-CLIP	77.47	0.773	71.58	0.714	74.41
(c) Caltech101
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	96.84	–	94.00	–	95.40
CoOp	98.20	0.979	89.81	0.899	93.73
CoCoOp	97.96	0.965	93.81	0.937	95.84
MaPLe	97.74	0.956	94.36	0.943	95.02
RAPL-CLIP	98.34	0.963	95.21	0.951	96.75
MCPL-CLIP	98.76	0.989	95.96	0.957	97.34
(d) OxfordPets
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	91.17	–	97.26	–	94.12
CoOp	93.67	0.935	95.29	0.952	94.47
CoCoOp	95.20	0.951	97.69	0.975	96.43
MaPLe	95.43	0.943	97.76	0.976	96.58
RAPL-CLIP	95.62	0.957	97.94	0.977	96.77
MCPL-CLIP	96.27	0.973	97.92	0.983	97.09
(e) StanfordCars
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	63.37	–	74.89	–	68.65
CoOp	78.12	0.780	70.00	0.601	73.64
CoCoOp	70.49	0.702	73.59	0.735	72.01
MaPLe	72.94	0.715	74.00	0.736	73.47
RAPL-CLIP	74.00	0.737	74.95	0.750	74.67
MCPL-CLIP	74.24	0.737	74.87	0.748	74.55
(f) Flowers102
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	72.08	–	77.50	–	74.83
CoOp	97.60	0.975	59.47	0.574	74.06
CoCoOp	85.23	0.889	65.90	0.645	74.31
MaPLe	95.92	0.937	64.89	0.601	77.39
RAPL-CLIP	97.08	0.968	73.37	0.737	83.55
MCPL-CLIP	97.01	0.968	73.49	0.737	83.55
(g) Food101
(h) FGVCAircraft
CLIP	90.10	–	91.22	–	90.66
CoOp	88.33	0.882	82.26	0.821	85.19
CoCoOp	90.70	0.907	91.29	0.910	90.99
MaPLe	90.71	0.897	92.05	0.918	91.38
RAPL-CLIP	90.74	0.907	91.65	0.917	91.19
MCPL-CLIP	91.38	0.926	92.53	0.934	91.95
CLIP	27.19	–	36.29	–	31.09
CoOp	40.44	0.401	22.30	0.222	28.75
CoCoOp	33.41	0.334	23.71	0.235	27.74
MaPLe	37.44	0.365	35.61	0.353	36.50
RAPL-CLIP	36.82	0.361	34.11	0.343	35.41
MCPL-CLIP	39.76	0.396	36.48	0.374	38.05
(i) SUN397
Model	Base Acc.	Base mF1	New Acc.	New mF1	HMCLIP	69.36	–	75.35	–	72.23
CoOp	80.60	0.795	65.89	0.624	72.51
CoCoOp	79.74	0.773	76.86	0.742	78.27
MaPLe	80.82	0.781	78.70	0.763	79.75
MCPL-CLIP	81.95	0.822	79.98	0.781	80.95
(j) DTD
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	53.24	–	59.90	–	56.37
CoOp	79.44	0.787	41.18	0.415	54.24
CoCoOp	77.01	0.767	56.00	0.558	64.85
MaPLe	80.36	0.791	59.18	0.589	68.16
RAPL-CLIP	81.18	0.812	56.84	0.568	66.86
MCPL-CLIP	81.06	0.807	58.91	0.587	68.23
(k) EuroSAT
Model	Base Acc.	Base mF1	New Acc.	New mF1	HM
CLIP	56.48	–	64.05	–	60.03
CoOp	92.19	0.919	54.74	0.545	68.69
CoCoOp	87.49	0.874	60.04	0.597	71.21
MaPLe	94.07	0.939	73.23	0.727	82.35
RAPL-CLIP	90.07	0.898	71.38	0.712	73.16
MCPL-CLIP	94.46	0.951	74.65	0.740	83.39
(l) UCF101
Model	Base Acc.	Base mF1	New Acc.	New mF1	HMCLIP	70.53	–	77.50	–	73.85
CoOp	84.69	0.839	56.05	0.545	67.46
CoCoOp	82.33	0.821	73.45	0.717	77.64
MaPLe	83.00	0.827	78.66	0.768	80.77
MCPL-CLIP	85.05	0.847	79.43	0.784	82.14

Note. MCPL-CLIP = multimodal coupling prompt learning contrastive language-image pretraining.

On general image classification datasets such as ImageNet and Caltech101, MCPL-CLIP achieves the best results in base class accuracy, new class accuracy, and HM. This proves that MCPL-CLIP can effectively leverage the knowledge learned during the pretraining stage and apply it to new, unseen classes through vision-language coupled interaction prompt learning. On fine-grained datasets such as StanfordCars and Flowers102, MCPL-CLIP’s performance still leads other benchmark models, indicating that MCPL-CLIP is more advantageous in handling data samples with subtle differences in categories. On other datasets such as satellite images (EuroSAT), scene understanding (SUN397), and action recognition (UCF101), MCPL-CLIP also generally leads, demonstrating its robust visual-language alignment and generalization capabilities.

MCPL-CLIP couples text and visual features through bidirectional information flow, a complex mechanism that may not effectively generalize the fine-grained features required when handling new classes, especially in tasks that demand capturing subtle differences. In tasks that require fine-grained feature extraction and recognition (such as Stanford Cars), MCPL-CLIP may not be as effective as some models specifically optimized for such tasks, such as CoOp. The bidirectional information flow and complex prompt design may introduce interference or noise in certain cases, leading to insufficient extraction and generalization of features for new classes.

5.1. Conclusion

The main research work of this paper includes:

(1)

Identification of feature extraction bias and domain shift issues in CLIP’s visual encoder between pretraining and fine-tuning phases. To address these, we propose a two-stage MCPL framework that synergistically constructs and optimizes text-visual prompts through cross-modal interactions. The first stage employs text-semantic hierarchical injection to embed learnable prompts from the text encoder into visual transformer layers. The second stage enhances cross-modal alignment via visual-semantic attention interaction, generating instance-aware text prompts.

(2)

This paper proposes MCPL-CLIP to improve CLIP’s performance in image classification and generalization tasks. Experiments demonstrate that MCPL-CLIP achieves varying degrees of performance improvement over multiple baseline models such as CLIP, CoOp, CoCoOp, and MaPLe in cross-dataset transfer, domain generalization, and base-to-new class generalization settings. For example, MCPL-CLIP leads by 3.35% to 0.93% in cross-dataset transfer scenarios, 3.88% to 0.78% in domain generalization scenarios, and 7.55% to 0.70% in base-to-new class generalization scenarios. This indicates that compared to using single-modal features to construct prompts, the paradigm of constructing prompts using multimodal interactive information in MCPL-CLIP is more effective for image classification tasks and their generalization scenarios.

5.2. Future Work

Through systematic analysis of experimental results, we identify the following key limitations of the MCPL-CLIP approach: (1) performance improvement remains limited on datasets with highly similar visual features (e.g., Aircraft) due to insufficient sensitivity to fine-grained differences; (2) cross-modal feedback mechanisms may introduce noise interference in complex domain transfer tasks, leading to suboptimal performance on datasets with multiperspective variations such as Caltech101; (3) the multimodal coupling structure shows limited adaptability to stylized data (e.g., ImageNet-S sketches), where bidirectional information flow becomes less efficient in simplified feature extraction scenarios; (4) computational cost and resource consumption are higher than baseline models, restricting deployment in resource-constrained environments; and (5) complex prompt designs may induce feature interference in fine-grained classification tasks (e.g., Stanford Cars), compromising novel class generalization. To address these findings, future work will focus on: (1)

Fine-Grained Feature Enhancement

Developing dynamic attention-focusing mechanisms with differentiable feature masks to strengthen learning of local discriminative regions, particularly in aerospace recognition scenarios requiring subclass differentiation.