2.1. Multimodal Sentiment Analysis

In the early stages of affective computing, research mainly focused on single-modal tasks, studying affective computing from textual, audio, and visual perspectives. However, a single modality cannot effectively capture semantic conflicts and emotional complexity in emotional expressions, such as irony and metaphors, especially when dealing with complex emotional expressions, which greatly limits the performance of sentiment recognition models.

With the rapid development of social media and multimedia technologies, research on sentiment analysis has gradually shifted toward multimodal data analysis. Compared to a single modality, multimodal data can provide richer sentiment information, and its core goal is to improve the accuracy and robustness of sentiment analysis by effectively fusing information from multiple modalities, such as text, images, and audio. Multimodal sentiment computing aims to develop models that interpret and explain sentiments and emotional states across multiple modalities. Truong and Lauw ⁹ proposed the VistaNet model, which significantly improved the performance of sentiment analysis by combining textual and visual information and highlighting important aspects of entities. This approach provides new ideas for MSA, especially when dealing with complex sentiments, such as sarcasm and metaphors, where the complementary nature of images and text can effectively compensate for the lack of a single modality.

Early research on MSA primarily used simple feature-splicing methods, such as directly combining text and image features for sentiment classification. Pranesh and Shekhar ¹⁰ proposed a Memesem model for MSA through transfer learning that used VGG19 and BERT to extract visual and verbal features, respectively, and then connected these two features to generate a multimodal vector representation. Although this approach is simple and easy to implement, it ignores the interactions and complementary relationships between modalities, resulting in limited performance enhancement for sentiment analysis. Researchers have proposed various improved fusion methods to achieve full interaction and fusion between modalities. Cheng et al. ¹¹ proposed extracting unimodal temporal features through an attention temporal convolutional network and fusing them based on the correlation of different features using a multilayer feature fusion model. Zhou et al. ¹² proposed a cross-attention and hybrid feature-weighting network to make full use of the complementary information between images and emotion recognition within contextual features. Li et al. ¹³ proposed a multilayer fusion model for aligning and fusing token-level features of aligned text and images and designed two comparative learning tasks to help the model learn emotion-related features in multimodal data.

With the advancements in deep learning, researchers have increasingly adopted deep neural networks for MSA ¹⁴. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have become foundational architectures for such tasks. Rehman et al. ¹⁵ introduced a hybrid CNN-LSTM framework, leveraging CNN’s local feature extraction and LSTM’s sequential dependency modeling to enhance sentiment prediction accuracy. Durga and Godavarthi ¹⁶ devised an aspect-prioritized sentiment analysis system using a decision-based RNN that achieved superior classification performance through cross-modal attention mechanisms.

In MSA, researchers have extensively explored the application of attention mechanisms to the dynamic adjustment of modal weights. Usama et al. ¹⁷ proposed a hybrid model combining RNN and CNN to enhance the model’s focus on key features by extracting features through CNN and calculating the attention scores of the features using the attention mechanism, and Basiri et al. ¹⁸ proposed a bidirectional CNN-RNN deep model based on the attention mechanism, which highlights the key information by weighting the contextual features extracted by bidirectional LSTM and GRU through the attention mechanism.

Existing MSA methods usually rely on large-scale labeled datasets; however, in practical applications, especially when labeling images and texts is costly and difficult to obtain, the performance of these methods is often limited by data scarcity. Therefore, improving the generalization ability and performance of models in data-scarce environments is a major challenge in MSA.

2.2. Prompt-Based Approach to Few-Shot Learning

To address the problem of data scarcity, an FSL method was developed. Few-shot learning significantly mitigates the problem of data scarcity and enhances the accuracy and robustness of sentiment analysis by leveraging common cross-task knowledge and prior experience, thereby allowing the model to quickly adapt to new tasks with a limited number of samples. Recently, with the widespread adoption of large-scale multimodal foundation models, parameter-efficient fine-tuning has become the mainstream paradigm for few-shot adaptations. Three major technical directions have emerged in this field: prompt, adapter, and knowledge-based methods, which provide systematic solutions to the problem of scarce multimodal data ¹⁹. In particular, adapter-based methods achieve efficient cross-modal feature transfers by introducing lightweight modules. For example, the text-driven cross-modal feature fusion adapter (TCFF-Adapter) enables robust and parameter-efficient transfer through cross-modal feature alignment and adapter optimization ²⁰. In contrast, prompt-based methods have demonstrated significant advantages ²¹. Prompt-based methods design prompt templates to reformulate the original input into a structure with unfilled slots, guiding the pre-trained language model (PLM) to perform inference under few-shot conditions, thereby fully leveraging the PLM’s knowledge for sentiment prediction ²².

Prompt learning methods are widely used in textual sentiment analysis. Dong et al. ²³ proposed MetricPrompt, which alleviates the difficulty of descriptor design by framing a few-shot text classification task as a text-pair relevance estimation task. Hosseini-Asl and Liu ²⁴, on the other hand, utilized a generative language model for FSL and proposed a method to perform sentiment analysis tasks by generating language, thus avoiding the need for task-specific training layers. Jian et al. ⁶ applied a supervised contrastive learning framework to few-shot plain-text sentiment analysis and optimized the model performance by enhancing intra-class aggregation and inter-class differentiation.

As the demand for MSA increases, prompt learning methods have been extended to collaborative vision-language modeling. Tsimpoukelli et al. ²⁵ pointed out that current large-scale language models, while capable of processing textual information, face limitations when dealing with other modalities such as images. Therefore, they proposed a frozen method, which encodes images as word embeddings in the word embedding space of a PLM, thus allowing the model to process image information. Zang et al. ²⁶ designed shared initial prompts to simultaneously optimize text and visual encoders in the CLIP model. Yang et al. ⁷ proposed multimodal probabilistic prompt-based prompts for few-shot sentiment analysis, which significantly improved the robustness of the model by providing diverse prompting cues. Yu and Zhang ⁸ proposed a PVLM approach that combines visual information with a PLM, bridging the gap between pre-trained and multimodal sentiment categorization tasks through cue-tuning techniques. Zhou et al. ²⁷ proposed a hybrid cueing model that combined handcrafted and learnable cues by utilizing techniques such as attention mechanisms to improve model performance.

The prompt-based FSL approach shows great potential in MSA, effectively alleviating the data scarcity problem and significantly improving model performance. However, when passing images directly to the language model, due to the decoupling of the image encoder and the language model, the image information cannot be fully integrated into the text, resulting in discrepancies between modalities, which may affect the accuracy of sentiment analysis. Moreover, in FSL tasks, increasing the diversity and consistency of input data through effective data augmentation and optimizing the clustering of inputs within the same category in the feature space remain key challenges for improving the robustness and discriminative ability of sentiment analysis models. In addition, the choice of the learning rate significantly affects the convergence of the model in FSL, and selecting the appropriate learning rate to ensure stable convergence in different scenarios remains a pressing issue in current research.

4.1. Multimodal Feature Alignment and Projection

To align the visual and linguistic modalities in the same vector space, we used a pre-trained BLIP model to generate natural language descriptions, converting visual information into textual semantics:

First, the image features were extracted using the NF-ResNet50 model, which was designed to capture a global visual representation of the image. The global visual feature vector was then obtained through a pooling operation:

Second, a learnable linear mapping matrix \(W_{p}\) and a bias vector \(b_{p}\) were introduced to align the visual features with text embedding. The global visual features \(h_{v}\) of the image were projected onto a space aligned with the text embedding as follows:

Next, the projected feature \(h'_{v}\) is reconstructed into \(N\) independent pseudo-visual tokens whose structural form is consistent with textual word embeddings:

Subsequently, the textual description \(C_{\mathrm{cap}}\) of the image, the pseudo-visual markers \(\tilde{V}\), the original textual sentence \(S\), and the aspect word \(A_{i}\) (if any) were co-assembled into a prompt template, which was input into the model, effectively fusing the image and textual information.

4.2. Prompt Template Construction

To effectively integrate visual and textual information in MSA tasks, this study designed two structured prompt templates for sentence- and aspect-level tasks. These templates were aimed at adapting to different task requirements and minimizing the differences between modalities.

For sentence-level sentiment analysis, the multimodal prompt templates used in this study are as follows:

For the aspect-level sentiment analysis task, aspect words \(A_{i}\) were introduced into the prompt template to provide explicit semantic constraints:

The “for \(A_{i}\)” clause constrains the sentence to a specific aspect (e.g., “screen size”), guiding the model to focus on relevant regions and ignore irrelevant ones. These uniformly processed multimodal prompts were fed into a PLM, which utilized a masked language modeling (MLM) header to predict the location of the \([\mathrm{MASK}]\) marker.

4.3. Prompt-Based Image-Text Contrastive Learning

To enhance the robustness and generalizability of the model in FSL scenarios, this study proposed a contrastive learning framework based on prompt learning. The framework incorporates contrastive learning for both text and image modalities and aims to enhance the model’s ability to capture emotional features through multiview enhancement and data augmentation techniques.

Contrastive learning for the textual modality improves the discriminative ability of the model by bringing the representations of similar samples closer together and pushing the representations of dissimilar samples further apart. Specifically, the model employs different prompt templates to generate two different perspectives of text input: \(X_{1}\) and \(X_{2}\). For example, given the original input text \(S_{0}\), the first perspective \(X_{1}\) can be represented as

In this study, we used a different prompt template \(\mathit{temp}_{j}\) from \(X_{1}\) to generate a second perspective \(X_{2}\), ensuring that it differs from \(X_{1}\) in terms of syntactic structure and contextual expression while maintaining semantic consistency:

Positive example pairs include sentences from the same sentiment category (e.g., \(S_0\) and \(S_2\)), or different prompt templates applied to the same input text (e.g., \(S_0\) and \(S_1\)). Negative example pairs consist of sentences from different sentiment categories (e.g., \(S_0\) and \(S_3\)). Through contrastive learning, the model clustered texts of the same category in the feature space and separates them from different categories, thereby effectively learning sentiment-related textual features.

In contrastive learning for image modality, emotionally relevant features should remain consistent across transformations because the meaning that the user intends to convey should not change with the image. Inspired by automatic enhancement ²⁸, in this study, we applied random enhancement operations (e.g., rotation, cropping, flipping, and brightness adjustment) to an original image (e.g., \(V_{0}\)) to generate an enhanced version of the image (e.g., \(V_{1}\)). Positive example pairs consisted of images or enhanced image pairs (e.g., \(V_{0}\) and \(V_{1}\)) with the same emotion label, whereas negative example pairs consisted of images with different emotion labels. Through contrastive learning, the model focuses on emotionally relevant features in an image that remain unchanged during data enhancement (e.g., character expressions). Simultaneously, the model learns to ignore surface features that are not emotionally relevant (e.g., crop position and brightness changes). This strategy helped the model focus on emotionally relevant core features and improves emotion classification accuracy.

Throughout the training process, the total loss of the model comprises the following components: masked language model loss \(L_{\mathrm{MLM}}\), text contrast loss \(L_{\mathrm{text\textrm{-}SupCon}}\) for FSL, and image contrast loss \(L_{\mathrm{img\textrm{-}SupCon}}\) for prompt-based FSL, as follows:

4.4. Dynamic Adjustment of Learning Rates

In an FSL task, a fixed learning rate is often inadequate for adapting to the different stages of training. This limitation may lead to slow convergence in the early stages, and oscillation or overfitting in the later stages. To address this issue, this study proposed a dynamic learning rate adjustment mechanism based on the long short-term memory (LSTM) network. This mechanism predicts the learning rate at each time step and flexibly adjusts the step size of the gradient descent to accommodate dynamic changes in the gradient of the model. This accelerated the convergence of the model and improved its final performance.

In traditional gradient descent optimization, the update of model parameters follows the rules:

Based on this motivation, this study designed an LSTM-based learning rate prediction model as follows:

To prevent the predicted learning rate from becoming excessively large or small, this work applied the \(\mathrm{Tanh}\) activation function to constrain the value of \(\tilde{\alpha}_{t}\):

This constraint ensures that the predicted value is mapped within the range \([-1, 1]\). Subsequently, a linear scaling mechanism limits the final learning rate to a reasonable range of \([1 \times 10^{-5}, 1 \times 10^{-1}]\), avoiding the instability caused by excessively large learning rates or slow convergence owing to overly small learning rates. This adaptive mechanism maintains parameter stability while improving the convergence efficiency in few-shot scenarios.

In this study, \(\tilde{\alpha}_{t}\) was used as the dynamic learning rate to update the model parameters at each time step, following the rule:

However, dynamically adjusting the learning rate may lead to gradient explosion issues, particularly during backpropagation. To address this, we introduced gradient clipping before each gradient update to constrain the gradient magnitude, as shown below:

4.5. Classification

In the multimodal sentiment categorization task, sentiment category prediction was achieved by establishing a mapping relationship between the sentiment label space \(Y\) and vocabulary list \(V\) of the pre-trained masked language model (MLM). Taking aspect-level sentiment analysis as an example, given the input \(D = (S_{i}, V_{i}, A_{i})\), the model accomplished the sentiment categorization task by calculating the conditional probability \(y \in Y\) for each category.

This conditional probability \(p(y \mid S, V, A)\) is expressed by the following equation:

5.1. Datasets

We conducted experiments on five publicly available MSA datasets to validate the accuracy of the model in predicting sentiments.

For the sentence-level task, two datasets, MVSA-Single ²⁹ and TumEmo ³⁰, were used. MVSA-Single is derived from Twitter and includes a tag set of Positive, Neutral, and Negative tags. TumEmo is a dataset from Tumblr with the tag sets angry, bored, calm, fearful, happy, love, and sad.

For the aspect-level task, three datasets, Twitter-15, Twitter-17 ³¹, and MASAD ³², were selected for testing. The Twitter-15 and Twitter-17 datasets were sourced from Twitter with a tag set of Positive, Negative, and Neutral. MASAD is a large-scale aspect-level sentiment analysis dataset from Flickr with a tag set of Positive and Negative.

To ensure the fairness and comparability of the experimental results, this study followed the data preprocessing and partitioning strategy used in previous research ⁸. For datasets with predefined splits (such as Twitter-15 and Twitter-17), the original partitions were adopted directly. Datasets without official splits (such as MASAD, MVSA-Single, and TumEmo) were randomly divided into training, validation, and test sets at a ratio of \(8:1:1\).

During the preprocessing stage, systematic data cleaning and normalization were performed, including the removal of noisy textual information and redundant symbols as well as the filtering of overly short samples. In the few-shot experiments, approximately 1% of the samples were randomly selected from the training set of each dataset to construct the few-shot training subset while ensuring a balanced distribution across sentiment categories to avoid class imbalance bias. The sizes of the validation and test sets were kept unchanged to maintain a consistent and comparable model evaluation. Detailed data distributions are presented in Table 3.

5.2. Experimental Setup

In this study, following the PVLM framework ⁸, we integrated the BERT-based and NF-ResNet-50 to construct an MSA model. To capture visual semantic information more effectively, the BLIP model was introduced to generate semantic descriptions from images and associate them with textual prompts, thereby enhancing the model’s understanding of emotional visual features.

During the model-training process, the batch size was adjusted according to the requirements of the different datasets. Specifically, the batch size was set to 32 for the MVSA-Single, Twitter-15, Twitter-17, and MASAD datasets, while it was set to 16 for the TumEmo dataset. The initial learning rate and number of pseudo-visual tokens were optimized using a grid search method. The initial learning rate was selected from the set \(\{\mbox{1e-5}, \mbox{2e-5}, \mbox{3e-5}, \mbox{4e-5}, \mbox{5e-5}\}\), and the number of pseudo-visual tokens was chosen from \(\{1, 2, 3, 4, 5\}\). During training, both coefficients \(\alpha\) and \(\beta\) in Eq. \(\eqref{eq:9}\) were set to 1.0. For each set of hyperparameter configurations, three experiments were conducted with different random seeds, and the mean and standard deviation are reported as the final results to ensure the robustness of the experimental findings.

The evaluation of the model adopted two widely used metrics in sentiment analysis: accuracy (Acc) and macro-averaged F1 score (Mac-F1), where higher values indicate better model performance. To maintain consistency in the sentiment label system, label mappings were unified across datasets. Specifically, for the Twitter-15, Twitter-17, and MVSA datasets, the original labels \(\{\textrm{negative}, \textrm{neutral}, \textrm{positive}\}\) are mapped to \(\{\textrm{bad}, \textrm{no}, \textrm{good}\}\); for the MASAD dataset, the labels \(\{\textrm{negative}, \textrm{positive}\}\) are mapped to \(\{\textrm{bad}, \textrm{good}\}\); and for the TumEmo dataset, the original sentiment labels are directly used for training.

5.3. Baseline Model

To comprehensively evaluate the performance of the models presented in this study, a variety of existing representative methods were selected as comparison baselines, covering two major categories of models: textual unimodal and multimodal, as outlined below:

1. (1)

   BERT ³³ utilizes a bidirectional transformer architecture to pre-train a large-scale unlabeled corpus to obtain a deep linguistic representation and adapt it to downstream tasks such as sentiment analysis through simple fine-tuning.

2. (2)

   PT ⁸ serves as an ablated version of PVLM, retaining only textual prompts and directly removing the image modality input.

3. (3)

   BERT+BL ³¹ introduced a bilinear attention mechanism on top of BERT and stacked an additional transformer layer on top to enhance the modeling of sentiment information.

4. (4)

   LM-BFF ³⁴ performs few-shot fine-tuning through natural language prompts with example demonstrations combined with dynamically selected demonstration contexts to effectively enhance the model performance under low-resource conditions.

1. (1)

   Tombert ³⁵ modeled the interaction among text, images, and aspects using an image-aware word representation and entity span prediction mechanism for a sentence-level MSA task.

2. (2)

   MMAP ³² designed a multimodal interaction layer to capture deep semantic connections between text, images, and aspects and introduced adversarial training to align the representation space of different modalities.

3. (3)

   EF-CaTrBERT ³⁶: A fine-grained approach with state-of-the-art results on Twitter-15/17 transforms images into auxiliary sentences in the input space and introduces an object-aware converter with a non-autoregressive text generator to improve the multimodal perception of language models.

4. (4)

   MBERT ³⁵ is based on a unified multimodal transformer structure for sentence-level sentiment analysis and the modeling of associations between cross-modal entities.

5. (5)

   MVAN ³⁰ is a novel MSA model based on a multiview attention network for sentence-level sentiment analysis that utilizes a continuously updated memory network to access the deep semantic features of image text.

6. (6)

   MGNNS ³⁷: A sentence-level multimodal approach with better results on MVSA and TumEmo, designing multichannel graph neural networks for emotion perception and using a graph structure to model the relationship between graph text.

7. (7)

   MultiPoint ⁷ proposes a multimodal probabilistic fusion-prompt method to provide diverse semantic cues for multimodal sentence-level sentiment analysis.

8. (8)

   UP-MPF ³⁸ narrows the semantic gap between modalities through unified vision-language pre-training. During the fine-tuning stage, the visual encoder was frozen to enhance cross-modal sentiment understanding in few-shot scenarios.

9. (9)

   MTVAF ³⁹ effectively bridges the cross-modal semantic gap and suppresses irrelevant visual noise through multi-level text-visual alignment and dynamic attention fusion.

10. (10)

    PVLM ⁸ fuses visual modal information into a linguistic model through a prompting mechanism for text modeling of visual perception, with superior performance in FMSA tasks.

5.4. Main Results

To validate the performance of the DACL-FMSA model in a few-shot multimodal emotion recognition task, comparative experiments were conducted on five datasets. The results of the performance evaluation using the baseline model on the aspect- and sentence-level datasets are presented in Tables 4 and 5. The key observations are as follows.

Compared with the baseline model, DACL-FMSA significantly outperformed the other baseline models in terms of both accuracy and F1 metrics across all datasets. DACL-FMSA demonstrated significant advantages over the baseline model in both types of sentiment analysis tasks. In the aspect-level sentiment analysis task, the accuracy improved by 1.03%, 1.71%, and 4.46% for the Twitter-15, Twitter-17, and MASAD datasets, respectively, and the F1 scores improved by 1.77%, 1.01%, and 4.99%, respectively. In the sentence-level sentiment analysis task, the accuracy on the MVSA-S and TumEmo datasets improved by 4.65% and 3.34%, respectively, corresponding to F1 score improvements of 7.49% and 3.48%, respectively.

Overall, the MSA model shows a significant improvement compared to the unimodal sentiment analysis model, which indicates that the performance advantage of the MSA model arises from its ability to compensate for the incompleteness of unimodal information and contextual ambiguities. Unimodal data, such as text or visual information, typically capture only part of the sentiment dimensions. The semantic complementarity of multimodal data helps to reconstruct a more complete emotional representation space. This feature is critical in few-shot scenarios. Although pure text baseline models can capture deeper semantics through pre-trained language models (e.g., BERT) or prompt fine-tuning techniques (e.g., LM-BFF), their inherent modal limitations make it difficult to parse how visual contexts reinforce or modify emotions. For example, the PT, an ablated version of the PVLM, showed a significant drop in performance after removing the image modality, confirming the necessity of visual cueing information in FSL.

The performance advantage of the DACL-FMSA model compared with other state-of-the-art MSA models arises primarily from the effective combination of its key design strategies. The BLIP-based image semantic description generation strategy transforms visual content into text-aligned contextual prompts, establishes an efficient semantic bridge between modalities, and enhances the model’s understanding of multimodal inputs. Second, a contrastive learning framework is introduced to enhance the distribution of samples in the feature space by constraining the graphic multiview enhancement. This improves the robustness of the model to noise and sample sparsity while enhancing its ability to learn invariant features. Additionally, the LSTM adaptive learning rate adjustment mechanism optimizes the training process by dynamically adjusting the learning rate, eliminating the need for manual parameter tuning at a fixed learning rate. The synergy of the above modules enables DACL-FMSA to exhibit stronger cross-modal alignment and higher classification accuracy in MSA tasks.

5.5. Ablation Experiments

To assess the validity of the design of the key components of the DACL-FMSA model, we evaluated the contributions of the image description generation, text contrast learning, image contrast learning, and dynamic learning rate adjustment modules to the sentiment classification task through an ablation experiment. The results are presented in Table 6.

During the experiment, we sequentially removed the key modules to evaluate their impact on model performance. First, when the image descriptions generated by the BLIP model were removed (without BLIP Cap) and raw image features were used directly, significant performance degradation was observed on the Twitter-15, MVSA-S, and MASAD datasets. This can be attributed to the characteristics of these datasets: Twitter-15 and MVSA-S contain short and noisy texts, whereas MASAD, sourced from an image-sharing platform, is predominantly visual. BLIP-generated captions provide crucial emotional semantic cues that significantly enhance the model’s discriminative capability in few-shot settings. Subsequently, the text-contrastive learning module was removed (without Text CL). The experimental results indicated that this module is particularly important for Twitter-15 and MVSA-S, where the text is noisy and limited in length. By enhancing semantic consistency, Text CL effectively improves the model’s ability to perceive emotional semantics in short texts, thereby compensating for the lack of textual information. When the image contrastive learning module was disabled (without Image CL), the model was most significantly affected on the Twitter-related datasets and MVSA-S. This may be because social media images often contain substantial background noise and style variations. The Image CL module employs contrastive learning to help the model focus on invariant emotional visual features (such as facial expressions), thereby improving robustness and generalization. Finally, removing the dynamic learning rate adjustment module (without Dynamic LR) led to an overall decline in performance. This indicates that the adaptive learning rate strategy effectively balances the convergence speed and training stability under few-shot conditions, thereby enhancing model performance across datasets.

In summary, each module plays an indispensable role in the model’s performance, and their synergistic effects significantly strengthen the model’s ability to integrate and classify multimodal emotional cues.

5.6. Effect of Different Number of Pseudo-Visual Tokens

To determine the optimal number of pseudo-visual tokens, we conducted a grid search experiment. Fig. 3 illustrates the impact of varying the number of pseudo-visual tokens on model performance.

For aspect-level tasks that require fine-grained semantic alignment between text and images, the experimental results showed that both Twitter-15 and MASAD achieved optimal performance at \(k=4\). This indicates that an appropriate number of pseudo-visual tokens can effectively supplement limited contextual information in the text, thereby enhancing the model’s ability to recognize sentiment targets. However, when \(k\) increases beyond this point, redundant visual information introduces noise, leading to performance degradation. By contrast, Twitter-17, which exhibits higher text quality and more consistent annotations, shows lower reliance on visual information. Its performance remained relatively stable for the first three pseudo-visual tokens, began to improve at \(k=4\), and achieved better performance at \(k=5\).

In sentence-level tasks, the performance of MVSA-S decreased as \(k\) increased, suggesting that global sentiment discrimination relies more on a small number of critical visual features. An excessive number of pseudo-visual tokens diminishes the discriminative power of the model. On the other hand, TumEmo shows minimal fluctuations in performance, likely because of its larger sample size and the fact that sentiment prediction in this dataset depends primarily on textual labels with relatively little contribution from visual information. Compared with aspect-level tasks, sentence-level sentiment analysis is inherently coarser in granularity and typically requires only minimal visual cues for prediction.

In summary, aspect-level tasks benefit significantly from a moderate increase in pseudo-visual tokens, whereas sentence-level tasks require fewer tokens to avoid noise interference. The variation in optimal \(k\) across different datasets reflects not only the quality of their textual and visual features, but also the relative contribution of images to the task. It is essential to flexibly choose the number of pseudo-visual tokens according to specific task.

5.7. Training Convergence Analysis

To validate the effectiveness of the LSTM-based dynamic learning rate adjustment method, we compared the training loss curves of the proposed DACL-FMSA model with those of the baseline model (PVLM) using a fixed learning rate. As shown in Fig. 4, the DACL-FMSA model not only achieves a significantly lower final loss value than the baseline but also demonstrates superior convergence behavior. Specifically, our method exhibited a noticeable acceleration in convergence during the early training stage, reaching a loss level comparable to that of the baseline model in approximately half the number of training epochs. Furthermore, the loss curve of DACL-FMSA decreases in a smoother and more stable manner, indicating a faster convergence speed and better convergence performance.

5.8. Impact of Different Training Scales

To further validate the effectiveness of the proposed model in few-shot scenarios, we conducted comparative experiments with different training scales (1%, 2%, and 4%) on Twitter-15 and Twitter-17 datasets. As shown in Table 7, our method outperformed the PVLM baseline at all data proportions, with performance continuously improving as the training data increased while maintaining a leading position, demonstrating good generalization capability and stability. At a training proportion of 4%, the model achieved the maximum performance gain in both accuracy and F1-score.

5.9. Parameter Sensitivity Analysis

To evaluate the model’s robustness to hyperparameters, we conducted sensitivity experiments on the loss function weight coefficients \(\alpha\) and \(\beta\) using the MVSA-S dataset and visualized their impact on performance through a heatmap, as shown in Fig. 5. The results demonstrate that the model maintains stable performance across a wide range of parameter values, with optimal performance achieved when \(\alpha=1\) and \(\beta=1\). When either parameter value becomes excessively large, the overall model performance exhibits some fluctuations, indicating that maintaining a balance among the various loss components is crucial for enhancing MSA performance.