2.1. Emotional Feature Extraction

Facial expression recognition is challenging because different emotional states are conveyed through local fine-grained features and global structural patterns. For instance, happiness is typically indicated by the upward curvature of the corners of the mouth, whereas anger may manifest through the formation of wrinkles on the forehead. Inadequate capturing of either local details or overall facial contours can significantly degrade the accuracy of emotion recognition. Furthermore, emotional speech signals exhibit substantial variability, and analyzing them solely from a single perspective, whether in the time domain or frequency domain, may result in the loss of critical information embedded within the speech signal.

Feature pyramid representations are widely used in the era of handcrafted features ¹⁴. The main advantage of extracting features at each level of the image pyramid is that it produces multi-scale feature representations with strong semantic information at all levels. The single shot detector ¹⁵ was the first to leverage a hierarchical convolutional pyramid structure, enhancing the diversity and richness of feature extraction by reusing multi-scale feature maps. Recently, top-down architectures with skip connections have become popular ¹⁶. These aim to generate high-resolution, high-level feature maps for prediction. Although these maps contained rich semantic information, they often lacked detailed features. To capture fine details while supplementing the global context, multi-scale feature pyramids ¹⁷ were constructed by fusing features of different resolutions through pathway connections. Convolutional neural networks (CNNs) have become the backbone of deep learning models for visual recognition. The standard CNN was first introduced in ¹⁸ to distinguish handwritten digits. Souza et al. ¹⁹ proposed a lightweight network architecture called RS-Xception that facilitates deployment on embedded devices for facial emotion recognition. ResNet ²⁰ introduced skip connections within the convolutional blocks, enabling the training of very deep networks and achieving impressive results.

The application of Transformers in computer vision, particularly in image classification and detection, has become an important research topic. The innovation of image tokenization in ViT ²¹ has sparked widespread interest in the utilization of inherent attention mechanisms. A multi-scale Transformer (MViT) has been proposed, which combines Transformer with multi-scale feature hierarchies by hierarchically expanding the channel capacity and reducing the spatial resolution ²². By merging the pyramid structure of CNNs with ViT, the Pyramid Vision Transformer was introduced, overcoming many challenges in transferring Transformer models to dense prediction tasks ²³. Chen et al. ²⁴ proposed a dual-branch ViT model that combines image patches of different sizes to produce stronger image features.

Face-expression recognition based on multi-scale ViT focuses more on processing image details while effectively capturing the similarity between local and global features, thereby improving the recognition accuracy. Therefore, we utilize the characteristic of ViT, which divides images into patches, and design a dual-branch ViT structure to extract multi-scale facial features for better visual representation. In addition, we convert the speech signal into a spectrogram and leverage the residual structure of ResNet to extract refined speech emotion features.

2.2. Attentional Model for Multimodal Emotion Recognition

Multi-scale facial emotion recognition effectively addresses the challenges posed by facial feature scale variations and the limitations of single-scale approaches in capturing complex expressions. However, in interpersonal and human-robot interactions, emotional information is often expressed in the form of multimodal fusion rather than being concentrated in a single modality. Tzirakis et al. ²⁵ proposed an A–V fusion method that directly concatenates A and V features before feeding them into a long short-term memory (LSTM) model. Schoneveld et al. ²⁶ explored knowledge distillation using a teacher-student model for the V modality and a CNN model for the A modality using spectrograms, combining them together. Liu et al. ²⁷ proposed an end-to-end A–V emotion recognition model, EAVFormer, which effectively mitigates the issues of redundancy and dependency between audio and visual modalities. Although these methods have led to significant progress in emotion recognition, they fail to effectively capture the task-specific inter-modal relationships and key features.

Inspired by the attention mechanism, attention models for A–V fusion have been widely explored to model both the intra-modal and inter-modal relationships between the A and V modalities. Zhang et al. ²⁸ proposed an attention fusion mechanism that re-weights the obtained A and V features using a scoring function based on relevant information in each modality. Liu et al. ²⁹ proposed a novel contrastive learning-based module to address the inter-modal gap caused by missing modalities, with the aim of extracting modality-invariant features under complete modality conditions. Ghaleb et al. ³⁰ used a Transformer-based encoder to obtain attention weights through self-attention in emotion classification. They explored multimodal Transformers and self-attention modules, where the audio modality was used to attend to the visual modality to generate robust multimodal feature representations ³¹.

In this task, the fused emotional features are high-dimensional, making redundancy suppression and robustness enhancement critical. L1 regularization, through its sparsity constraint, removes irrelevant features while highlighting discriminative cues, thereby improving robustness and generalization. Compared to L2, L1 tends to produce sparse solutions, making it more suitable for emotion recognition tasks with redundant features. Therefore, incorporating L1 regularization into cross-modal fusion is both reasonable and necessary.

Conventional methods use Transformer-based cross-attention to model cross-modal relationships but often fail to fully exploit the correlations between A–V features. To overcome this problem, we propose a cross-modal cross-attention framework that constructs joint audio–visual representations. Unlike prior approaches, it emphasizes the key elements within each modality and integrates unimodal and joint features to capture complementary information, enabling deeper and more discriminative cross-modal interactions and improving the robustness and accuracy of emotion recognition.

3.1. Emotional Feature Extraction in MSFCA

ViT processes the input by dividing the image into fixed-size patches and flattening each patch into a vector. To enable information interaction between patches of different scales, this study divides the image into two sets of patches with different scales and added positional encoding to each set. A special identifier was added at the beginning of each input sequence, similar to BERT. By interacting with the patch token sequences from the other branch, the model can obtain feature information and pass it back to the original branch. Finally, the model uses the vector corresponding to the identifier as a global representation of the image for classification prediction.

3.1.1. Facial Feature Extraction Using Multi-Scale ViT

ViT performs the process of feature extraction from image data as follows: ViT divides the input image into a number of fixed-size patches, and each patch is transformed into a fixed-dimension vector by a linear transformation. The specific calculation process is expressed as follows:

\begin{equation} Z_{i} = x_{i} W + b, \end{equation}

where \(Z_{i}\) is the embedding representation of the patch and \(b\) is the bias term. To enable the model to understand the position of a patch in the sequence, ViT introduces position coding. The final representation of each patch combines the embedding vector and position encoding as follows:

\begin{equation} Z_{i}' = Z_{i} + PE_{i}, \end{equation}

where \(PE_{i}\) is the position encoding corresponding to patch \(i\). After this process, each image block is not only transformed into a patch representation, but also contains positional information, which can subsequently be fed into the Transformer encoder for further processing.

In ViT networks, the division size of image blocks significantly affects the accuracy of the model. If the image block division is too large, it will cover a large area, which may lead to important local details being ignored; if the image block division is too small, each image block can only capture subtle local features, making it difficult to obtain the global context of the overall image. In addition, too small image blocks can also lead to an increase in computational complexity, thus affecting the efficiency of the model. Specifically, we introduce a two-branch ViT network, where each branch handles image blocks of different sizes, and then propose a simple and efficient module for the interaction between the two-branch features, as shown in Fig. 2.

In facial image processing, the model mainly consists of \(k\)-multiscale Transformer encoders, and each encoder contains two branches: (a) L-branch: the large branch, which is used to extract the coarse-grained information of the image; and (b) S-branch: the small branch, which is used to extract the fine-grained information of the image. By fusing the information from the two branches, the classification task was finally completed using the CLS markers of each branch.

Effective feature fusion is key to learning multi-scale feature representations. To reduce computational complexity, we fuse the CLS tokens of the two branches by simply adding them together, rather than concatenating all the tokens. However, this approach does not enable feature interaction between the two branches. In the proposed method, the CLS token interacts with all image patches through a multi-head attention mechanism to accumulate global features. Subsequently each branch’s CLS token exchanges information with the patch proxy of the other branch and projects the information back to its respective branches, interacting with its own patch tokens, thereby enhancing the representation ability of each patch.

Specifically, for the big branch, it first collects patch tokens from small branches and connects its own CLS tokens to these patch tokens as follows:

\begin{equation} X'^{l} = \left[\left[f^{l}\left(X_{cls}^{l}\right) \right] \| X_{\mathit{patch}}^{s}\right], \end{equation}

where \(f^{l} (\cdot )\) is the projection function for size alignment. Then, cross-attention is computed between \(x_{cls}'^{l}\) and \(x'^{l}\). The formula is expressed as follows:

\begin{align*} \left\{\begin{aligned} Q &= W_q \times x_{cls}'^{l}, \\ K &= W_k \times x'^{l}, \\ V &= W_v \times x'^{l}, \end{aligned}\right. \end{align*}

where \(W_{q}\), \(W_{k}\), \(W_{v} \in R^{C\times (C/h)}\) are the learnable parameters, and \(C\) and \(h\) are the embedding dimension and number of heads, respectively. Based on the \(Q\), \(K\), \(V\) matrices, the cross-attention can be expressed as follows:

\begin{align*} \left\{\begin{aligned} &A = \mathit{softmax}\left(\frac{QK^{T} }{\sqrt{\dfrac{C}{h}} } \right), \\ &CA\left(x'^{l} \right) = AV, \end{aligned}\right. \end{align*}

where \(A\) denotes the attention weight matrix. The acquired \(y_{cls}^{l}\) is concatenated with the original \(x_{\mathit{patch}}^{l}\) to generate a new, big-branch feature that has finished interacting with the smaller branches. The feature connection formula is expressed as follows:

\begin{equation} Z^{l} = \left[ g^{l} \left(y_{cls}^{l}\right) \| x_{\mathit{patch}}^{l} \right], \end{equation}

where \(Z^{l}\) is the new big branch obtained after the CLS tokens in the big branch have completed their feature interactions, and \(g^{l} \left ( \cdot \right )\) is the size-aligned projection and backprojection function.

3.2. Emotional Feature Fusion in MSFCA

Although A–V fusion can be achieved through unified multimodal training, joint training of multimodal networks often underperforms compared with single-modality training. Therefore, we independently trained deep learning models for the A and V modalities to extract A and V features, which were then fed into a cross-modal attention module for A–V fusion to produce the final emotion category output.

Specifically, by separately focusing on the A and V features, we concatenate the A–V features to obtain the correlation matrices \(C_{a}\) and \(C_{v}\), which represent the correlation of the A and V features relative to the concatenated features. Next, the correlation matrices and original feature matrices are reduced to a new feature space to obtain \(H_{a}\) and \(H_{v}\), effectively removing redundant features and fully exploring the correlation between modalities, thereby enhancing the information representation ability of the features. Furthermore, by introducing an activation function, the reduced-dimensional features undergo a nonlinear transformation, further improving their expressive power. Finally, the \(H_{a}\) and \(H_{v}\) feature matrices are projected back into the original space and added to the original feature matrices, generating feature representations with richer semantic information.

Let \(X_{a}\) and \(X_{v}\) denote the image features and spectrogram features of the input video sequence \(S\) at a certain moment in time, respectively, where \(X_{a}\left \{ x_{a}^{1} ,x_{a}^{2} ,\dots ,x_{a}^{m}\right \} \in R^{d_{a} \times m}\) and \(X_{v}\left \{ x_{v}^{1} ,x_{v}^{2} ,\dots ,x_{v}^{m}\right \} \in R^{d_{v} \times m}\). Here, \(m\) denotes the number of images captured in the hunt \(S\), \(d_{a}\) and \(d_{v}\) denote the dimensions of A and V features, respectively, and the vectors \(x_{a}^{m}\) and \(x_{v}^{m}\) denote the eigenvectors of A and V modalities respectively, where \(l = 1,2,\dots,m\).

As shown in Fig. 3, cascade feature \(F\) of A–V is obtained by directly splicing the features of A and V. \(X_{f} =\left [ x_{a} ; x_{v} \right ] \in {\mathbb{R} ^{d \times m} }\), where the feature dimension of the cascade feature \(d=d_{a} + d_{v}\) . Using audio features as an example, the correlation matrix of the A features was calculated as follows:

Because the correlation matrix \(h_{a}\) and the feature matrix \(x_{a}\) have different dimensions, we map them to a uniform low-dimensional space by introducing learnable weight matrices \(W_{h_a}\) and \(W_{a}\), thus removing redundant features, making full use of inter-modal correlation, and enhancing the information representation of features. In addition, the representation of the features is further enhanced by introducing an activation function to transform the dimensionality reduction features nonlinearly. During the dimensionality reduction process, to enhance the sparsity of the model and reduce overfitting, we applied an L1 regularization operation to the weight matrix \(W_{a}\). The formula used is as follows:

The process of performing the L1 regularization operation on the weight matrix \(W_{a}\) is first performed by computing the sum of the absolute values of all elements in the computed weight matrix \(W_{a}\) as follows:

Next, the computed L1 norm is added to the loss function by multiplying it by the regularized intensity \(\lambda\) as follows:

Finally, the downscaled vector \(H_{a}\) is back-projected into the original space and summed with the original feature matrix to generate a feature representation with richer semantic information as follows:

4.1. Experimental Environment

A human-computer interaction system was constructed based on the proposed multimodal emotion recognition algorithm, as shown in Fig. 4. The system primarily consists of a Kinect camera, data transmission device, router, and emotion computing workstation. The workstation is equipped with an NVIDIA RTX A6000 GPU featuring 48 GB of dedicated memory and 128 GB of shared memory, an Intel Xeon Silver 4214 CPU with a maximum clock speed of 3.20 GHz, and 256 GB of RAM, running a 64-bit operating system. The experimental platform used PyCharm as the development environment, and the test datasets used were the multimodal emotion datasets eNTERFACE’05, RAVDESS, and IEMOCAP.

4.2. Data Settings

The eNTERFACE’05 dataset ³² is a multimodal affective dataset containing 42 subjects of 14 different nationalities, 81% of whom were male and 19% were female. All the experiments were conducted in English. The database contains a total of 1,166 video sequences, with video processing in Microsoft AVI format at \(720 \times 576\) resolution and audio sampling rate of 48,000 Hz in uncompressed 16-bit stereo format. Fig. 5 shows some samples of eNTERFACE’05.

The RAVDESS dataset ³³ consists of 7,356 files containing 24 professional actors (12 males and 12 females) speaking two lexically-matched statements in a neutral North American accent with seven emotions: calmness, happiness, sadness, fear, anger, surprise, and disgust. Samples selected from the RAVDESS dataset are shown in Fig. 6.

The IEMOCAP dataset ³⁴ is a widely used resource in emotion recognition research. It comprises approximately 12 hours of audiovisual recordings, performed by 10 actors through both scripted and improvised dialogues. The dataset encompasses a variety of emotional expressions, including happiness, sadness, anger, frustration, and neutral states. Samples selected from the IEMOCAPS dataset are shown in Fig. 7.

4.3. Simulations and Analysis of Results

To investigate the impact of image patch size ratios on facial emotion recognition performance, a series of experiments were conducted on the RAVDESS dataset. Specifically, the size of the small-scale image patches was fixed at \(12\times12\) pixels, whereas the side lengths of the large-scale patches were set to 2, 4, 5, 8, and 10 times those of the small patches. To ensure that images could be evenly divided into an integer number of patches, all input images were first resized to \(240\times240\) pixels before being fed into the multi-scale feature extraction network. During training, the initial learning rate was set to 0.0005, and the dropout rate was set to 30% to effectively mitigate the risk of overfitting.

As shown in Fig. 8, we plotted the relationship curves between the patch size ratio and recognition accuracy for the RAVDESS and eNTERFACE’05 datasets, where the black and blue curves correspond to the results on RAVDESS and eNTERFACE’05, respectively. The results indicate that the recognition accuracy initially increased with the ratio and then declined. Specifically, when the ratio of large to small patches was \(1:5\), the model achieved the highest accuracy of 83.67% on the RAVDESS dataset and 86.16% on the eNTERFACE’05 dataset. However, when the ratio was further increased to \(1:8\), the recognition accuracy decreased to 82.45% and 83.72%, respectively. This trend suggests that an excessively large patch ratio may introduce redundant background information or non-emotion-related features, thereby impairing the ability of the model to correctly recognize emotions. Moreover, when the discrepancy between patch sizes becomes too pronounced, the fine-grained emotional details captured by smaller patches may fail to effectively complement the features of larger patches, ultimately weakening the overall feature fusion and reducing the recognition performance.

Based on these experimental results, an image block scale ratio of \(1:5\) was selected for facial feature extraction in the subsequent stages of this study. In addition, the ResNet50 network was employed to extract audio modality features, which were then fused with facial image features for multimodal emotion recognition. To eliminate redundant information and enhance feature discriminability, dimensionality reduction was first applied to the extracted features, followed by L1 regularization to promote feature sparsity. Finally, a cross-attention fusion mechanism is introduced to effectively integrate complementary emotional information from both audio and visual modalities, thereby improving the overall performance of multimodal emotion recognition.

To verify the effectiveness of the proposed method, experiments were conducted using three benchmark datasets: eNTERFACE’05, RAVDESS, and IEMOCAP. To alleviate the impact of the limited dataset size on the experimental results, a five-fold cross-validation was employed, and all reported results represent the average performance across the five folds. In addition, regularization strategies were incorporated during training to enhance the generalization capability of the model. Figs. 9–11 show the confusion matrices of the proposed method for the eNTERFACE’05, RAVDESS, and IEMOCAP datasets, respectively. As shown in Fig. 9, the proposed method achieved an average recognition accuracy of 86.16% across six emotional states on the eNTERFACE’05 dataset. Fig. 10 shows the results on the RAVDESS dataset, where the average recognition accuracy across seven emotional states reached 87.14%. Fig. 11 shows the confusion matrix on the IEMOCAP dataset, where the proposed method achieved an average recognition accuracy of 73.16% across the six emotional states.

In the experimental results on the eNTERFACE’05 dataset, the model achieved the highest recognition accuracy for happiness, whereas the recognition performance for fear was the lowest. Fear was also prone to being confused with sadness or anger. This phenomenon is primarily attributed to the similarity between fear and sadness in facial expressions and vocal signals. At the facial level, fear and sadness often share similar muscle movement patterns, such as raised eyebrows, drooping eyelids, and widened eyes. At the vocal level, both emotions are typically characterized by lower pitch, slower speech rate, and reduced volume. A high overlap of these multimodal features can interfere with the classification process of the emotion recognition model, thereby affecting the overall recognition accuracy.

In the experiments conducted on the RAVDESS dataset, the model achieved relatively high recognition accuracy for happiness and anger, whereas the recognition performance for fear was comparatively poor. Notably, there is a high rate of confusion rate between happiness and surprise. This confusion is primarily attributed to similarities in facial expressions, as both emotions may exhibit features such as widened eyes or raised corners of the mouth, making them visually less distinguishable. Additionally, in the vocal modality, both emotions are typically associated with higher pitch and faster speech rate, leading to a certain degree of overlap in the audio feature space. This overlap reduces the sensitivity of the system to the differences between these emotions and further increases the difficulty of accurate classification.

As shown in Fig. 11, the model achieved satisfactory recognition performance across the six emotion categories on the IEMOCAP dataset. The recognition accuracies for excited, sad, and happy were the highest, reaching 0.80, 0.78, and 0.73, respectively. However, confusion was observed between happy and excited, as well as between anger and frustrated, owing to their semantic and acoustic similarities. Despite these overlaps, the overall classification results demonstrate that the proposed model exhibits strong generalization capability under complex emotional conditions.

By analyzing the results shown in Figs. 9–11, it can be observed that there are noticeable differences in the recognition performance across different emotion categories. This can be explained from several perspectives. Happiness typically exhibits more distinctive facial and vocal characteristics, such as a pronounced upward movement of the corners of the mouth, higher pitch, and faster speech rate, which make it easier for the model to capture and distinguish. In contrast, fear shows a higher degree of similarity with sadness and anger in its manifestations, such as raised eyebrows, drooping eyelids, and lowered pitch, which increases the likelihood of misclassification. Imbalanced data distribution may weaken the learning ability of the model for certain minority classes. Moreover, individual differences in the expression of fear and sadness further increase the recognition difficulty of these categories. Collectively, these factors contribute to the performance discrepancies observed across the different emotion categories.

The proposed MSFCA model contained approximately 120 million trainable parameters. Training was conducted on an NVIDIA RTX A6000 GPU with a batch size of eight and an input image resolution of \(224\times224\). With these settings, the training of each fold (350 epochs) required approximately 13 hours. The model has an estimated computational complexity of approximately 20 GFLOPs, and the average forward inference time is approximately 20 ms per sample, indicating that the MSFCA can achieve real-time inference performance in practical A–V emotion recognition applications.

The ablation experiments conducted on the RAVDESS dataset aimed to evaluate the effectiveness of each module within the proposed model. The detailed results are listed in Table 1. The baseline model, C3D-DBN ²⁷, achieved a recognition accuracy of 85.79%. When employing a single-scale ViT with a cross-attention mechanism, the accuracy decreased significantly to 78.53%, indicating that a single-scale structure was insufficient for adequately capturing multimodal features. With the further introduction of L2 and L1 regularization, the accuracy improved to 80.24% and 82.48%, respectively, suggesting that sparse constraints help to mitigate redundancy in high-dimensional features. In contrast, integrating a multi-scale ViT with cross-attention considerably enhanced the performance, achieving an accuracy of 84.82%, thereby confirming the effectiveness of multi-scale modeling in emotion recognition. Building on this, incorporating L2 regularization further increased the accuracy to 85.39%, whereas the complete model, MSFCA, which combined low-dimensional projections with L1 regularization, achieved the highest accuracy of 87.14%. Overall, these results demonstrate the significant advantages of multi-scale feature modeling and L1-based sparsity in cross-modal emotion recognition, thereby validating the effectiveness and superiority of the proposed method.

Table 2 presents a performance comparison of the different multimodal emotion recognition methods on the eNTERFACE’05 dataset. Traditional approaches such as DIFL-VGG6 ³⁵ and DST ³⁶ achieved recognition accuracies of 80.89% and 82.83%, respectively, whereas more advanced models, such as CNN-LSTM ³⁷ improved the accuracy to 85.43%. By contrast, the proposed MSFCA method achieved the highest accuracy of 86.16%, demonstrating the effectiveness of the proposed fusion strategy in multimodal emotion recognition tasks.

Table 3 presents a performance comparison of different emotion recognition methods on the REDVESS dataset. Att-Net ³⁸ and AttA-Net ³⁹ achieved recognition accuracies of 80.00% and 82.62%, respectively, whereas the A8-V4 method ⁴⁰ further improved the accuracy to 86.62%. The proposed MSFCA method achieved the best performance with an accuracy of 87.14%.

Table 4 presents the performance comparison of the different multimodal emotion recognition methods on the IEMOCAP dataset. Traditional methods such as DCCAE ⁴¹ and MMIN ⁴² achieved recognition accuracies of 52.12% and 66.36%, respectively, whereas the more advanced model TIM-Net model ⁴³ improved the accuracy to 72.53%. In contrast, the proposed MSFCA method achieved the highest accuracy of 73.16%, further demonstrating the effectiveness and superiority of the proposed fusion strategy in multimodal emotion recognition tasks.

We selected two models with relatively high accuracy for further cross-dataset evaluation to verify the generalization capability of the proposed model across different corpora. The results are listed in Table 5. Specifically, \(M_{R}\) denotes the model trained on the RAVDESS dataset, while \(M_{e}\) denotes the model trained on the eNTERFACE’05 dataset. When the model \(M_{R}\) trained on RAVDESS was tested on the eNTERFACE’05 and IEMOCAP datasets, it achieved an accuracy of 84.42% and 68.27%, respectively. Conversely, when the model \(M_{e}\) trained on the eNTERFACE’05 was evaluated on RAVDESS and IEMOCAP datasets, it achieved accuracies of 85.87% and 70.38%, respectively. These results validate the robustness and effectiveness of the proposed method under cross-dataset scenarios.