2.1. Problem Definition

The Sybil attack detection problem addressed in this study focuses on identifying critical information and time-dependent features in vehicle-to-vehicle communication data within VANETs. As illustrated in Fig. 1, a Sybil attack involves an attacker (a malicious vehicle) creating multiple fake vehicles (Sybil vehicles) to disrupt the communication and decision-making processes of legitimate vehicles (normal vehicles). Each vehicle is represented by a multi-dimensional feature vector \(K\), where \(K_i\) denotes the \(i\)-th feature for each vehicle (including both normal and Sybil vehicles), with \(i\) belonging to feature index set \(I\). The total number of vehicles in the system (including both normal and Sybil vehicles) is denoted as \(N = |V| + |S|\). Each vehicle (normal or Sybil) can be represented by a feature vector \(F\) containing multiple \(K_i\) features.

The true label of \(F_i\) is denoted by \(T_i\). When \(F_i\) represents a fraudulent vehicle, we set \(T_i = 1\), and when it indicates a legitimate vehicle, we assign \(T_i = 0\). To address this binary classification problem, we developed an LSTM–AttnAE model (\(F_i\)) to predict true label \(T_i\).

2.2. LSTM Model

LSTM networks are a specialized class of recurrent neural networks (RNNs) designed to capture long-term temporal dependencies in sequential data, making them well-suited for modeling dynamic vehicle behavior in VANETs. As illustrated in Fig. 2, an LSTM unit comprises four core components—memory cells, input gates, output gates, and forget gates—that operate in tandem to regulate the flow of temporal information ²⁸. The core output of an LSTM unit at time step \(t\) can be expressed as follows:

Here, \(x_t\) represents the input at time step \(t\), and \(h_{t-1}\) represents the hidden state from the previous step. This formulation enables LSTM to retain critical temporal information while mitigating the vanishing gradient problem of standard RNNs; this is a key advantage in analyzing time-series vehicular data that evolve over multiple time steps.

Although LSTM excels at temporal modeling, it lacks mechanisms to compress high-dimensional VANET features or dynamically weight critical attack-related attributes. Existing LSTM-based detection methods often treat all features equally, reducing their ability to identify subtle Sybil attack patterns in noisy vehicular data. To enhance the LSTM capability, Zhong et al. proposed an LSTM–BiGAN framework that combines temporal modeling with generative learning to improve the detection of hidden Sybil nodes ²⁹.

2.3. Autoencoder Model

By contrast, an autoencoder, as an unsupervised learning model ³⁰, reconstructs the input data through encoding and decoding processes to learn its latent features. Its structure consists of an encoder and a decoder. The encoder compresses input data \(x \in R^{n}\) into a low-dimensional latent space \(z \in R^{m}\) (where \(m < n\)), and the encoding process is as follows:

Here, \(W_{e} \in \mathbb{R}^{m \times n}\) represents the weight matrix of the encoder, and \(b_{e} \in \mathbb{R}^{m}\) represents the bias term of the encoder. The decoder attempts to reconstruct the original input. The decoder maps latent representation \(z\) back to original data space \(x^{\prime} \in \mathbb{R}^{n}\), and the decoding process is as follows:

Here, \(W_{d} \in \mathbb{R}^{n \times m}\) represents the weight matrix of the decoder and \(b_{d} \in \mathbb{R}^{n}\) represents the bias term of the encoder. By minimizing the difference between the input data and reconstructed data \(x^{\prime}\), the autoencoder can extract the essential features of the data and play an important role in anomaly detection.

The primary objective of an autoencoder is to minimize the reconstruction error between the input and output data. In this study, we employed the MSE method to compute the reconstruction error according to Eq. (6).

Despite their effectiveness in feature compression, anomaly detection, and reconstruction error minimization, traditional autoencoders focus on static feature reconstruction and fail to capture temporal dependencies in vehicle behavior, while treating all features uniformly, which limits their ability to detect evolving Sybil attack patterns relying on subtle temporal anomalies. Tang et al. addressed this issue by proposing a GCN-GRU hybrid model that integrates spatiotemporal features to detect multiple types of Sybil nodes ³¹.

2.4. Attention Mechanism Model

The origin of attention mechanisms can be traced back to 2014 when Bahdanau et al. applied them to machine translation ³². These mechanisms dynamically adjust the focus of the model on different features, enabling it to more effectively identify the key elements when processing large-scale data. Essentially, attention works by assigning varying weights to different input data, thereby allowing the model to prioritize critical information.

As shown in Fig. 3, in attention mechanisms, sequences or word vectors are mapped to query (\(Q\)), key (\(K\)), and value (\(V\)) vectors ³³. The attention scores are used to measure the correlation between \(Q\) and \(K\). Considering the self-attention mechanism as an example, \(Q\), \(K\), and \(V\) can be linearly transformed by multiplying the input word vectors with different trainable weight matrices \(W_q\), \(W_k\), and \(W_v\), respectively.

For a specific word vector, the similarity between \(Q\) and \(K\) is calculated as follows:

This mechanism, \(QK^{T}\), measures the correlation between input vectors \(d_k\) and indicates their dimensions that are used to scale the dot product and prevent numerical overflow from affecting gradient stability. This approach is particularly crucial in VANETs because it enables models to maintain a high detection precision and response speed when faced with diverse and dynamically evolving attack patterns.

Despite their ability to enhance feature weighting and prioritize critical attack-related information, attention mechanisms are rarely combined with temporal models such as LSTM in existing Sybil detection approaches, indicating that they cannot leverage temporal context to identify evolving attack strategies; this is a critical limitation in dynamic vehicular environments.

2.5. Limitations of Existing Approaches

The preceding analysis reveals the critical limitations of the current Sybil attack detection methods in VANETs, highlighting the necessity of a hybrid modeling strategy. Although LSTM excels at temporal dependency modeling and autoencoders are effective for high-dimensional feature compression, only few studies have integrated these two models to leverage their complementary strengths. This gap prevents existing methods from simultaneously capturing dynamic vehicle behavior and compressing noisy vehicular data, creating a trade-off between temporal modeling accuracy and feature processing efficiency. As noted by Hadri et al., the gap between temporal modeling and feature compression remains a major unresolved challenge in this field ¹.

Another key limitation is the novelty gap in this field; to the best of our knowledge, no existing work has applied the LSTM–AttnAE hybrid model to Sybil attack detection in VANETs, with most relying on single-model solutions or traditional techniques (for example, RSSI and reputation systems) that poorly adapt to evolving attack patterns.

Additionally, traditional approaches adopt static feature weighting, treating all inputs equally instead of dynamically prioritizing Sybil attack-related features in real time. This reduces the sensitivity to subtle, time-varying attack signatures critical for an early detection in dynamic vehicular environments. Sultana et al. and Gebrezgiher et al. emphasized that dynamic feature weighting is essential for detecting time-varying Sybil attacks, with Gebrezgiher et al. achieving 0.95 F1-score using a VAE-based real-time detection approach ^27,34.

Collectively, these limitations motivated our proposed LSTM–AttnAE hybrid model that integrates LSTM for temporal modeling, autoencoders for feature compression, and attention mechanisms for dynamic feature weighting, to address these gaps comprehensively.

3.1. Data Preprocessing

Histogram and boxplot analyses reveal that certain features exhibit significant skewness that can potentially impact subsequent model training. To ensure data quality, missing values are addressed by comparing multiple imputation methods, with forward filling ultimately adopted to handle them. Duplicate records are removed to ensure the uniqueness of each sample. To eliminate scale differences among the features, data normalization is performed using the \(z\)-score method, scaling the feature values to the range of \([0, 1]\). The formula used is as follows:

Considering the highly dynamic and real-time nature of vehicular network data, to enhance the capability of capturing short-term behavioral patterns, the data are segmented using fixed time windows, with each time window treated as an independent input sample. Each window contains a fixed number of time steps \(T\), and the corresponding features and labels are extracted. Assuming a window size of \(T\) and a time series denoted by \(X=[x_{1}, x_{2},\ldots, x_{n}]\), the windowed data can be represented as follows, where \(i\) denotes the starting index of the window.

The window size is determined based on the communication characteristics of vehicular networks. In terms of node mobility, the average speed of vehicles in VANETs is approximately 60 km/h, meaning that the vehicle travels approximately 83 m within 5 s; this indicates that the communication topology of nodes changes significantly in a short time. Combined with the 1-s sampling frequency of the dataset, a window size of \(T=5\) is extremely small to capture stable behavioral features, whereas a window size of \(T=15\) introduces redundant information and increases computational overhead. We set the window sizes to {5,10,15,20} for experimental evaluation, and the results show that \(T=10\) achieves the optimal balance between detection accuracy and computational efficiency. Therefore this window size was finally selected.

3.2. Proposed LSTM–AttnAE Model

Deep learning can help represent massive amounts of network traffic with more discriminative feature spaces. This technology employs multiple processing layers to accurately model the input features. Compared with traditional manual feature descriptions, this approach offers advantages because deep learning techniques can automatically extract discriminative features compared to manually generated features. However, traditional LSTM models often struggle with complex vehicle network data because they lack the ability to assign varying degrees of attention to sub-window features across multiple time steps. They frequently encounter issues such as single temporal dependency, feature loss, and reliance on preset thresholds.

Inspired by ³⁵, this study developed an LSTM-autoencoder fusion model with an attention mechanism for anomaly detection, named LSTM–AttnAE. The model comprises of three modules: an LSTM encoder, attention module, and LSTM decoder. To reduce the temporal dependency and enhance the ability of the model to capture feature diversity in vehicle network time-series data, this study integrates attention mechanisms after the LSTM encoder output to improve key feature tracking and prevent feature loss. Additionally, this study adopts autoencoders in the anomaly detection model because they aim to learn the optimal parameters for reconstructing the input patterns at the output layer.

The proposed model progressively extracts temporal sequence features using an LSTM encoder layer to capture temporal dependencies in the data, as illustrated in Fig. 4. The LSTM encoder processes the input time-series features to capture temporal dependencies and generates hidden states that encode historical behavioral contexts. This study employs a unidirectional LSTM to preserve the natural forward flow of vehicular data, ensuring that the model focuses on historical patterns to predict future states.

Subsequently, to ensure technical completeness, the key design choices of the scaled dot-product attention mechanism are elaborated; the 64-dimensional outputs of the LSTM encoder are projected onto 64-dimensional \(Q\), \(K\), and \(V\) vectors, preserving critical temporal information while maintaining computational efficiency. For weight matrix initialization, the Xavier uniform initialization strategy is employed to mitigate vanishing or exploding gradient issues. A single-head self-attention architecture is adopted instead of multi-head attention because vehicular network temporal sequences exhibit strong local correlations, making a single-head design sufficient to capture short-term dependencies while avoiding excessive overhead.

Subsequently, the scaled dot-product attention layer weights the LSTM outputs, dynamically emphasizing the critical time steps associated with anomalous behavior. This enhances the focus on key time points while appropriately suppressing non-critical information, thereby enabling the model to concentrate on significant temporal anomalies. This approach mitigates numerical instability in attention weight calculations, ensuring a smooth weight distribution across time steps and improving sensitivity to stealthy attacks. The attention layer further maps the data into high-dimensional feature representations, automatically extracting stage-specific characteristics during this process.

After attention weighting, stacked dense layers perform dimensionality reduction and non-linear feature transformation using ReLU activation, reducing the computational complexity while preserving critical temporal and attention-aware features. The LSTM decoder reconstructs the original input sequence from the compressed features and is initialized with the final hidden state of the encoder to maintain temporal coherence. The model attempts to reconstruct the original input data based on these high-dimensional features, with the reconstruction errors serving as key indicators for subsequent anomaly detection.

Finally, the autoencoder reconstruction step compares the decoder output with the original input, using reconstruction errors (measured using the MSE) to identify anomalies. Samples with reconstruction errors exceeding a learned threshold are flagged as anomalous, leveraging the ability of the autoencoder to model the normal behavior in an unsupervised manner.

Throughout this process, the data undergo multi-layered feature extraction and transformation from the original input sequence, ultimately enabling anomaly detection. This architecture leverages three core strengths: the temporal modeling capability of LSTM, the efficient information extraction of the attention mechanism, and the robust compression and reconstruction properties of the autoencoder. Thus, it achieves efficient compression and anomaly detection of time-series data. To further clarify the synergistic working principle of the model components and the solutions to technical challenges during integration, the relevant details are elaborated in the subsequent sections.

3.3. Model Fusion Mechanism and Technical Challenges

The LSTM–AttnAE model achieves robust anomaly detection through the synergistic interaction of its core components. The LSTM module acts as the temporal backbone of the framework, encoding historical vehicular behavior into hidden states to capture the time-evolving dependencies that are essential for detecting gradual attacks such as prolonged Sybil impersonation. Building on this temporal foundation, the attention mechanism refines the LSTM outputs by dynamically weighting the time steps to highlight the critical anomalies that may be obscured in standard LSTM outputs, thereby enhancing the detection of stealthy attacks in vehicular networks. Finally, the autoencoder module compresses the attention-weighted features and reconstructs the original input sequences, with a reconstruction error used to identify deviations from the normal behavior. This unsupervised framework enables the detection of Sybil attacks without relying on labeled attack data; this is critical in dynamic vehicular environments where attack patterns evolve rapidly.

The integration of the LSTM, attention, and autoencoder components introduces several technical challenges that require targeted solutions. The LSTM encoder outputs high-dimensional hidden states that increase the computational cost of the attention weight calculation and can hinder the real-time performance in vehicular network applications. To resolve this, we introduce stacked dense layers after the attention layer to reduce the dimensionality and extract deep features, ensuring efficiency while preserving the critical temporal and attention information essential for accurate anomaly detection.

In addition, the random initialization of attention weights often leads to unstable training and slow convergence because the model struggles to learn meaningful weight distributions early in the training. This issue is addressed by adopting Xavier initialization to balance the variance in the forward and backward passes, combined with dropout regularization to prevent overfitting, resulting in faster convergence and improved generalization to unseen data.

Finally, the reconstruction loss of the autoencoder and the temporal loss of the LSTM exhibit inherent trade-offs because overemphasizing one can degrade the performance of the other. A multi-task loss function with a tunable weight \(\lambda = 0.7\) is implemented to balance both objectives, ensuring the model preserves temporal dependencies while maintaining a high reconstruction accuracy.

Here, \(\mathcal{L}_{\text{recon}}\) denotes the reconstruction loss (mean squared error between the original input and decoder output), \(\mathcal{L}_{\text{temporal}}\) represents the temporal loss (mean squared error between the hidden states of the LSTM encoder and decoder), and \(\lambda\) is a hyperparameter that controls the trade-off between the two loss components. This loss function is integrated into the training pipeline outlined in Algorithm 1 to ensure that the model learns to accurately reconstruct input sequences and preserve temporal dependencies that are critical for detecting anomalies in vehicular time-series data.

3.4. Algorithm Implementation and Post-Processing

The complete LSTM–AttnAE algorithm is detailed in Algorithm 1, encompassing encoding, attention weighting, decoding, and reconstruction error calculation. Batch normalization layers are integrated post-encoder and post-decoder to stabilize the training, whereas dropout layers prevent overfitting. The model is trained to minimize the multi-task loss function, with the reconstruction errors serving as discriminative features for anomaly detection.

Following the reconstruction error computation, an SVM classifier is employed to perform the final anomaly classification. The SVM is trained on the reconstruction errors from the validation set, learning a decision boundary to distinguish between normal and anomalous behaviors. This hybrid approach leverages the strength of LSTM–AttnAE in unsupervised feature learning and reconstruction, whereas the SVM provides a robust and interpretable classification step to validate anomalies.

4.1. Experimental Setup

Before model training, \(z\)-score normalization was applied to rescale the original input data. To ensure the reproducibility of the experiments and the optimal performance of the model, systematic hyperparameter optimization was implemented via a grid search combined with three-fold cross-validation. We evaluated the candidate sets of hidden layer numbers {1,2,3}, hidden layer dimensions {32,64,128}, learning rates {1e-4,5e-4,1e-3,5e-3,1e-2}, batch sizes {16,32,64}, and regularization strategies including dropout values ranging from 0.1 to 0.5 and L2 regularization values ranging from 1e-4 to 1e-2. The quantitative results of performance and convergence characteristics for different hyperparameter combinations are presented in Table 1. Although the 128-hidden-dimension configuration yielded a slightly higher F1-score of 0.982, it required more training epochs and incurred greater computational overhead. By contrast, the 64-hidden-dimension setup achieved a strong F1-score of 0.980 with faster convergence and lower computational cost. The optimal configuration was determined as follows: the LSTM encoder was configured with two hidden layers, each with 64 hidden states; the batch size was set to 32; the dropout value was 0.2; and the learning rate of the Adam optimizer was set to 1e-3.

Based on the optimal configuration, an attention mechanism layer with consistent input and output dimensions of 64 was constructed following the LSTM encoder that extracted high-value feature information through weighted feature summation. Then, a fully connected layer with an input dimension of 64 and an output dimension of 128 was introduced that further enhanced the feature extraction and representation capabilities of the model via high-dimensional mapping. The Tanh activation function was adopted for model training, and the Adam optimizer with a learning rate of 1e-3 was used to minimize the mean squared error between the predicted and actual values. The total number of training epochs was set to 100. All experiments were conducted based on the PyTorch framework for model training and performance evaluation.

4.2. Evaluation Metrics

Several evaluation metrics have been introduced to study binary classification problems. In this study, four commonly used metrics were considered: precision, recall, F1-score, and ROC curve. The first three measures are expressed by Eqs. (12)–(14).

4.3. Experimental Results

To evaluate the performance of the LSTM–AttnAE model in detecting Sybil attacks in VANETs, we conducted evaluations not only on individual attack types across five categories within the VeReMi extended dataset but also on a combined dataset integrating all five attack types.

As presented in Table 2, the model demonstrated an exceptional performance with high precision, recall rates, F1-scores, and AUC values approaching or exceeding 0.97 when identifying specific attack types. These performance differences stemmed from the distinct anomaly salience of each attack pattern. Type-4 (random position attack) exhibited the highest detection accuracy owing to its frequent extreme spatiotemporal discontinuities that made the anomalies most salient. By contrast, Type-1 (constant position attack) showed a slightly lower performance because its fixed signals lacked high-frequency random jumps. Type-2, Type-8, and Type-16 attacks retained partial normal spatiotemporal correlations, leading to more subtle anomalies. The integration of the attention mechanism still enabled the model to better detect these covert attacks, resulting in a relatively strong performance. Furthermore, the model maintained robust performance even when processing comprehensive datasets containing all attack types, consistently delivering high precision, recall rates, F1-scores, and AUC values that highlighted its generalization capability and robustness across diverse datasets.

As shown in Fig. 5, the model demonstrated a higher precision for Type-4 random-position Sybil attacks and a better recall rate for Type-1 fixed-position Sybil attacks among the five attack types. This was because Type-4 and Type-1 attacks exhibited greater positional data discrepancies than the normal vehicles that the attention mechanism could more easily detect.

Comparative analyses were conducted with baseline methods, including random forest, decision tree, AdaBoost, and KNN. All the methods were trained and tested for five rounds, and random errors were minimized by averaging the results. To further verify the performance advantages of the LSTM–AttnAE model, the comparison was extended to two classical Sybil attack detection methods for VANETs, namely, the RSSI-based and social network-based (SN-based) detection methods, and detection latency was added as a key efficiency evaluation index for real-time VANET scenarios.

As presented in Table 3, the proposed LSTM–AttnAE model significantly outperformed all the comparison methods in four core indicators, namely precision, recall, F1-score, and AUC, with both precision and AUC reaching 0.99. The traditional KNN-based detection method showed low precision and recall, with recall of only 0.80, complicating the effective identification of Sybil attack behaviors. Although the decision tree and AdaBoost models showed certain improvements in the detection performance, they still had evident detection shortcomings when faced with complex attack patterns and dynamic data noise. The random forest model had a relatively better detection performance, with a precision of 0.92 and an AUC of 0.96; however, it could not effectively focus on key abnormal features in diverse attack scenarios, leaving room for improvement in the detection effect. The results of this extended comparison showed that the RSSI-based detection method had an unstable detection performance owing to signal fluctuations, whereas the SN-based detection method had a high computational overhead that limited the detection efficiency. By contrast, the LSTM-AttnAE model realized dual optimization of detection performance and efficiency while maintaining high detection precision, with an F1-score of 0.99 and average detection latency of only 12.3 ms. This was the only comparison method that achieved both high precision and low latency.

As shown in Fig. 6, evident differences existed in the detection latencies of the different detection methods that were determined by the core computing logic and architectural design of each method. The SN-based detection method required to construct and maintain the social relation graph between vehicles in real time, involving complex graph structure calculations and information propagation mechanisms, with the largest computational overhead. Its average detection latency reached 32.6 ms that was the highest among all the compared methods. KNN was a computation-intensive detection method. In the model inference stage, all samples in the training set had to be traversed, and the distance between the samples had to be calculated, resulting in a long detection time and an average detection latency of 24.5 ms. The random forest model was constructed by integrating multiple decision trees. In inference, the output results of all decision trees had to be synthesized to complete the final judgment, and the amount of computation was significantly higher than that of a single decision tree. Therefore, its average detection latency was 18.7 ms that was higher than 14.1 ms of the decision tree model. The RSSI-based detection method completed the attack judgment only through the signal strength threshold, with the simplest computing logic and an average detection latency of 9.8 ms that was the lowest among all the compared methods. However, the detection precision of this method was also the worst among all the compared methods, with evident performance shortcomings.

The LSTM–AttnAE model achieved low detection latency mainly owing to its lightweight hybrid architectural design. The LSTM encoder of the model adopted a lightweight structure with two layers and 64 units that could efficiently extract the temporal features of the vehicle position, avoid the computational redundancy caused by deep networks, and significantly reduce the computational overhead in the feature extraction stage. The attention mechanism introduced in the model performed dynamic weighted calculations on only key abnormal features such as the position jump amplitude and identity switching frequency, without complex processing on all the features. To effectively improve the detection precision of the model, no additional computational burden was introduced. This architectural design enabled the model to quickly process vehicle position sequence data that not only met the core requirements of the VANET for the real-time performance of the detection model, but also ensured the high detection precision of the model, adapting to the dynamic and complex application scenarios of the VANET.

4.4. Ablation Analysis

The experimental results demonstrated that the proposed LSTM–AttnAE anomaly detection hybrid model outperformed traditional methods across multiple public datasets. Figs. 7 and 8 present the key performance metrics (including the precision, recall rate, F1-score, and AUC) of the model on both the VeReMi extended dataset and another public dataset, the Car-Hacking dataset, compared with conventional LSTM-based detection methods, autoencoder models, and LSTM-autoencoder models without attention mechanism integration.

The ablation study demonstrated that integrating the attention mechanism improved the F1-score from 0.96 to 0.99. This improvement stemmed from two key advantages. (1) The attention mechanism dynamically weighted high-dimensional temporal features extracted by the LSTM, emphasizing critical anomaly indicators such as the position jump magnitude and identity switching frequency that enhanced the focus of the model on discriminative signals. (2) The attention mechanism mitigated the long-sequence forgetting limitation of the LSTM by assigning higher weights to features at critical anomaly timestamps, enabling a more robust detection of sparse and covert anomalies and reducing false negatives in complex scenarios.

As shown in Fig. 7, the proposed model achieved a precision of 0.99 and an AUC value of 0.99 on the VeReMi extended dataset, representing 18% and 17% improvements, respectively, compared with the LSTM-based detection methods. This significant performance enhancement primarily stemmed from the dynamic weight allocation capability of the attention mechanism for key features, enabling the model to more accurately capture the anomalous characteristics of Sybil attacks. Compared with the LSTM-autoencoder model without integrated attention mechanisms, the proposed model demonstrated 8% and 6% improvements in precision and F1-score, respectively, further validating the effectiveness of the attention mechanisms in detecting complex attack patterns.

As shown in Fig. 8, the experimental results on the Car-Hacking dataset demonstrated that the proposed model achieved a precision rate of 0.98 and an AUC value of 0.99, significantly outperforming the comparison methods. Notably, the F1-score metric showed a 6% improvement over the LSTM-autoencoder model without attention mechanism integration. These findings indicated that the attention mechanism exhibited strong adaptability and generalization capabilities across various datasets and attack scenarios.