2.1. Scene Text Recognition

In recent years, with the further development of unmanned systems and related research, the study of STR has continued to grow. Existing STR methods can generally be classified into language-free and language-aware approaches. Language-free STR methods treat recognition as a character prediction task by directly exploiting visual features without modeling relationships between characters. Among them, CTC-based research ¹⁸ has played a long-lasting role. Methods using CTC decoders ^{14,23,24,25,26} directly convert image features into text sequences by employing blank labelling and post-processing to address alignment issues. These methods achieve fast inference but their performance is limited due to the lack of explicit character alignment. Language-aware STR methods incorporate attention mechanisms ^15,27,28,29 to locate each character by querying the corresponding visual features. Building upon this idea, many works integrate language modelling concepts into the decoding process to further enhance recognition accuracy. Such models are typically categorized into those using external language models (LMs) ^15,30,31 and those learning internal LMs during training ^32,33,34. For instance, SRN ³⁰ and ABINet ¹⁵ refine visual predictions by feeding them into external language models, although depending solely on external LMs without considering visual features may lead to incorrect corrections. On the other hand, methods such as PARSeq ¹⁶ and MAERec ⁶ implicitly train internal LMs in an autoregressive manner and have achieved strong results. Overall, despite the progress of both language-free and language-aware methods, achieving robust generalization in diverse real-world scenarios remains a key challenge for STR research.

However, most existing STR approaches focus on improving sequence modeling or language refinement, while relying primarily on large-scale data or fine-tuning to handle appearance variations. Few methods explicitly incorporate structured character-level visual priors to model the similarity and discrepancy between canonical character forms and scene instances. This limitation motivates the exploration of prior-guided feature modeling mechanisms for improved generalization.

2.2. Prompt Learning

Prompt Learning in NLP: With the success of the large language model ³⁵, Brown et al. ³⁶ introduced GPT-3 to demonstrate strong performance in a variety of NLP tasks simply by specifying tasks and few-shot demonstrations through textual interactions with the model. Prompts further proved to be useful tools for solving various NLP tasks and benchmarks ^37,36.

Prompt Learning in CV: Inspired by the success of prompts in NLP, there has been similar research on prompts in computer vision, especially with the rapid development of multimodal large models, where prompt learning can also play a large role in establishing links between image and language modalities. Recent visual prompts works, e.g., Bar et al. ³⁸ studied visual prompts: sample input-output images for a new task at test time and a new input image, with the goal of automatically generating an output image that is consistent with the given example. InCTRL ³⁹ uses normal images as prompt samples and learns the contextual information between query images and prompt samples through residuals, thus effectively improving the anomaly recognition accuracy. Li et al. ⁴⁰ developed a general-purpose prompt encoder that can support a wide range of visual prompts, such as strokes, boxes, and dots.

Although prompt learning has shown strong capability in task adaptation and multimodal alignment, existing visual prompts are typically designed as task-conditioning signals rather than structured character prototypes. Moreover, prior works rarely model the explicit similarity-discrepancy relationship between reference prompts and input instances. In contrast, our framework treats character prompts as structured visual priors and integrates them through differential feature modeling.

3.1. Pipeline

The architecture of VPDL is shown in Fig. 2 and consists of a visual encoder and a sequence decoder. We used Vision Transformer (ViT) ⁴¹ as the image encoder, constructed character-level visual prompts as a reference for the model, and then designed a visual prompt-guided local-to-global differential learning strategy. Through this differential learning, the model can learn both similar and differing information under the guidance of character visual prompts, thereby enhancing the discriminative capability of the input image features. These image features are then combined with text prompts as inputs to the decoder. The text prompts guide the decoder to further focus on the textual information within the image features, ultimately enabling more accurate decoding of the text sequence.

3.2. Character-Level Visual Prompt

To build a more generalizable scene text recognition model, we made the following observations when designing the visual prompt samples: the diverse appearances of characters pose significant challenges for recognition, resulting in suboptimal performance in some difficult scenarios. Therefore, we considered how humans accurately recognize various character shapes across diverse scenes. The reason lies in the human ability to reference memories of previously seen character shapes. By comparing these memories to characters encountered in scene images, humans can deduce the textual content by identifying similarities and differences.

Based on the above insights, we directly designed character-level visual prompts with the primary goal of providing the model with visual reference information for characters across different scenes and forms using a small but diverse character set. This functions similarly to human memory, helping the model learn the variations in character appearances across various scenes. This approach enhances the generalizability and recognition accuracy of the model.

Specifically, we collected images of each character appearing in existing scene text datasets, which were manually cropped from the original images. Some examples are shown in Fig. 3. In our implementation, we construct an offline prompt library covering the full recognition vocabulary (96 character categories), and keep it fixed during training and inference. Unless otherwise specified, 20 prompt samples are used per category (1920 in total), and similarity matching is explicitly performed against the full prompt library during both training and inference stages. We further enhance these character-level prompts using sharpening and random Gaussian noise to better simulate real-scene degradations. Overall, the character-level visual prompts provide structured visual priors for the model, enabling more robust recognition under diverse and complex scene variations.

3.3. Local-to-Global Differential Learning

To effectively capture the fine-grained differences between input images and character visual prompts, we design a local-to-global differential learning strategy. This strategy enables the prompts to enhance the understanding of image content at the character level across different scenes, thereby improving text recognition accuracy. We use ViT as the image encoder, which contains multiple layers, where shallow layers learn low-level features of the image such as background information, while high-level layers learn visual features at an abstract level, such as textual information in the image. In the local feature extraction stage, we leverage differential learning between the patch blocks of the prompted image and those of the input image within different blocks of the ViT. At the end of each layer, we obtain enhanced patch representations of the original input, along with the CLS token, which encapsulates the global features of the image. By establishing a connection between these two types of global features, we derive the final representation of the input image. Through multi-level differential learning, the model’s ability to handle different scenarios is enhanced.

Local-Level: In local level differential learning, the method first calculates the similarity between each patch embedding of the input image and each character embedding in the character library, and then this similarity result is used to select the most similar \(K\) character features for the purpose of ensuring that each patch interacts with only the most relevant character features in order to make the feature recognition capability effective, as shown in the following equation:

At each layer of the encoder, the above computational process is repeated, such that the layer-by-layer enhancement process ensures that the model gradually focuses on the most relevant prompts in the character library, making the image feature representation more discriminative. It should be noted that, in the actual implementation, we progressively tighten the screening range by decreasing the value of \(K\) across layers. Specifically, for an encoder with \(L\) layers and an initial value (\(K_0\)), the Top-\(K\) value at layer \(l\) is defined as: \(K_l = \max(1,\lfloor K_0 (1 - ({l-1})/{L} ) \rfloor )\). This linear decreasing strategy allows shallow layers to retain richer character-level candidates for coarse alignment, while deeper layers focus only on the most relevant prompts, thereby improving the precision of feature refinement.

Global-Level: Since each patch represents only the local information within the current image, and typically only high-level encoded features emphasize the text information in the image, global image information is required to supplement this local information. After local differential learning, these patch features not only encapsulate the local information of the image but also incorporate prompt information from the character library. Rather than relying on simple aggregation to obtain global features, we combine the locally augmented patch features \(I \in \mathbb{R}^{N \times d}\) with the CLS embedding \(F_{cls} \in \mathbb{R}^{1 \times d}\). This approach preserves the significance of local information at the global level.

In order to obtain the global embedding of the input image in the global phase, we use the attention mechanism to perform feature interactions between the CLS token and the locally augmented patches. The weight matrix \(\alpha^g\) is derived by calculating the similarity between the CLS embedding and the individual patch embeddings. This matrix indicates how much information from each patch is aggregated into the CLS embedding. Specifically, the CLS embedding and the patch embeddings of the input image are projected into a query vector \(Q = F_{cls}W_Q\), key vectors \(K_i = I_iW_K\), and value vectors \(V_i = I_iW_V\).

To leverage character visual prompts at the global level, we compute each character’s global features \(F = \{F_1, F_2, \dots, F_c\} \in \mathbb{R}^{c \times d}\). These character global features are then used in differential learning with the global features \(I_g\) of the input image, primarily to measure which characters would be useful for the input image. The character library’s global features are weighted by similarity weights \(W_g \in \mathbb{R}^{1 \times c}\) to generate the character prompt vector \(T_g \in \mathbb{R}^{1 \times d}\).

If the global feature \(I_g\) contains only the prompt feature \(T_g\) similar to the character library, the model may over-rely on the character library information, limiting its adaptability to unique character variations in the input image. To address this, we introduce a residual mapping network \(g(\cdot)\) to learn the key differential features \(R_g\) between \(I_g\) and \(T_g\). This approach enables the model to retain the unique visual information of the input image through these residuals, allowing it to handle situations that deviate from the character library with greater accuracy.

3.4. Context-Aware Fusion Decoder

To alleviate exposure bias in autoregressive decoding, we design textual prompts as prior knowledge. This helps the model to further mine the text information hidden in the image feature embedding space for accurate recognition.

We first feed the textual prompts into an encoder to obtain textual embeddings. We then adopt a context fusion strategy to integrate the contextual information contained in the image features with the prior knowledge provided by the textual prompts. In this way, the decoder is guided to focus on plausible character sequences and generate more accurate predictions. Specifically, we first obtain the textual feature encoding \(I_t \in \mathbb{R}^{L \times d}\) (\(L\) is sequence length), then use the learning matrix to map the image feature \(\hat{I}_g\) to \([Q,K,V]=\hat{I}_gW_d\), where \(W_d \in \mathbb{R}^{d \times d}\), and use the direct fusion method to incorporate the textual information into the decoding process of the visual features:

3.5. Training Objective

The training objective of the scene text recognition task is to accurately recognize the text information in the image, and the specific training loss is shown in the following equation:

4.1. Datasets

For the STR task, the availability of the real-world training dataset Union14M-L ⁶ has allowed this research to be pushed further in terms of performance. This dataset contains more than four million labelled images from a variety of real-world scenarios, so we also chose to use Union14M-L for model training. To address current challenges in the STR domain, Union14M-L contains an extensive challenge-driven benchmark (Union14M benchmark) as an evaluation benchmark for real-world scene text. The benchmark consists of seven subsets totaling 409,393 images, characterized by complexity and diversity. These subsets include curved, multi-oriented, artistic, contextless, salient, multi-word, and general text. The performance is also evaluated on six commonly used benchmarks, IIIT 5K-Words (IIIT5K) ⁴², ICDAR2013 (IC13) ⁴³, ICDAR2015 (IC15) ⁴⁴, Street View Text (SVT) ⁴⁵, Street View Text-Perspective (SVTP) ⁴⁶, and CUTE80 (CUTE) ⁴⁷. We also evaluate the model on the more challenging real-world occlusion benchmarks OST (HOST and WOST) ²⁸ and the artistic-text benchmark WordArt ⁴⁸. In addition, we evaluate our model on three larger and more challenging benchmarks: COCO-Text (low-resolution and occluded text) ⁴⁹, ArT (curved and rotated text) ⁵⁰, and Uber-Text (vertical and rotated text) ⁵¹. Comprehensive details regarding these datasets can be found in prior works ^27,15. We use word accuracy ignoring case and symbols (WAICS) ⁶ as the evaluation metric for all compared methods.

4.2. Implementation Details

The size of the patch in ViT is \(4 \times 4\) and the dimension of visual feature is 768 with a depth of 12 where the number of heads used for attention is 12. The dimension of text feature is also 768 with a depth of 6 where the number of heads used for attention is 8. We use the cosine learning rate scheduler without warm-up and the AdamW optimizer with a weight decay of 0.01. We conduct the experiments on eight NVIDIA RTX 4090 GPUs with a batch size of 64 per GPU for 10 epochs. For STR, the vocabulary size \(C\) is set to 96, including 0–9, a–z, A–Z, special characters.

4.3. Comparisons with State-of-the-Arts

Result on Common Benchmark: The comparative performance of VPDL on common benchmark datasets is presented in Table 1. Although these six widely-used benchmarks have been extensively studied and their performance is close to saturation, VPDL remains competitive with recent methods, including several recently published works. Specifically, VPDL achieves the best overall average accuracy (95.9%) and obtains leading performance on SVT, SVTP, and CUTE. On IIIT5K and IC15, our method achieves comparable performance to the strongest baselines. These results indicate that incorporating character-level visual prompts with differential learning provides consistent improvements, even under highly saturated evaluation settings. To further understand the remaining errors, we examined failure cases on IIIT5K and IC15. Most incorrect predictions occur in images with severe blur, low resolution, heavy distortion, or ambiguous character boundaries, where visual cues are extremely limited. Such challenging samples remain difficult for most existing STR methods. We note that different approaches may adopt varying pre-training strategies and training data configurations as reported in their original works. Following standard practice, we compare results under the commonly reported benchmark settings to ensure fair and consistent evaluation.

Result on Union14M Benchmark: As one of the newest and most realistic benchmarks in the field of scene text recognition, the Union14M benchmark provides a robust validation of the effectiveness and robustness of our method. The results are shown in Fig. 4 and Table 2. The benchmark contains datasets from seven real scene categories, with large gaps and challenges in different scene categories, and from the results our model outperforms the current SOTA methods in most scene categories, although it does not achieve the best results in all categories.

(1) Our method achieves the best results in challenging scenarios such as contextless and multi-word text. We attribute this improvement to two factors. First, the character-level visual prompts help the model focus on fine-grained character cues. Second, the textual prompts used in autoregressive decoding alleviate exposure bias, which is particularly important for semantic-free or long text sequences, where an early prediction error may otherwise propagate through the entire sequence.

(2) The inference results of some scenes from Fig. 4 as well as Table 2 show that our method is slightly less effective in curved and multi-oriented text scenes, and we hypothesize that this result occurs because these non-standard text shapes increase the visual complexity to a great extent compared with regular text images arranged in horizontal sequences, and instantly, we use character visual prompts to allow the model to understand as much as possible through differential learning. But the fact that the character shapes change irregularly with the images leads to a large difference between the learned semantic features and the levels, so it still poses a great challenge to recognize the corresponding visual features.

4.4. Result on More Complicated Datasets

Small-Scale Benchmark: We conducted comparison experiments on more complex and challenging scene text datasets, including the occlusion datasets WOST and HOST as well as the artistic-text dataset WordArt. We chose the models that reported the evaluation results of these datasets in the paper for comparison, and the results are shown in Table 3, where our model outperforms all the compared modeling approaches, achieving 1.46% and 0.91% in average performance improvement compared to DARLING ⁵ and MAERec ⁶, respectively. The designed character-level visual prompts simulate occlusion-like variations during data augmentation, enabling the model to learn more robust representations through differential learning. As a result, VPDL achieves stronger performance in these complex scenarios and demonstrates better generalization ability than the compared baseline models.

Large-Scale Benchmark: We also conducted evaluations on three larger and more challenging benchmarks involving low-resolution, occluded, and rotated text. The results are shown in Table 4. Our results demonstrate that VPDL consistently outperforms numerous baseline models, highlighting its strong generalization ability across diverse scene text recognition scenarios. However, compared with its performance on these three large-scale benchmarks, our model shows relatively smaller gains on the curve and multi-oriented categories of Union14M benchmark. This leads us to two key insights. First, we believe this is because rotated text undergoes a global transformation while maintaining a structured layout, whereas curved text involves local transformations that introduce complex character distortions, making it inherently more difficult to process. Second, while our model demonstrates strong cross-scenario generalization in most cases, its performance in curved and rotated text scenarios still requires further improvement.

4.5. Ablation Study

The Evaluation of Differential Learning: In order to verify the effectiveness of this module, we added local and global differential learning to ViT respectively, and the results are shown in Table 5. After incorporating local differential learning, the model demonstrates a notable improvement in performance compared to ViT. This enhancement stems primarily from the ability to enrich each input image patch with more contextually similar visual prompts. For instance, characters with varying curvatures, like different forms of the letter “C,” should share common features. This approach enables the model to better capture and utilize feature information that was previously less effectively learned. Adding a global differential learning layer directly to ViT yields better results than using a local layer. We attribute this improvement to the more comprehensive capture of text-related information in the globally obtained image features. Our residual mapping network further enhances the model’s ability to focus not only on similarities with character features but also on the distinctive information of the input image itself at a global level. This broader focus enables the model to adapt more effectively to scenarios that differ from the character prompts, resulting in improved performance. In summary, the local-to-global differential learning design, based on character visual prompts, indeed enhances the model’s ability to learn more discriminative feature representations.

The Evaluation of Text Prompt: To verify the effectiveness of the textual prompts, we compare the results obtained using the original label sequence with those obtained using our textual prompts, as shown in the w/o TP setting in Table 5. Without textual prompts, the performance decreases by 1.24%, which confirms their effectiveness. Compared with directly using labels as decoder inputs, textual prompts encourage the model to focus more on the textual information embedded in the image features. This not only reduces interference from irrelevant features during decoding, but also alleviates error propagation from previously decoded characters, leading to more accurate recognition results.

The Evaluation of Image Prompts Size: Although, in principle, a more diverse set of character samples may provide richer visual references, our experiments show that increasing the number of prompts does not always lead to consistent gains, especially in scenarios such as multi-oriented text. To evaluate the effect of prompt pool size, we conduct experiments on three challenging datasets while varying the number of prompt samples per character. As shown in Fig. 5, introducing visual prompts consistently improves performance. For example, on WOST, the accuracy increases from 81.02% to 83.00% when the number of prompts per character increases from 5 to 20. However, the gains gradually saturate as more samples are added, suggesting that only a moderate number of prompt samples is sufficient to provide effective visual guidance.

The Effect of Initial Top-\(\boldsymbol{K}\) Value: To evaluate the rationality of the Top-\(K\) configuration in local differential learning, we vary the initial value \(K_0\) while keeping the linear decreasing schedule unchanged. Specifically, we test \(𝐾_0 \in \{1,5,9,12\}\) on three challenging benchmarks (WOST, HOST, WordArt). As shown in Table 6, using a small value significantly limits multi-candidate alignment and leads to inferior performance. Increasing \(K_0\) improves accuracy by allowing richer prompt interaction, and the best performance is achieved at \(K_0=9\). Further increasing \(K_0\) to 12 does not bring additional gains, likely due to increased noise from less relevant prompts. These results validate the rationality of our Top-\(K\) design.

4.6. Effect of Different Text Prompt

To validate the impact of text prompts on auto-regressive models, we designed experiments using different types of text prompts to compare text-guided fusion with others. Specifically, during training, we used three types of prompts: TP1: “The text starts with {P} and ends with {y}. Full text is {Party},” TP2: “The text in images is {Party},” and TP3: “Party.” The results, obtained using the same decoder, are shown in Table 7. From the table, we observe that not all text prompts effectively help the model accurately recognize image-based textual features. Unlike direct character embeddings, fixed-format text prompts in STR provide character-level information while introducing additional language constraints and semantic priors during decoding. This helps the model make more accurate inferences when dealing with blurred or ambiguous characters. Moreover, text prompts strengthen the connection between visual features and text space, reducing the modality gap and enabling the model to better learn semantic information from image features. Additionally, we explored the performance improvements brought by different text decoders. Since these decoders can generate better feature representations for long text sequences, they ultimately contribute to more effective fusion with image features, leading to improved results.