Top > JACIII > Mixture of Experts Approach for Domain Adaptation in Finance

single-jc.php

JACIII Vol.30 No.4 pp. 1093-1101
(2026)
doi: 10.20965/jaciii.2026.p1093

Research Paper:

Views over last 60 days: 231

Mixture of Experts Approach for Domain Adaptation in Finance

Masahiro Suzuki* ORCID Icon, Hiroki Sakaji** ORCID Icon, and Kiyoshi Izumi* ORCID Icon

*The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan

**Hokkaido University
Kita 14, Nishi 9, Kita-ku, Sapporo, Hokkaido 060-0814, Japan

Received:
June 29, 2025
Accepted:
February 11, 2026
Published:
July 20, 2026
Creative Commons License
Keywords:
Mixture of Experts, domain adaptation, financial text mining, pre-trained language model
Abstract

Pre-trained language models (PLMs) have demonstrated high performance across various tasks and domains. Among these PLMs, Mixture of Experts (MoE) models also exhibit high performance with fewer active parameters. In domain adaptation, generally, continual pre-training is performed with existing models using domain-specific corpora. However, few efforts have been made to transform and train these models into models with MoE architectures. We propose a method to construct domain-adapted MoE models from general pre-trained models that do not initially have MoE architectures. By independently training multiple experts using domain corpora and integrating them into an MoE architecture, we constructed a domain-adapted MoE model. We performed this MoE transformation in the financial domain and verified its effectiveness in financial tasks. The evaluation results indicate that our domain-adapted MoE models perform better than those without MoE architectures. Our domain-adapted MoE models achieved consistent improvements over domain-adaptive pretraining, task-adaptive pretraining, and domain- and task-adaptive pretraining, with an average improvement of approximately 0.08 in F1 across six financial benchmark tasks for encoder-based models.

Two-stage MoE domain adaptation

Two-stage MoE domain adaptation

Close full text
Cite this article as:
M. Suzuki, H. Sakaji, and K. Izumi, “Mixture of Experts Approach for Domain Adaptation in Finance,” J. Adv. Comput. Intell. Intell. Inform., Vol.30 No.4, pp. 1093-1101, 2026.
Data files:

1. Introduction

Pre-trained language models (PLMs), including large language models (LLMs), have demonstrated high performance across various tasks in natural language processing 1,2. With the rapid evolution of language models, numerous models have been proposed and released, and these models are increasingly being applied to specialized domains such as finance 3. However, directly applying general-purpose language models to the financial domain may not always yield satisfactory performance.

Various approaches have been proposed to construct domain-specific language models 4,5,6. One method of adapting a general-purpose model to a specific domain is to use a domain-specific corpus during pre-training. In the financial domain, pre-training from scratch or continual pre-training using a financial corpus can adapt a general-purpose model to the financial domain 7,8,9. However, as language models become larger, the cost of pre-training increases, necessitating more efficient adaptation methods 10,11.

The Mixture of Experts (MoE) model 12 has recently garnered significant attention. The MoE model features multiple pathways for processing the input and selects different modules used for each input. This architecture allows the model to maintain a large number of parameters while reducing the number of parameters used per input, thereby demonstrating a high parameter efficiency. Relatively few publicly available MoE models exist, including Mixtral 13 and Switch Transformers 14, and most existing models are not MoE models.

We propose a method for efficiently constructing domain-adapted MoE models from non-MoE general-purpose pre-trained models. In domain adaptation, continual pre-training is typically employed, and few attempts have been made to efficiently perform domain adaptation by improving the architecture of existing models 6,15. We extend publicly available general-purpose models to construct domain-adapted MoE models. While continual pre-training improves performance, our method introduces conditional computations through architectural transformation. Specifically, we independently trained multiple experts on models using financial corpora that did not have an MoE architecture. Subsequently, we constructed a finance-adapted MoE model by aggregating these experts into an MoE architecture and training the gating networks. By maximizing the reuse of the existing PLM weights and avoiding the simultaneous training of experts, the proposed method alleviated GPU memory usage and computational requirements, enabling practical and parameter-efficient domain adaptation without training the MoE models from scratch. Unlike prior MoE studies that assumed MoE architectures from scratch, we addressed the practically important yet underexplored problem of transforming existing non-MoE PLMs into domain-adapted MoE models. We selected the financial domain as the target domain for adaptation because it has a well-established variety of tasks and is easy to evaluate the performance on both encoder and decoder models. We evaluated the finance-adapted MoE models for financial tasks. The results of the financial evaluation tasks showed that our MoE transformation method was effective for adapting language models to the financial domain. Notably, our experiments showed that the domain-adaptive MoE (D-MoE), trained exclusively on financial corpora, consistently achieved the best performance for encoder-based models. This indicated that domain-level expert specialization combined with sparse routing can provide stable and effective adaptation, even without task-specific data.

The contributions of this study are as follows:

  • We proposed a method for constructing a domain-adapted model with an MoE structure.

  • We demonstrated the effectiveness of our transformation from an existing pre-trained model without an MoE structure into an MoE model in the financial domain.

  • Our MoE encoder model, constructed using the financial corpus, outperformed the domain-adapted model without an MoE structure for financial tasks, demonstrating the effectiveness of our transformation into the domain-adapted MoE architecture.

3. Method

In this study, we construct a domain-adapted MoE model from an existing general-purpose language model without an MoE architecture. We independently train each expert using a domain corpus and then integrate the experts into the MoE architecture. The transformation process to the MoE model comprises two stages: Stage 1, in which each expert is trained using a domain corpus, and Stage 2, in which the gating networks are adjusted. In Stage 1 (Fig. 1(a)), we train the feed-forward networks (FFNs) in the transformer layers as experts using the domain corpus, as in 20.

In Stage 2 (Fig. 1(b)), we combine the experts and integrate them with the gating networks. The gating networks select which expert to route an input to. For each token, output vector \(x\) from the attention layer is processed using the MoE architecture. The gating networks weight and sum the outputs from \(n\) experts to produce final output vector \(y\) as follows:

\begin{equation*} y = \sum_{i=1}^{n} G(x)_{i} \cdot E_{i}(x). \end{equation*}

Here, \(G(x)_i\) represents the probability where the token selects the \(i\)-th expert, and \(E_i(x)\) denotes the output vector from the \(i\)-th expert. \(n\) denotes the number of experts. To reduce computational complexity, \(G(x)\) works as the gating network and is computed using a sparse gate 12, where the top-k is applied to the output from a linear layer, followed by the softmax function as follows:

\begin{equation*} G(x) = \textrm{Softmax}(\textrm{TopK}(\textrm{Linear}(x), k)). \end{equation*}

Here, the \(i\)-th element of variable \(v\) after applying top-k is defined as follows:

\begin{equation*} \textrm{TopK}(v, k)_{i} = \left\{ \begin{aligned} &v_{i}, &&\parbox[t]{100mm}{if $v_i$, $i$-th element of $v$, is\\ in the top-$k$ values of $v$,} \\ &-\infty, &&\mbox{otherwise.} \end{aligned}\right. \end{equation*}

For the loss calculation in Stage 2, in addition to the loss used in the original model, \(\mathcal{L}_{LM}\), we apply load balancing loss \(\mathcal{L}_{LB}\) 14 to maintain the balance of the experts. \(\mathcal{L}_{LB}\) is multiplied by 0.02 and added to \(\mathcal{L}_{LM}\) to form loss \(\mathcal{L}\) during training.

By changing the type of corpus used for training in Stage 2, we propose three types of MoE models:

  • Domain-adaptive MoE (D-MoE)

  • Task-adaptive MoE (T-MoE)

  • Domain and task adaptive MoE (D&T-MoE)

In Stage 1, for all the three MoE models, the FFNs of the base models are trained using a domain corpus. D-MoE and T-MoE are trained in Stage 2 using the domain and task corpora, respectively. In Stage 2 of D&T-MoE training, the gating networks and FFNs are first trained using the domain corpus and then further trained using the task corpus. Consequently, because the gating networks are adjusted using both the domain and task corpora, D&T-MoE is expected to outperform T-MoE, which adjusts the gating networks using only the task corpus. We freeze all the other parameters, such as embedding, attention, and language model head layers, throughout all the stages.

4. Experiment

Table 1. Details of pre-training datasets.

figure

Table 2. Parameters for MoE training.

figure

4.1. Pre-Training Datasets

We used four English financial domain corpora to train each expert: 1) 10-K and 10-Q reports, 2) earning call transcripts, 3) financial news from Reuters, and 4) financial news from Investing.com. Table 1 presents the details of domain corpora. From each corpus, 500k samples of 512 tokens were randomly extracted and used to train each expert.

10-K and 10-Q Reports

Many U.S. companies submit their annual reports known as 10-K and quarterly reports known as 10-Q to the U.S. Securities and Exchange Commission (SEC). We used 10-K and 10-Q filings from Q1 of 1993 to Q1 of 2024. They are authorized by the SEC to be copied and distributed.(1)

Earning Call Transcripts

Earnings calls are occasions where companies report their performance to shareholders and investors on a quarterly basis. Many transcripts of earnings calls conducted by U.S. companies are available from various sources. We used data on earnings call transcripts issued between April 2004 and February 2024 from Seeking Alpha(2) according to its terms of service.(3)

Financial News from Reuters

We used English news data published between January 1996 and February 2023 provided by Reuters.

Financial News from Investing.com

Investing.com(4) is a portal site that provides news, analyses, technical data, and financial tools related to international financial markets. We used English news data issued on Investing.com from October 2008 to February 2024 according to its terms of service.(5)

4.2. Pre-Training with MoE Architecture

As described in Section 3, the training and construction of the MoE model were performed in two stages. In Stage 1, only the FFNs were trained for 50k steps using each financial corpus, as described in Section 4.1. We prepared four financial domain corpora and independently trained four experts. In Stage 2, the corpus of each model and the number of training steps or epochs differed. D-MoE was trained for 10k steps using a combination of four financial corpora. T-MoE was trained for 20 epochs using the training portion of each evaluation task. D&T-MoE was trained for 10k steps using the financial corpus, followed by 20 epochs using the training portion of the evaluation task.

We used three model architectures to construct the MoE models: DeBERTaV3 32, T5 33, and Llama 3 34. DeBERTaV3 is based on a transformer encoder, T5 uses both an encoder and a decoder, and Llama 3 is based on a decoder. We used these different architectures to evaluate the effectiveness of the MoE transformation. The model sizes were base for DeBERTaV3, large for T5, and 8B for Llama 3. The number of experts was set to five: four experts trained with the financial corpora used in Stage 1 and the base expert of the original general-purpose model.

The implementation of the gating networks was based on 13. The number of experts selected by top-k was set to \(k=2\). Bfloat 16 precision was used for all training and evaluation. The implementation used for model construction is made public on GitHub.(6)

The hyperparameters used to construct the MoE model are listed in Table 2. The optimizer was set to AdamW with the default settings of the Transformers library 35. The learning rate was linearly increased up to 5% of the total number of steps and then linearly decreased. To improve the computational efficiency, we used the ZeRO module from DeepSpeed 36. We used ZeRO-1 for DeBERTaV3 and T5 and ZeRO-2 for Llama 3. For the training of Llama 3 in Stage 1, we used three NVIDIA A100 80 GB GPUs. We used two NVIDIA A6000 Ada GPUs to train other models in Stage 1 and Stage 2. The training of Llama 3 and T5 in Stage 1 required approximately three days, and the training of DeBERTaV3 required approximately 17 hours.

For the original pre-trained models used in this study, Microsoft licenses DeBERTaV3 under the MIT license, Google licenses T5 under the Apache 2.0 license, and Meta licenses Llama 3 under the Llama 3 license.(7)

4.3. Baselines

As baselines, we used three models: domain-adaptive (DAPT), task-adaptive (TAPT), and combined DAPT+TAPT (D&T-APT) 4. DAPT and TAPT are models that continue to pre-train the original general-purpose model on financial and task corpora, respectively. D&T-APT is a model that continues to pre-train the original model on a financial corpus and then continues to pre-train on a task corpus. The corpus described in Section 4.1 is combined and used in the continual pre-training. The parameters for the baseline models in continual pre-training using the financial and task corpora were the same as those in Stages 1 and 2 in Table 2, respectively.

Table 3. Evaluation tasks.

figure

4.4. Evaluation

We evaluated the constructed MoE and baseline models using six English financial domain tasks. The tasks comprised three types: 1) sentiment classification using Financial Phrase Bank (FPB) 37 and FiQA-SA 38, 2) category classification using Headline 39 and Federal Open Market Committee (FOMC) 40, and 3) named-entity recognition using NER 41 and FiNER-ORD 42. Financial encoder models such as FLANG-BERT and FLANG-ELECTRA 8 often focus on classification tasks at the sentence, document, and token levels. Although various tasks are implemented in decoder models like FinBen 43, this study focused on the aforementioned tasks to evaluate both the encoder and decoder models.

Table 3 presents the details of the evaluation tasks. FPB 37 and FiQA-SA 38 are tasks that classify the sentiment of financial news headlines and texts extracted from blogs, news statements, and headlines into three categories: positive, negative, and neutral. Although the FiQA-SA task includes sentiment scores, this study evaluated models on it as a sentiment classification task. Headline 39 is a task that classifies whether news related to gold belongs to one of nine categories, such as whether it discusses price increases or is based on past prices. FOMC 40 is a task that classifies FOMC meeting minutes into one of three stances: hawkish, dovish, or neutral. NER 41 is a seven-value classification task that tags tokens in documents related to credit risk. FiNER-ORD 42 is a similar task for news articles, tagging tokens as person (PER), location (LOC), or organization (ORG) using the BIO scheme.

The evaluation metrics used were the F1 score for FPB, FiQA-SA, and FOMC; average F1 score across the nine categories for Headline; and entity F1 score for NER and FiNER-ORD.

Table 4. Fine-tuning parameters.

figure

Fine-tuning was performed for DeBERTaV3 and T5. In evaluating DeBERTaV3 and T5, we reported the average of the fine-tuning results obtained with five different seed values. The parameters used to fine-tune the evaluation tasks for DeBERTaV3 and T5 are listed in Table 4. The parameters for the optimizer and learning rate were the same as those described in Section 4.2. The load balancing loss was not used during the fine-tuning of DeBERTaV3 and T5.

For Llama 3, the implementation of FinBen(8) was used, and the evaluations were conducted in one-shot and three-shot settings.

5. Results and Discussions

Table 5 presents the average evaluation results. In DeBERTaV3 and T5, the average D-MoE scores were the highest. The MoE architecture provided performance benefits for DeBERTaV3 and T5. Additionally, the MoE models consistently ranked among the top performers for both DeBERTaV3 and T5, further demonstrating the effectiveness of the MoE architecture. Among the MoE models, the D-MoE model exhibited the best performance. Generally, the models continually pre-trained with domain and task corpora, such as D&T-APT, yield better performance than DAPT 4. This is also observed in the baseline models. However, in the MoE architecture, the D-MoE model, which was trained only with the domain corpus, showed higher performance than the D&T-MoE model, which was trained using both the domain and task corpora. This could be attributed to training instability with a smaller task corpus, because the increased number of parameters in the gating networks challenged MoE architecture construction. The MoE models, particularly in T5, gained more performance benefits compared with the baseline methods than in DeBERTaV3, suggesting that they might perform better in models with encoder architectures with a larger number of parameters or a larger number of frozen parameters.

Table 5. Average evaluation results. Bold and underline indicate the first and second best performance in the same base model and number of few-shots, respectively.

figure

In Llama 3, the average score of TAPT was the highest. The second highest score was achieved by D&T-MoE, indicating that MoE also had a certain effect. Two possible reasons existed for the lower effectiveness of the MoE architecture. The MoE model might not work best because experts were explicitly assigned to the specific corpora; this might not be optimal. The autoregressive generation process in the decoder models might have caused more difficulty in controlling the routing for each expert in MoE to collaboratively generate consistent outputs. In particular, the expert specialized for each domain corpus is focused on specific knowledge domains rather than general generation capabilities for diverse tasks, which might have prevented it from fully demonstrating its capabilities in the decoder’s generation process. Although the proposed expert-wise training reduced GPU memory usage by avoiding simultaneous FFN training, training without explicit role assignment for each expert might be more efficient 12. The second reason was that the MoE architecture with a large number of parameters might require additional training steps. The MoE model might have insufficient training of the gating networks because the number for training steps after transforming the architecture into the MoE architecture was fewer than that of the baseline continual pre-training. Autoregressive generation in the decoder models required MoE gate control that considered dependencies on previous tokens; this might have made optimization more difficult than in the encoders. Because the few-shot tasks used with Llama 3 required text generation capabilities, the high inherent generation and instruction comprehension capabilities of the base model were dominant in the decoders. Although the addition of specialized knowledge through MoE was effective for encoder models, the performance improvement of the decoder from MoE might have been limited. Training with the MoE architecture using domain corpora from the beginning might improve the results because the model could implicitly learn the roles of experts. Among the Llama 3 MoE models, D&T-MoE exhibited the best performance. This led to the inference that using both domain and task corpora was effective for the MoE architecture of the decoder model.

The detailed results for the evaluation tasks are presented in Table 6. Many models exhibited particularly low accuracy on NER and FiNER-ORD tasks when using Llama 3. During continual pre-training with the task corpus, we aligned the input lengths with the longest sample in the training dataset. This applied to both the baseline methods (TAPT and D&T-APT) and MoE models (T-MoE and D&T-MoE). Therefore, in few-shot input scenarios, the input length often exceeded that used during continual pre-training. In particular, NER and FiNER-ORD performed tagging for each word in the input sentence, requiring more input tokens than other sentence classification tasks when inputting in few-shot scenarios. Consequently, the continual pre-training conducted in this study could not be performed with sufficiently long input samples, potentially diminishing its effectiveness. Although reducing the number of samples that can be used for continual pre-training can be disadvantageous, we believe that the input length issue can be addressed by combining multiple samples for training.

Table 6. Detailed evaluation results. Bold indicates the best performance in the same base model and the same number of few-shots.

figure

6. Limitations

6.1. Diversity of Domain and Task Corpora

Of the six evaluation tasks, five (excluding FOMC) were constructed from datasets similar to the financial corpus used in the pre-training. FPB, FiQA-SA, Headline, and FiNER-ORD were partially or entirely constructed from news articles, whereas NER was constructed from the data submitted to the U.S. SEC. In Stage 1, the stage of expert construction using the domain corpus, experts might have already been specialized for those tasks. Thus, the D-MoE model using only the domain corpus could exhibit high performance, as discussed in Section 5. Although the load balancing loss prevents over-reliance on specific experts, analyzing which experts (news-trained vs. SEC-trained) are most frequently used during fine-tuning or few-shot inference remains a future task.

Evaluating tasks constructed from corpora with characteristics different from that of our financial corpus remains a topic for future work.

6.2. Limited Parameter Training in T5 and Llama 3

In T5 and Llama 3, due to computational resource constraints, certain parameters in the early layers of the model were fixed during pre-training, preventing full-model training. Evaluating the MoE performance with fully trainable T5 and Llama 3 models remains a topic for future work.

6.3. Instruction Tuning for Llama 3

In this study, we used the base model of Llama 3 as the foundational model. This model solves only a causal language modeling task during pre-training and does not undergo any fine-tuning. Causal language models improve their task response performance in few-shot scenarios through instruction tuning 26. Because we did not use an instruction-tuned model in this study, the evaluation of instruction-tuned models remains future work.

7. Conclusions

We proposed a method for constructing domain-adapted MoE models from existing non-MoE architectures. We independently trained experts using domain corpora from an existing model. We integrated these experts and trained the gating networks to construct the MoE model. Our experiments in the financial domain demonstrated that the MoE models outperformed the continual pre-training approaches for DeBERTaV3 and T5. Notably, the MoE models trained exclusively on domain corpora achieved strong performance without requiring task-specific data. This contrasted with continual pre-training in which combining domain and task corpora yielded the best results. However, for Llama 3, continual pre-training achieved superior performance than our MoE approach. This study provides new directions for efficient domain adaptation through MoE transformation, particularly for encoder-based models.

Acknowledgments

This work was supported by JST PRESTO Grant Number JPMJPR2267, Japan.

Footnotes

(1) https://www.sec.gov/privacy#dissemination [Accessed June 27, 2026]

(2) https://seekingalpha.com/ [Accessed June 27, 2026]

(3) https://about.seekingalpha.com/terms [Accessed June 27, 2026]

(4) https://www.investing.com/ [Accessed June 27, 2026]

(6) https://github.com/retarfi/FinDaLM [Accessed June 27, 2026]

(8) https://github.com/The-FinAI/PIXIU [Accessed June 27, 2026]

References
  1. [1] OpenAI, “GPT-4 technical report,” arXiv:2303.08774, 2024. https://doi.org/10.48550/arXiv.2303.08774
  2. [2] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv:2307.09288, 2023. https://doi.org/10.48550/arXiv.2307.09288
  3. [3] X. Li et al., “Are ChatGPT and GPT-4 general-purpose solvers for financial text analytics? A study on several typical tasks,” Proc. of the 2023 Conf. on Empirical Methods in Natural Language Processing: Industry Track, pp. 408-422, 2023. https://doi.org/10.18653/v1/2023.emnlp-industry.39
  4. [4] S. Gururangan et al., “Don’t stop pretraining: Adapt language models to domains and tasks,” Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, 2020. https://doi.org/10.18653/v1/2020.acl-main.740
  5. [5] K. Singhal et al., “Large language models encode clinical knowledge,” Nature, Vol.620, No.7972, pp. 172-180, 2023. https://doi.org/10.1038/s41586-023-06291-2
  6. [6] Y. Xie, K. Aggarwal, and A. Ahmad, “Efficient continual pre-training for building domain specific large language models,” Findings of the Association for Computational Linguistics: ACL 2024, pp. 10184-10201, 2024. https://doi.org/10.18653/v1/2024.findings-acl.606
  7. [7] D. Araci, “FinBERT: Financial sentiment analysis with pre-trained language models,” arXiv:1908.10063, 2019. https://doi.org/10.48550/arXiv.1908.10063
  8. [8] R. Shah et al., “When FLUE meets FLANG: Benchmarks and large pretrained language model for financial domain,” Proc. of the 2022 Conf. on Empirical Methods in Natural Language Processing, pp. 2322-2335, 2022. https://doi.org/10.18653/v1/2022.emnlp-main.148
  9. [9] S. Wu et al., “BloombergGPT: A large language model for finance,” arXiv:2303.17564, 2023. https://doi.org/10.48550/arXiv.2303.17564
  10. [10] N. Pahari and K. Shimada, “Layer configurations of BERT for multitask learning and data augmentation,” J. Adv. Comput. Intell. Intell. Inform., Vol.28, No.1, pp. 29-40, 2024. https://doi.org/10.20965/jaciii.2024.p0029
  11. [11] S. O. Khairunnisa, Z. Chen, and M. Komachi, “Improving domain-specific NER in the Indonesian language through domain transfer and data augmentation,” J. Adv. Comput. Intell. Intell. Inform., Vol.28, No.6, pp. 1299-1312, 2024. https://doi.org/10.20965/jaciii.2024.p1299
  12. [12] N. Shazeer et al., “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 5th Int. Conf. on Learning Representations, 2017. https://openreview.net/forum?id=B1ckMDqlg [Accessed June 27, 2026]
  13. [13] A. Q. Jiang et al., “Mixtral of experts,” arXiv:2401.04088, 2024. https://doi.org/10.48550/arXiv.2401.04088
  14. [14] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” J. of Machine Learning Research, Vol.23, Article No.120, 2022.
  15. [15] M. Suzuki, H. Sakaji, M. Hirano, and K. Izumi, “FinDeBERTaV2: Word-segmentation-free pre-trained language model for finance,” Trans. of the Japanese Society for Artificial Intelligence, Vol.39, No.4, pp. FIN23-G_1-14, 2024 (in Japanese). https://doi.org/10.1527/tjsai.39-4_FIN23-G
  16. [16] D. Lepikhin et al., “GShard: Scaling giant models with conditional computation and automatic sharding,” 9th Int. Conf. on Learning Representations, 2021. https://openreview.net/forum?id=qrwe7XHTmYb [Accessed June 27, 2026]
  17. [17] Y. Zhou et al., “Mixture-of-experts with expert choice routing,” Proc. of the 36th Int. Conf. on Neural Information Processing Systems, pp. 7103-7114, 2022.
  18. [18] J. Lee-Thorp and J. Ainslie, “Sparse mixers: Combining MoE and mixing to build a more efficient BERT,” Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 58-75, 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.5
  19. [19] N. Du et al., “GLaM: Efficient scaling of language models with mixture-of-experts,” Proc. of the 39th Int. Conf. on Machine Learning, pp. 5547-5569, 2022.
  20. [20] S. Diao, T. Xu, R. Xu, J. Wang, and T. Zhang, “Mixture-of-domain-adapters: Decoupling and injecting domain knowledge to pre-trained language models’ memories,” Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Vol.1: Long Papers), pp. 5113-5129, 2023. https://doi.org/10.18653/v1/2023.acl-long.280
  21. [21] M. Suzuki, H. Sakaji, M. Hirano, and K. Izumi, “Constructing and analyzing domain-specific language model for financial text mining,” Information Processing & Management, Vol.60, No.2, Article No.103194, 2023. https://doi.org/https://doi.org/10.1016/j.ipm.2022.103194
  22. [22] Z. Liu, D. Huang, K. Huang, Z. Li, and J. Zhao, “FinBERT: A pre-trained financial language representation model for financial text mining,” Proc. of the 29th Int. Joint Conf. on Artificial Intelligence, pp. 4513-4519, 2020. https://doi.org/10.24963/ijcai.2020/622
  23. [23] J. Hoffmann et al., “Training compute-optimal large language models,” Proc. of the 36th Int. Conf. on Neural Information Processing Systems, pp. 30016-30030, 2022.
  24. [24] BigScience Workshop (T. L. Scao et al.), “BLOOM: A 176B-parameter open-access multilingual language model,” arXiv:2211.05100, 2023. https://doi.org/10.48550/arXiv.2211.05100
  25. [25] S. Biderman et al., “Pythia: A suite for analyzing large language models across training and scaling,” Proc. of the 40th Int. Conf. on Machine Learning, pp. 2397-2430, 2023.
  26. [26] J. Wei et al., “Finetuned language models are zero-shot learners,” 10th Int. Conf. on Learning Representations, 2022. https://openreview.net/forum?id=gEZrGCozdqR [Accessed June 27, 2026]
  27. [27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017. https://doi.org/10.48550/arXiv.1707.06347
  28. [28] Y. Bai et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv:2204.05862, 2022. https://doi.org/10.48550/arXiv.2204.05862
  29. [29] L. Ouyang et al., “Training language models to follow instructions with human feedback,” Proc. of the 36th Int. Conf. on Neural Information Processing Systems, pp. 27730-27744, 2022.
  30. [30] Q. Xie et al., “PIXIU: A comprehensive benchmark, instruction dataset and large language model for finance,” 37th Conf. on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. https://openreview.net/forum?id=vTrRq6vCQH [Accessed June 27, 2026]
  31. [31] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023. https://doi.org/10.48550/arXiv.2302.13971
  32. [32] P. He, J. Gao, and W. Chen, “DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” The 11th Int. Conf. on Learning Representations, 2023. https://openreview.net/forum?id=sE7-XhLxHA [Accessed June 27, 2026]
  33. [33] C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. of Machine Learning Research, Vol.21, Article No.140, 2020.
  34. [34] AI@Meta, “Llama 3 model card,” 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md [Accessed June 27, 2026]
  35. [35] T. Wolf et al., “Transformers: State-of-the-art natural language processing,” Proc. of the 2020 Conf. on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 2020. https://doi.org/10.18653/v1/2020.emnlp-demos.6
  36. [36] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters,” Proc. of the 26th ACM SIGKDD Int. Conf. on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020. https://doi.org/10.1145/3394486.3406703
  37. [37] P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala, “Good debt or bad debt: Detecting semantic orientations in economic texts,” J. of the Association for Information Science and Technology, Vol.65, No.4, pp. 782-796, 2014. https://doi.org/10.1002/asi.23062
  38. [38] M. Maia et al., “WWW’18 Open Challenge: Financial opinion mining and question answering,” Companion Proc. of the Web Conf. 2018, pp. 1941-1942, 2018. https://doi.org/10.1145/3184558.3192301
  39. [39] A. Sinha and T. Khandait, “Impact of news on the commodity market: Dataset and results,” Advances in Information and Communication: Proc. of the 2021 Future of Information and Communication Conf., Vol.2, pp. 589-601, 2021. https://doi.org/10.1007/978-3-030-73103-8_41
  40. [40] A. Shah, S. Paturi, and S. Chava, “Trillion dollar words: A new financial dataset, task & market analysis,” Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Vol.1: Long Papers), pp. 6664-6679, 2023. https://doi.org/10.18653/v1/2023.acl-long.368
  41. [41] J. C. S. Alvarado, K. Verspoor, and T. Baldwin, “Domain adaption of named entity recognition to support credit risk assessment,” Proc. of the Australasian Language Technology Association Workshop 2015, pp. 84-90, 2015.
  42. [42] A. Shah, A. Gullapalli, R. Vithani, M. Galarnyk, and S. Chava, “FiNER: Financial named entity recognition dataset and weak-supervision model,” arXiv:2302.11157v1, 2023. https://doi.org/10.48550/arXiv.2302.11157
  43. [43] Q. Xie et al., “FinBen: An holistic financial benchmark for large language models,” arXiv:2402.12659, 2024. https://doi.org/10.48550/arXiv.2402.12659

Creative Commons License  This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Jul. 19, 2026