Research Paper:
Topic Modelling for Biomedical Literature Retrieval via a Dual-Stream Embedding Framework
Tao Tao*
, Amena Mahmoud**
, and Eiko Takaoka*

*Graduate School of Science and Technology, Sophia University
7-1 Kioi-cho, Chiyoda-ku, Tokyo 102-8554, Japan
**Department of Computer Science, Faculty of Computers and Information, Kafrelsheikh University
El-Geish Street, Kafrelsheikh 33516, Egypt
The rapid expansion of biomedical literature has created substantial challenges in retrieving and synthesizing relevant knowledge, particularly for novice and interdisciplinary users. Conventional keyword-based methods often lack semantic depth, whereas traditional topic modelling techniques struggle to capture domain-specific contexts and rarely integrate structured metadata, such as medical subject headings (MeSH). To address these limitations, this study introduces a dual-stream embedding framework that integrates unstructured textual content with curated biomedical metadata using transformer-based encoders and a parameterized fusion mechanism. The architecture combines contextual semantics from article titles and abstracts with expert-annotated MeSH descriptors and incorporates a large language model to generate concise and human-readable topic labels. Experimental results show that three of the four evaluated embedding configurations outperformed the single-stream baseline and that the dual-stream fusion surpassed all ablation models built from the same inputs. The best configuration, Dual PubMedBERT, achieved the highest overall topic modelling quality, improving the combined measure of topic coherence and topic diversity by 7.3% relative to the baseline. These results demonstrate the effectiveness of integrating contextual textual embeddings with structured biomedical metadata using the proposed fusion mechanism. This approach is scalable, reproducible, and domain-adaptable, providing both a methodological advancement in semantic topic modelling and a practical foundation for the development of biomedical information retrieval systems, semantic search engines, and thematic exploration tools across knowledge-intensive domains.
Dual-stream fusion of text and MeSH
- [1] Q. Jin, R. Leaman, and Z. Lu, “PubMed and beyond: biomedical literature search in the age of artificial intelligence,” eBioMedicine, Vol.100, Article No.104988, 2024. https://doi.org/10.1016/j.ebiom.2024.104988
- [2] H. C. Huurdeman and J. Kamps, “Designing Multistage Search Systems to Support the Information Seeking Process,” W. T. Fu and H. van Oostendorp (Eds.), “Understanding and Improving Information Search,” pp. 113-137, Springer, 2020. https://doi.org/10.1007/978-3-030-38825-6_7
- [3] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” J. of Machine Learning Research, Vol.3, pp. 993-1022, 2003.
- [4] D. M. Blei, “Probabilistic topic models,” Communications of the ACM, Vol.55, Issue 4, pp. 77-84, 2012. https://doi.org/10.1145/2133806.2133826
- [5] S. Kavvadias, G. Drosatos, and E. Kaldoudi, “Supporting topic modeling and trends analysis in biomedical literature,” J. of Biomedical Informatics, Vol.110, Article No.103574, 2020. https://doi.org/10.1016/j.jbi.2020.103574
- [6] H. Kilicoglu, “Biomedical text mining for research rigor and integrity: tasks, challenges, directions,” Briefings in Bioinformatics, Vol.19, Issue 6, pp. 1400-1414, 2018. https://doi.org/10.1093/bib/bbx057
- [7] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” arXiv:1301.3781, 2013. https://doi.org/10.48550/arXiv.1301.3781
- [8] A. B. Dieng, F. J. R. Ruiz, and D. M. Blei, “Topic Modeling in Embedding Spaces,” Trans. of the Association for Computational Linguistics, Vol.8, pp. 439-453, 2020. https://doi.org/10.1162/tacl_a_00325
- [9] D. Angelov, “Top2Vec: Distributed Representations of Topics,” arXiv:2008.09470, 2020. https://doi.org/10.48550/arXiv.2008.09470
- [10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol.1 (Long and Short Papers), pp. 4171-4186, 2019. https://doi.org/10.18653/v1/N19-1423
- [11] I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A Pretrained Language Model for Scientific Text,” Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. on Natural Language Processing (EMNLP-IJCNLP), pp. 3615-3620, 2019. https://doi.org/10.18653/v1/D19-1371
- [12] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “BioBERT: A pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, Vol.36, Issue 4, pp. 1234-1240, 2020. https://doi.org/10.1093/bioinformatics/btz682
- [13] M. Grootendorst, “BERTopic: Neural topic modeling with a class-based TF-IDF procedure,” arXiv:2203.05794, 2022. https://doi.org/10.48550/arXiv.2203.05794
- [14] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing,” ACM Trans. on Computing for Healthcare, Vol.3, Issue 1, Article No.2, 2022. https://doi.org/10.1145/3458754
- [15] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” arXiv:1908.10084, 2019. https://doi.org/10.48550/arXiv.1908.10084
- [16] Z. Yu, E. Bernstam, T. Cohen, B. C. Wallace, and T. R. Johnson, “Improving the utility of MeSH® terms using the TopicalMeSH representation,” J. of Biomedical Informatics, Vol.61, pp. 77-86, 2016. https://doi.org/10.1016/j.jbi.2016.03.013
- [17] D. Mimno, H. Wallach, E. Talley, M. Leenders, and A. McCallum, “Optimizing Semantic Coherence in Topic Models,” Proc. of the 2011 Conf. on Empirical Methods in Natural Language Processing, pp. 262-272, 2011.
- [18] O. Lezhnina, “Depression, anxiety, and burnout in academia: Topic modeling of PubMed abstracts,” Frontiers in Research Metrics and Analytics, Vol.8, Article No.1271385, 2023. https://doi.org/10.3389/frma.2023.1271385
- [19] Y. Wang, R. Benavides, L. Diatchenko, A. V. Grant, and Y. Li, “A graph-embedded topic model enables characterization of diverse pain phenotypes among UK biobank individuals,” iScience, Vol.25, Issue 6, Article No.104390, 2022. https://doi.org/10.1016/j.isci.2022.104390
- [20] M. Röder, A. Both, and A. Hinneburg, “Exploring the Space of Topic Coherence Measures,” Proc. of the 8th ACM Int. Conf. on Web Search and Data Mining, pp. 399-408, 2015. https://doi.org/10.1145/2684822.2685324
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.