1. Introduction

Nowadays large language models (LLMs) are used to solve a lot of tasks related to natural language processing (NLP) in different areas. For example, as demonstrated in ¹, it is possible to use abilities of artificial intelligence (AI) to estimate quality of financial report. In this case, a report already exists. Another area is healthcare and medicine, where LLMs are used for treatment planning or patient engagement. There are a lot of ethical aspects of deploying LLMs in a clinical environment. For example, the most frequently discussed concerns are social bias, fairness, safety and privacy ². All these issues should also be considered during medical report generation in radiology. In this field, AI models offer a promising pathway to automate report generation ³. A radiology report consists of a findings section that contains informative, fact-based observations of a patient’s conditions and an impression section that contains a thoughtful synthesis of the meaning of the findings leading to a diagnosis, a differential diagnosis, and management recommendations ⁴. The process of impression creation is time-consuming and requires a lot of concentration from a radiologist. There are several LLM-based approaches aimed to reduce the workload. For example, Sun et al. ⁵ explored the GPT-4 capabilities and limitations in performing zero-shot generation of an impressions section from a radiology report findings section. Serapio et al. ⁶ have shown that open-sourced fine-tuned text-to-text transformer (T5) LLM is able to generate a radiology report impression out of key findings with a satisfactory level of clinical and grammatical accuracy. In ⁷, Zhang et al. have shown that given a large dataset of 20 GB of medical and general-purpose texts for pre-training and 1.5 GB for fine-tuning, it is possible to train LLM that would be able to generate professional-grade and linguistically correct texts for a full spectrum of radiology examinations. The case when there is no such amount of data was investigated by Voinea et al. in ⁸ where parameter-efficient fine-tuning technique was used to fine-tune Llama3-8b for automatic generation of magnetic resonance imaging (MRI) and computer tomography (CT) impressions from radiology reports. Emerging of multi-agent systems inspired Zeng et al. ⁹ to develop multi-agent LLM framework for generation radiology report’s impression from the finding section.

The other side of radiology reports generation is the creation of findings section itself. This part contains observations of abnormalities and different complex patterns found in MRI or CT scans. As highlighted by Voinea et al. ⁸, report creation plays the key role in documenting found pathologies. It is important to mention that during the MRI scans interpretation, different specialists use individual vocabulary for describing similar structures. This writing style should be preserved in automatically generated reports. So, generation of personalized findings is a complex and relevant task. One of the ways to solve it is to fine tune LLM on existing reports. There are two major problems with this approach: (1) a limited number of reports belong to the specialist and (2) emergence of new reports that were not presented in the training set. In order to mitigate these issues retrieval augmented generation (RAG) strategy can be used ¹⁰. It helps to keep information up to date and significantly improve generation quality of the model.

The area of automatic generation of a findings section hasn’t been extensively explored yet. The current study introduces the novel approach to generate a findings section with help of key phrases, the RAG technique, and the LLM agent. Because of the RAG strategy, the generated section has unique writing style of the author. The RAG technique utilizes the vector database where reports along with the author metadata (author ID) are stored. So, during the retrieval process, only reports that belong to the certain author are extracted. An impression section is generated by another LLM agent by means of the RAG approach. For organizing information exchange between two above-mentioned LLMs the graph data structure is used. The generation quality of findings section was estimated with BLEU, ROUGE, and BERTScore metrics. Experiments have shown that it is possible to generate well-structured and consistent personalized MRI reports without fine-tuning LLM on large amount of data.

2. Related Works

The use of RAG in radiology and multi-agent systems today is a fairly popular approach to generating reports’ impressions on medical images (MRI, X-rays, mammography, CT, etc.), as well as for forming substantiated answers based on the information received from various sources and building diagnostic conclusions, even in cases where there are no direct answers within the received context ¹¹. In ¹², Alam et al. introduce the multi-agent automated radiological report generation system based on the use of RAG-driven LLM for recognizing problem areas in chest X-ray images and final report creation. Close approach was proposed in ¹³, where authors present a multi-layered system based on LLM agents that is able to retrieve information from the BioBERT vector store as well as from the web. The developed framework is claimed to be a clinical intelligence partner designed to learn and actively contribute to the decision process. All the above-mentioned solutions are multi-purpose. They either need to have original radiological images, a large corpus of data, or access to the web. In the current study single-task approach based on limited number of existing medical reports is proposed.

The current study correlate with the research described in ⁸ were authors fine-tuned LLM to generate impression (conclusion) part. In this paper, inside the “Conclusion” section, authors mentioned that “LLMs can assist clinicians not only with conclusions but also with the entire report”. The current study proposes method to implement this. But instead of fine-tuning LLM on existing reports, RAG technique was used. Also, in the current study, a personalized approach to generating radiology reports was implemented. Fine-tuned LLM, on the contrary, is able to produce only generalized content.

Multi-agent approach and RAG technique in relation to medical report generation was also described in ⁹. This aligns with the current research, but that study on the one hand, was focused only on impression generation, and on the other hand, implementation of the proposed system assumed LLM with approximately a hundred billion of parameters, which needs environment with large GPU. In the current research full report was generated with LLM of only 8 billion parameters, which is suitable for environment with limited resource capabilities. Comparative synthesis is provided in Table 1. Table 1 shows that existing methods do not support personalization and are mainly focused on impression section generation based on findings section. Many of them use fine-tuning technique which requires dataset of significant size.

3. Materials

4. Methods

4.1. Vector Database

In order to effectively store and retrieve raw text a vector store is needed. This type of database uses vector space model to store data items along with associated numerical representations (vectors). Typically, vectors are word embeddings. Relevant documents can be retrieved out of a vector store with respect to distance metrics, for example, cosine similarity, which is expressed as follows:

\begin{equation} \cos \theta = \dfrac{\displaystyle \sum_{i=1}^{n} A_{i} B_{i}} {\displaystyle \sqrt{\displaystyle \sum_{i=1}^{n} A_{i}^{2}} \cdot \sqrt{\displaystyle \sum_{i=1}^{n} B_{i}^{2}}}, \label{eq:eq1} \end{equation}

where \(A_{i}\) and \(B_{i}\) are word embeddings.

Usually, a workflow with a vector database consists of two stages. In the first stage, raw texts are processed with an embedding function to obtain vector representation and after that are written into database. In the second stage, user query that was processed with the same embedding function is forwarded into database in order to retrieve top \(N\) semantically similar texts. It is important to note that the embedding function should be chosen with respect to the language the texts are written in.

4.3. LangChain Framework

LangChain is a platform for building multi-agent LLM applications. It includes two frameworks: LangChain that helps to quickly prototype an LLM-based application and LangGraph that provides functionality such as low-level orchestration, memory management, and human-in-the-loop support. In the current study LangGraph framework is used.

Application logic was broken into three nodes: information retrieval node, findings generation node, and impression generation node. The first node uses Chroma DB vector store to retrieve information. The second node utilizes Qwen3-8B LLM ¹⁷ to generate findings section based on key phrases and prompt that was enriched with the information provided by the first node. Qwen3 offers strong capabilities for multilingual instruction following, which is crucial for this task. The third node uses the findings part, relevant data from the vector store and Llama 3.1 8b ¹⁸ as LLM to summarize the impression section. Llama 3.1 8b LLM is used because it is highly effective for text summarization due to long-context understanding. At the same time, according to conducted experiments, it hallucinates during sequential merging operations of two texts written in Russian. So, it is not recommended to use it for the second node in similar settings.

4.4. Evaluation

In the current study, ROUGE, BLEU, and BERTScore metrics are used. These metrics are usually used in NLP-related tasks for estimation performance of the model.

ROUGE (recall-oriented understudy for gisting evaluation) is a set of metrics that is used to evaluate automatic summarization and machine translation ¹⁹. The main goal is to check how much of the important content from referenced text is preserved in the generated text. In order to do it, word overlap is calculated. ROUGE-1 is used to capture overlap between individual words, ROUGE-2—between word pairs, ROUGE-\(n\)—between \(n\)-words (\(n\)-grams). ROUGE-\(n\) is computed as follows:

\begin{equation} \mbox{\textit{ROUGE-}$n$} = \dfrac{\displaystyle \sum_{r_{i} \in \mathit{ref}} \sum_{n_{i} \in r_{i}} \mathit{Count}\bigl(n_{i}, C\bigr)} {\displaystyle \sum_{r_{i} \in \mathit{ref}} \mathit{numNgrams}\bigl(r_{i}\bigr)}, \label{eq:eq2} \end{equation}

where \(r_{i}\) are sentences in the reference document, \(n_{i}\) is an \(n\)-gram belongs to the document, \(\mathit{Count}(n_{i}, C)\) is the function that indicates the number of times the \(n\)-gram occurs in the candidate document (\(C\)), and \(\mathit{numNgrams}(r_{i})\) is the total number of \(n\)-grams in the reference document. ROUGE-L operates on the longest sequence of words that appear in the same order in both texts (longest common subsequence, LCS). This metric is calculated as follows:

\begin{equation} \mbox{\textit{ROUGE-L}} = \dfrac{\bigl(1 + \beta^{2}\bigr) \cdot P \cdot R}{\beta^{2}\cdot P + R}, \label{eq:eq3} \end{equation}

where \(P\) is the precision (the ratio of the number of matching \(n\)-grams to the total number of \(n\)-grams in the generated text), \(R\) is the recall (the ratio of the number of matching \(n\)-grams to the total number of \(n\)-grams in the referenced text), and \(\beta\) is the parameter that controls relative importance of precision and recall. ROUGE-Lsum implements ROUGE-L calculation method at the sentence level and provides aggregating information afterwards. In other words, ROUGE-Lsum splits the referenced and generated texts into sentences, performs ROUGE-L calculation for each sentence and finds the average of scores in the end.

BLEU (Bilingual Evaluation Understudy) score is a measure of precision of matching \(n\)-grams between text that was generated and referenced text ²⁰. The formula is given by Eq. 4.

\begin{equation} \textit{BLEU} = \mathit{BP} \cdot \exp \left(\displaystyle \sum_{n=1}^{N} \omega_{n} \log p_{n}\right) \label{eq:eq4} \end{equation}

Here, \(\mathit{BP}\) (brevity penalty) is a sizing coefficient, \(\omega_{n}\) are the weights for ngram precision, \(p_{n}\) is the ngram precision. BLEU score was designed to be a corpus measure, so for its application on the sentence level modifications are needed. In the current study Google BLEU variation of the BLEU metric is used. In this case final score is the minimum value of precision and recall that were calculated based on candidate and reference texts ²¹. Google BLEU score can be used in replace of usual BLEU metric.

BERTScore is a metric that is used to find semantic similarity between two texts ²². For this purpose, it utilizes contextual embeddings from a pre-trained BERT model to match words in generated and referenced texts using cosine similarity. The resulting score include precision (shows how generated tokens are covered by reference tokens), recall (shows how referenced tokens are covered by generated tokens) and F-score (harmonic mean of precision and recall).

4.6. Method to Generate Impression Section

An impression section was generated with help of another LLM agent (Llama 3.1 8b). Approach is similar to the mentioned before. It relies on the same vector database and RAG strategy. In the first stage, example impressions are obtained from the vector database. In the second stage, they are combined with the previously generated findings section, forming the context for the LLM agent. This context is used to enrich the prompt. After that, the LLM agent generates impression part. The prompt for LLM is shown in Table 4. Here, variable “examples” is a string that contains sample impressions sections obtained from vector store, variable “text_for_summary” is findings section that was generated by the LLM agent.

4.8. Ethical Aspects

According to ²³, over the past decade, AI has become one of the fundamental technologies used to create innovative solutions in medicine and healthcare. LLM, which allows for the analysis, interpretation, and generation of text information in human language, occupies a special position in the development of medical information systems. The implementation of LLM is accompanied by a number of ethical issues and problems, of which the following, grouped into separate categories, can be attributed to the field of automatic generation of medical reports ²⁴:

• The quality and reliability of the generated texts, since the health and lives of patients, as well as the professional reputation of medical workers and researchers, depend on this ²⁵. In particular, RAG-driven models can incorrectly aggregate conflicting information from different sources. LLMs often learn from their own mistakes. LLMs operate on response probabilities, which can lead to the potential for ignoring rare clinical cases. Most of the above issues are relevant to the proposed methods. Therefore, only those generated reports that have been verified and corrected by a medical professional are saved in the system. From a legal perspective, LLMs currently do not have the status of independent legal entities. Legal liability for the consequences of reports generated using LLMs rests with the medical professionals.

• The presence of “hallucinations” in LLM-based systems, as they are trained on text corpora that may contain historical, social, and cultural biases, an unbalanced representation of medical knowledge, and errors in the labeling of medical data ^26,27. The proposed methods do not include an LLM training module.

• Opacity of the mechanisms for generating conclusions, lack of validity. The proposed methods do have an opaque nature. However, if necessary, when implementing a system based on these methods, it is possible to reference the templates from which the texts were taken when generating reports. This can be used as additional information for the healthcare professional.

• Confidentiality and security of personal data ²⁸. Even if the data is anonymized, there is a risk of reconstructing the patient’s identity based on indirect signs, which is especially relevant for rare diseases or unique combinations of clinical signs ²⁹. The proposed methods are inherent to this problem, but given that they use only the templates of a specific healthcare professional to generate reports for their patients, and only locally, the risk of information leakage is no greater than when the healthcare professional works with paper MRI images or folders containing the patient’s MRI images stored on the physician’s computer. Otherwise, the use of intelligent systems based on the proposed methods requires patient consent to the processing of personal data.

5. Results

During the experiments, findings and impression parts were generated with the same key phrases for each of the seven radiologists for both variations of the method. The quantitative results of generation findings section include estimation by such metrics as BLEU (Google BLEU), ROUGE, and BERTScore. Evaluation results for the first variation of the method are provided in Tables 5 and 6. For the second variation are shown in Tables 7 and 8.

As can be seen from the tables, both variations provide results closed to mentioned in Table 1. ROUGE scores for every specialist indicate that short as well as long word sequencies are captured by the LLM correctly, therefore generated text is coherent and fluent. BLEU score shows that generation results are similar to the reference texts but at the same time different word order and possible synonyms of the words affect the resulting value. It should be noticed that ROUGE and BLEU scores do not take into account semantics of the compared texts. In order to mitigate this issue BERTScore was used. BERT score that was estimated for every specialist indicates that a generated text is semantically similar to a referenced text despite different word order.

Another metric that should be taken into account is generation time. In the current study the generation time of the full report (a findings section and an impression section) was estimated for both method’s variations. Results are depicted in Figs. 6 and 7, respectively.

Comparison shows that time needed to generate full report for the first variation of the method is much higher than that for the second one. This is due to the fact that merging operation of the two findings sections is time-consuming. Also, the first variation is quite sensitive to the number of key phrases given to the input.

During the clinical evaluation 50 radiologist-generated MRI findings sections were used along with 50 MRI findings sections generated by LLM with proposed method. Each findings section was assessed by two professional radiologists across 5 dimensions using a five-point Likert scale. Questionary for radiologists includes clinical correctness (assessment of anatomical accuracy, diagnostic terminology, and clinical validity), coherence (assessment of logical flow, internal consistency, and structural organization), completeness (assessment of required elements, missing information, and comprehensiveness), style fidelity (assessment of professional conventions, terminology, and formatting standards), and risk of harm (assessment of patient safety implications, diagnostic errors, and miscommunication risks). For disparities determination Mann–Whitney U test was used. Results of the test are provided in Table 9 and in Fig. 8. Statistical analysis was conducted in Python 3.12.1 with Numpy (version 2.4.2), SciPy (version 1.17.1), and Pandas (version 3.0.1) packages.

In Table 9, “Radiologist” is the assessment of radiologist-generated MRI findings performed by radiologists. “LLM” is the assessment of LLM-generated MRI findings performed by radiologists.

The Mann–Whitney U test results demonstrate that while LLM show impressive capabilities in certain areas, human radiologists maintain statistically significant advantages in clinical correctness, completeness, and overall evaluation performance. The statistically insignificant result in risk of harm assessment is particularly encouraging and suggests LLM could serve as valuable safety validation tools in clinical workflows.

Findings sections generated by the LLM with the second and first variations of the method are provided in Figs. 9 and 10, respectively. Key phrases are highlighted.

6. Conclusion

In this study a novel key phrases-driven method for MRI reports’ findings section generation was presented. This method was implemented in two different ways each of which has its own benefits and drawbacks. The generation quality was estimated with multiple metrics quantitatively as well as with the five-point Likert scale questionary qualitatively and showed optimistic results. Graph-based architecture helped to integrate multiple LLM agents that were able to use different language models for distinct tasks. Also, the RAG strategy made it possible to generate high-quality medical texts by small pre-trained LLMs using the thesaurus of a specific author.

The current research has limitation connected with the number of key phrases used to generate findings section. Experiments have shown, that optimal number of key phrases is not exceeded seven. It was noticed, that eight billion parameters LLM is unable to repeat merge operation of texts eight or more times. In some experiments it lost context and stood frozen in the loop of generating the same token; in others the final text produced was unexpected. It is possible, that LLM with more parameters would be able to handle larger number of key phrases. But it is worth to say that there is a trade-off between generation time and the number of key phrases.

Another limitation can be connected with language sensitivity. Different languages may require different prompts for LLMs. Also, hyperparameters for retrieval may be chosen differently as well as embedding model.

Further experiments can be connected with a choice of different LLM. For example, one can use model with greater number of parameters or model of different type. It is interesting to find out, how proposed method will act when number of reports increase significantly. Also, in the current study basic (vanilla) RAG technique was used, which can be optimized in the following aspects: pre-retrieval (structurization and preprocessing of input data for better indexing), retrieval (improving the embedding model and filtering strategy), post-retrieval (filtering noise from the texts that were retrieved). Another direction of future research includes developing of leakage prevention protocol (leakage between test targets and retrieval corpus) and verification statement.

Despite all challenges, proposed method gives promising results in generation full medical report. It is not limited to MRI reports, but can also be extended to CT reports. The ultimate goal is to reduce workload pressures on healthcare systems in context of automatic medical report generation.