2.1. Music Generation AI

Neural networks have revolutionized music generation, enabling models to handle increasingly complex compositional tasks. Early neural network-based music generation models like MiniBach ⁵ demonstrated basic capabilities but faced significant limitations. MiniBach uses a three-layer network structure to generate bass, tenor, and alto parts when given a soprano melody. However, this model struggles with time-series data processing, resulting in output melodies that lack complexity and variation.

To address these temporal modeling limitations, DeepBach was developed as an improved version of MiniBach ⁶. By incorporating long short-term memory (LSTM) networks, which excel at capturing long-term dependencies in sequential data ⁷, DeepBach can generate more varied and musically coherent melodies that better capture the temporal structure of music.

Yang et al. have developed MidiNet, a music generation model that treats music as image data ⁸. MidiNet converts music into image data and uses convolutional neural network (CNN) and generative adversarial networks (GANs) for learning ⁹. This image-based representation allows the model to consider global song structure rather than just local melodic relationships.

Current state-of-the-art music generation models leverage Transformer architectures ¹⁰, which can effectively model long-range dependencies and capture important musical relationships. Notable examples include Google’s MusicLM ¹¹ and Meta’s MusicGen ¹². MusicGen, which we utilize in our system, consists of three components: Encodec (an audio compression model ¹³), Text-to-Text Transfer Transformer (T5) (a pre-trained natural language processing model ¹⁴), and an autoregressive language model for audio token generation. MusicGen’s key advantages include high-speed parallel processing of acoustic tokens and the ability to condition generation on both text prompts and melodic input ¹⁵. Despite these advances, current music generation models require users to provide appropriate textual prompts to generate desired music. This presents a significant barrier for users who struggle to articulate their musical preferences in words, motivating our approach to optimize prompts automatically based on user feedback.

2.2. Interactive Evolutionary Computation

Traditional optimization methods work well for problems with clear numerical objectives, but many real-world applications involve subjective preferences that cannot be easily quantified ¹⁶. IEC addresses this challenge by incorporating human evaluation directly into the optimization process, making it particularly suitable for creative and aesthetic applications.

Typical EC techniques include genetic algorithms (GA), tabu search (TS), genetic programming (GP), evolution strategy (ES), evolutionary programming (EP), and particle swarm optimization (PSO).

Several researchers have successfully applied IEC to music generation and optimization. Moroni et al. developed a system that evolves MIDI-encoded music using genetic algorithms, where users can manipulate fitness criteria related to physical musical properties through graphical controls ¹⁷. Masuda and Iba proposed a system using recurrent neural networks with variational autoencoders (RNN-VAE) to model musical latent spaces, allowing users to explore musical styles through IEC-guided navigation ¹⁸. Their user studies demonstrated the effectiveness of this approach compared to traditional composition methods.

These previous works demonstrate IEC’s potential for music generation but focus primarily on evolving musical content directly. Our approach differs by using IEC to optimize the textual prompts that drive modern generative AI models, bridging the gap between human preferences and AI-generated content.

2.3. Natural Language Processing for Music Generation

Natural language processing refers to technology that allows computers to process natural language, which is used daily by humans. Natural language processing technology can handle a variety of tasks related to natural language, such as answering questions, summarizing and translating texts, and programming software.

The integration of natural language processing with music generation has become increasingly important as text-to-music models gain prominence. Word2Vec, introduced by Mikolov et al. ¹⁹, laid the foundation for modern vector-based representations of language.

Word2Vec’s simple structure allows for realistic calculations using large amounts of data, and has dramatically improved the capabilities of natural language processing. Models such as RNN, LSTM, and Transformer have been developed based on Word2Vec ^10,20.

Word2Vec has been shown to be capable of performing word calculations such as \(\textit{Paris} - \textit{France} + \textit{Italy} = \textit{Rome}\) as shown in Fig. 1, and calculating the similarity between words using cosine similarity, and is used to solve high-level language processing tasks ²¹.

The T5 is a state-of-the-art natural language processing model developed by Google that treats all NLP tasks as text-to-text problems. Built on the Transformer architecture, which uses attention mechanisms to achieve superior processing speed and accuracy compared to previous models ²², T5 represents the culmination of advances that also produced models like BERT and GPT ^23,24. Through systematic comparison of different model structures, objective functions, datasets, and scaling approaches, the T5 researchers developed a highly versatile model that excels across diverse language tasks. T5 achieved a score of 90.3 on the GLUE benchmark, surpassing average human performance of 87.1 ²⁵.

In our system, we leverage T5’s 768-dimensional word embeddings to create semantic representations of musical concepts. This allows us to define music vectors as composite representations of multiple words and perform mathematical operations in semantic space to guide music generation toward user preferences.

3.1. System Overview

Our system aims to generate music that matches user preferences without requiring users to formulate explicit prompts. The key idea is to have users provide a few general keywords about their desired music, while the system automatically generates additional descriptive keywords to create more specific prompts for the music generation AI. Fig. 2 shows an overview of the system. The process begins when users input initial keywords describing their musical preferences. Since these keywords alone are insufficient for generating targeted music, the system augments them with automatically selected additional keywords. This enhanced prompt is then fed to the music generation AI to produce candidate pieces. The system iteratively refines these prompts based on user feedback. Users evaluate the generated music, and the system modifies the keyword selection to optimize future generations toward the user’s taste. When satisfactory music is produced, the system presents both the final piece and the optimized keywords to the user, effectively verbalizing their previously implicit preferences. Furthermore, the system modifies the prompt keywords based on feedback from the user to optimize the prompt so that music that suits the user’s tastes can be generated. When a song that suits the user’s taste is finally generated, the prompt keywords are presented to the user. This makes it possible to verbalize the user’s latent tastes. Our approach uses T5’s natural language processing capabilities to represent keywords as vectors in semantic space, enabling mathematical operations on musical concepts to guide the optimization process.

3.2. Music Generation

In this system, the music vector is defined as a composite vector composed of multiple words in the input prompt. The music vector \(V_\text{music}\) is a composite vector calculated from \(N_\text{word}\) individual word vectors \(V_{\text{word},i}\) (\(i=1,2,\ldots,N_{\text{word}}\)) using Eq. \(\eqref{eq:1}\). Each word vector \(V_\text{word}\) refers to the 768-dimensional vector obtained by inputting the word into the T5 natural language model.

To measure similarity between musical pieces, we employ cosine similarity, which calculates the angle between two vectors regardless of their magnitude. The cosine similarity \(S_\text{a\_b}\) between music A and music B is calculated using Eq. \(\eqref{eq:2}\) using music A’s vector \(V_\text{music\_a}\) and music B’s vector \(V_\text{music\_b}\).

On the other hand, the vocabulary size used in the natural language processing model T5 is 32,000, and this system generates music from multiple words.

Therefore, the number of types of music that can be generated, \(N_\text{kind}\), is calculated using the number of words, \(N_\text{words}\), used to generate one piece of music, using Eq. \(\eqref{eq:3}\).

At this time, to search for music that is similar to a specific piece of music, the similarity between the specific piece of music and all music that can be generated must be calculated. The larger the \(N_\text{words}\), the more time this similarity calculation requires.

To address this, our system extracts a word list from the input music data and replaces some words with similar alternatives. Music data created in this way is defined as similar music data. Similar music can be generated by using this similar music data. By using this generation method, there is no need to perform calculations for all music that can be generated, making it possible to generate similar music in a short amount of time.

3.3. User Preference Optimization

We employ a hill-climbing approach combined with IEC-driven user feedback to navigate the music vector space toward user preferences. Fig. 3 visualizes this search process in the high-dimensional semantic space.

First, given a base music piece \(M_\text{base}\), \(N_{\text{similar}}\) similar base music pieces \(M_{\text{similar},j}\) (\(j=1,2,\ldots,N_{\text{similar}}\)) are generated. Next, the difference vector \(V_{\text{difference},j}\) (\(j=1,2,\ldots,N_{\text{similar}}\)) between each similar music vector \(V_{\text{similar},j}\) (\(j=1,2,\ldots,N_{\text{similar}}\)) and the base music vector \(V_\text{base}\) is calculated using Eq. \(\eqref{eq:3.4}\).

After users rate these similar pieces with scores \(E_{j}\) (\(j=1,2,\ldots,N_{\text{similar}}\)), we compute a weighted total difference vector \(V_\text{difference}\), total that aggregates all difference vectors according to user preferences using Eq. \(\eqref{eq:3.5}\).

Finally, the new base vector \(V_\text{base\_new}\) is calculated using \(V_\text{base}\) and \(V_\text{difference\_total}\) as in Eq. \(\eqref{eq:3.6}\).

This \(V_\text{base\_new}\) serves as a new starting point for the next generation.

3.4. Optimizing Music Generation for User Preferences

The system employs a structured approach to adapt music generation to individual user preferences through iterative keyword optimization. This process consists of five main phases: initialization, keyword classification, music generation, user evaluation, and optimization steps that continue until convergence.

Initialization: The system begins by establishing an initial set of keywords to create the first generation of music. Participants input four English words representing their desired music in order of importance and select a category for each word (e.g., genre, instrument, emotion). To ensure musical coherence and prevent conflicting descriptions, the system enforces several constraints:

1. Same-category restriction: Participants cannot select multiple words from the same category (e.g., cannot choose both “piano” and “guitar” from “Instruments”).

2. Minimum diversity requirement: Participants must select from at least three different categories among the eight available categories.

3. Prohibited combinations: Examples include selecting all four seasons (spring, summer, autumn, winter) or multiple conflicting emotions (joy, sadness, anger).

The system then randomly selects additional words from unused categories to ensure diversity, forming a combined set of user-provided and system-selected words as the initial keyword set.

Keyword classification: The system classifies keywords into two types: fixed keywords and variable keywords. Fixed keywords are user-designated important words that remain constant throughout optimization, while variable keywords can be modified during the search process. Initially, the first two user-provided words become fixed keywords, while the remaining user words and all system-selected words become variable keywords.

Music generation process: Each music piece is generated using a set of eight words (four user-provided and four system-selected). For each variable keyword, the system creates a “similar word group” through the following process:

1. T5 embedding query: The system queries T5’s 768-dimensional word embeddings to find semantically similar words within the same category.

2. Similarity calculation: Cosine similarity is computed between the target word’s embedding and all words in the same category.

3. Similar word selection: The five most similar words are selected to form alternatives.

4. Positive/Negative list filtering:

   • Words from highly-rated pieces are added to a positive list and given priority in selection.

   • Words from poorly-rated pieces are added to a negative list and excluded from candidate generation.

   • If a similar word appears in the negative list, it is replaced with the next most similar alternative.

5. Group formation: Each similar word group contains the original word plus five semantically similar alternatives (totaling six words).

To generate candidate music pieces, the system selects one word from each similar word group and combines these selections with all fixed keywords. For example, with three fixed words and five variable words, where each variable word has six alternatives, this creates \(6^5 =\) 7,776 possible combinations. From these candidates, ten pieces are randomly selected and presented to the user.

User evaluation and learning: After participants evaluate the generated music on a 10-point scale, the system updates its keyword preferences:

• Words from pieces rated \(\geq8\) are added to the positive list.

• Words from pieces rated \(\leq2\) are added to the negative list.

• The positive and negative lists influence future word selection as described above.

Optimization step: Using the user ratings, the system calculates an updated target vector as described in Section 3.3 and selects the keyword combination from the candidate set that produces a vector closest to this target using cosine similarity. This combination becomes the new base for the next generation.

Convergence: As optimization progresses, variable keywords that appear in the positive list multiple times (\(\geq3\) occurrences) are promoted to fixed status, gradually reducing the search space. The system terminates when nearly all keywords have become fixed, indicating that the search has converged to the user’s preferences and further optimization is unlikely to yield significant improvements.

4.1. Experiment Overview

We conducted experiments to evaluate whether our proposed system effectively generates music optimized to user preferences. The experiment involved 15 university students who interacted with the system to generate personalized music.

Table 1 shows the word categories and options used for system-selected keywords, which were chosen to cover common descriptive dimensions in music generation.

The experimental procedure consisted of the following steps:

1. Keyword input: Participants entered four English words representing their desired music in order of importance and selected the appropriate category for each word from a dropdown menu (Fig. 4). To ensure accurate spelling, participants were encouraged to use online resources as needed. After that, the participant selects which word category each entered word belongs to from a pull-down menu.

   

2. Music generation and evaluation: The system generated only ten different 10-second pieces of music in each generation and presented them to participants (Fig. 5). The choice of ten pieces per generation was based on several design considerations:

   • Cognitive capacity: Research in cognitive psychology suggests that humans can effectively evaluate \(7\pm2\) items simultaneously without significant performance degradation.

   • Computational efficiency: Generating ten pieces allows response times under 30 seconds while providing sufficient diversity for optimization.

   • Search space coverage: Our preliminary testing showed that ten pieces provide adequate coverage of the local search space around the current optimization point.

   • User experience: Pilot studies indicated that participants can complete evaluation of ten pieces within 3–5 minutes, preventing evaluation fatigue while maintaining engagement.

   

   The 10-second duration was selected to balance evaluation efficiency with sufficient musical content for preference assessment, based on research showing that musical preference judgments can be formed within the first few seconds of listening. The system generates exactly ten pieces per generation through random sampling from the candidate set. This approach ensures:

   • Unbiased representation: Random selection prevents algorithmic bias toward specific keyword combinations.

   • Computational tractability: Avoids the exponential complexity of evaluating all \(6^5 =\) 7,776 possible combinations.

   • Exploration vs. exploitation balance: Maintains diversity while focusing on promising regions of the search space identified by previous user feedback.

   Each piece was generated by MusicGen’s small model based on different keyword combinations. Participants evaluated each piece on a 10-point relative scale using dropdown menus. We emphasized relative rather than absolute ratings to maintain discrimination power throughout the optimization process, as absolute ratings tend to drift upward in later generations.

3. Optimization process: After each evaluation round, the system updated its keyword selections based on user feedback and generated a new set of music pieces. This process continued until convergence.

4. Final results: Upon completion, the system presented the optimized music and the keywords used to generate it (Fig. 6). Participants were encouraged to look up unfamiliar English keywords to understand the system’s verbalization of their preferences.

After the experiment, participants completed a questionnaire consisting of seven items (Table 2) including five 5-point Likert scale questions, one 6-point frequency question, and an open-ended comment section.

This experiment was approved by the Research Ethics Review Committee of Organization for Research and Development of Innovative Science and Technology (ORDIST) in Kansai University.

4.2. Experimental Results

The questionnaire results are presented in Figs. 7 and 8, with detailed analysis as follows. Fig. 7 shows responses to questions Q1–Q6 using 5-point Likert scales, while Fig. 8 displays the frequency distribution for Q7 regarding unfamiliar keywords.

Q1 (Music alignment – Fig. 7, top bar): 93.3% of participants (strongly agree: 26.6%, slightly agree: 66.7%) found that the generated music became more aligned with their preferences through the evaluation process. This indicates the effectiveness of our prompt optimization approach in capturing user preferences.

Q2 (Evaluation fatigue – Fig. 7, second bar): Mixed responses regarding evaluation process fatigue, with 26.7% of participants finding it tiring while 46.7% did not. This suggests that our implementation strikes a reasonable balance between thoroughness and user effort, though further improvements to reduce evaluation fatigue could be beneficial.

Q3 (Direction change – Fig. 7, third bar): Interestingly, 53.4% of participants reported that their desired musical direction changed during system interaction, suggesting that the process helped them explore and discover new preferences.

Q4 (Interest increase – Fig. 7, fourth bar): All participants (100%) reported increased interest in music generation after using the system.

Q5 (Keyword representation – Fig. 7, fifth bar): Regarding keyword representation, 86.6% of participants felt that the final keywords accurately represented the generated music, confirming the system’s ability to verbalize musical characteristics.

Q6 (System preference – Fig. 7, sixth bar): Most participants (100%) also expressed preference for systems that suggest input words, highlighting the value of our approach.

Q7 (Keyword familiarity – Fig. 8): 80.0% of participants encountered at least one unfamiliar system-generated keyword, indicating the system’s tendency to introduce specialized musical terminology.

Figures 9 and 10 track individual user progress across generations. Fig. 9 shows the average rating values within each generation, while Fig. 10 displays the standard deviation of ratings. Both metrics remain relatively stable across generations for most users, suggesting consistent relative evaluation throughout the optimization process.

4.3. Discussion

This section analyzes the experimental findings across four key dimensions: optimization effectiveness, user experience factors, system usability, and methodological insights. Each analysis connects questionnaire results to broader implications for interactive music generation systems.

Optimization effectiveness and technical validation: The results from Q1 demonstrate that our prompt optimization approach successfully adapts to user preferences, with 93.3% of participants reporting improved music alignment. This validates our core hypothesis that hill-climbing methods combined with IEC can effectively navigate high-dimensional semantic spaces. The mathematical framework using T5 word embeddings enables semantic operations that meaningfully connect user feedback to prompt modifications, confirming that composite vector representations can capture complex musical preference relationships. The consistent evaluation patterns shown in Figs. 9 and 10 further validate our approach. The stability of both average ratings and standard deviations across generations indicates that participants maintained reliable discrimination throughout optimization, enabling continuous refinement even in later stages. This consistency is crucial for hill-climbing algorithms, as it ensures that user feedback provides reliable gradient information for the search process.

User experience and fatigue analysis: User fatigue emerges as a critical design consideration, with statistical analysis revealing a strong correlation between reported fatigue and generation count (\(r = 0.771\), \(p < 0.01\), \(N = 15\)). This finding has immediate implications for system deployment:

Design implications:

• Adaptive termination: Systems should monitor engagement indicators and suggest breaks when fatigue patterns emerge.

• Session management: Save/resume functionality would allow optimization across multiple sessions.

• Evaluation efficiency: Active learning approaches could reduce required evaluations per generation.

• Alternative interfaces: Pairwise comparison or implicit feedback methods might reduce cognitive load.

Despite this correlation, 46.7% of participants reported no fatigue, suggesting that our 10-second, 10-piece format strikes a reasonable balance for many users. The short music duration likely contributed to lower fatigue levels, but this presents trade-offs between evaluation efficiency and musical content richness.

Preference evolution and discovery: A particularly interesting finding is that 53.4% of participants reported changes in their desired musical direction during interaction (Q3). This suggests our system functions beyond simple preference matching—it actively supports musical exploration and discovery. However, this preference evolution creates optimization challenges, as shown in Fig. 11. In Fig. 11, the large circles highlight two participants who exhibited extreme cases in the relationship between preference change and optimization duration. One participant (Q3 Answer: 2, Final Generation: 13) showed strong preference evolution requiring extended optimization, while another demonstrated rapid convergence despite preference changes. These cases illustrate the variability in individual adaptation patterns and suggest that some users require more exploration time to discover their true preferences, while others can quickly adapt to new musical possibilities. This variability suggests that adaptive algorithms could improve efficiency by detecting preference stability and adjusting search strategies accordingly—using broader exploration for evolving preferences and focused refinement for stable ones.

System usability and practical adoption: The system demonstrates strong usability indicators: 100% of participants expressed preference for word-suggestion systems (Q6) and reported increased interest in music generation (Q4). This universal positive response suggests significant demand for user-supportive generative AI interfaces. However, keyword comprehension presents a challenge, with 66.7% of participants encountering unfamiliar system-generated terms (Q7). While specialized musical vocabulary can enhance generation quality, it creates barriers to user understanding. The system’s ability to verbalize preferences (86.6% accuracy in Q5) is valuable, but the vocabulary gap suggests need for:

• Keyword explanation features providing definitions for specialized terms.

• Simplified vocabulary options for novice users.

• Progressive complexity that introduces advanced terms gradually.

Future research directions: Our findings highlight several important research directions:

• Music length optimization: The 10-second duration likely contributed to low fatigue levels but may limit preference expression richness. Future work should investigate adaptive duration strategies—shorter clips for initial exploration, longer samples for fine-tuning—and examine how music length affects both fatigue and preference accuracy.

• Algorithm enhancement: While hill-climbing proved effective, more sophisticated approaches could address preference evolution more efficiently. Algorithms that detect preference stability and dynamically balance exploration versus exploitation could reduce generation requirements while maintaining quality.

• Evaluation method innovation: Alternative evaluation interfaces beyond 10-point scales might reduce cognitive load while maintaining discrimination power. Approaches such as pairwise comparison, ranking-based selection, or implicit feedback through listening behavior could improve user experience.

• Scalability considerations: Extending to longer compositions while maintaining structural coherence represents a significant technical challenge requiring investigation of hierarchical optimization approaches and multi-level preference modeling.

• Methodological limitations and validation: Our relative evaluation approach, while maintaining discrimination consistency, creates challenges for optimization verification. Since average ratings remain stable across generations (as designed), confirming improvement requires questionnaire validation rather than direct measurement. Future work should incorporate complementary evaluation methods—such as comparing initial versus final generation music through independent listeners—to provide more robust optimization validation. The 15-participant sample, while sufficient for initial validation, limits generalizability. Larger-scale studies across diverse musical backgrounds and cultural contexts would strengthen findings and reveal potential population-specific optimization patterns.