Top > JACIII > Optimizing Speech Recognition and Feedback in Spoken English L ...

single-jc.php

JACIII Vol.30 No.4 pp. 966-976
(2026)
doi: 10.20965/jaciii.2026.p0966

Research Paper:

Views over last 60 days: 400

Optimizing Speech Recognition and Feedback in Spoken English Learning Using Long Short-Term Memory Networks

Yu Shang† and Shang Xue

Zhengzhou Railway Vocational and Technical College
No.9 Qiancheng Road, Zhengdong NewDistrict, Zhengzhou, Henan 451460, China

†Corresponding author

Received:
July 23, 2025
Accepted:
January 25, 2026
Published:
July 20, 2026
Creative Commons License
Keywords:
oral English learning, long short-term memory network, speech recognition, feedback mechanism, machine learning
Abstract

In the era of globalization, learning English through oral means is becoming increasingly important. However, inaccurate speech recognition and imperfect feedback mechanisms have significantly hindered the improvement of the oral ability of learners. To solve this problem, this study proposes an English oral learning speech recognition and feedback system based on a multilayer improved long short-term memory network (MLSTM). The study uses the Texas Instruments and Massachusetts Institute of Technology (TIMIT) speech database and employs a hidden Markov model (HMM) and standard LSTM systems as controls to conduct a comprehensive test of the proposed model. The experimental results showed that the MLSTM model achieved high speech recognition accuracy, with an overall accuracy of 86.3% that was significantly higher than the 65.2% of HMM and 80.1% of LSTM. In terms of feedback information targeting and effectiveness, the MLSTM model scored 4.35 and 4.5, respectively, that were suggestively better than those of the comparison models. This showed that the MLSTM model could accurately recognize speech and provide learners with highly personalized and effective feedback. The research results enrich the theory of computer-assisted language learning, provide practical and effective tools for oral English learning, and promote oral English learning toward greater intelligence and efficiency.

Close full text
Cite this article as:
Y. Shang and S. Xue, “Optimizing Speech Recognition and Feedback in Spoken English Learning Using Long Short-Term Memory Networks,” J. Adv. Comput. Intell. Intell. Inform., Vol.30 No.4, pp. 966-976, 2026.
Data files:

1. Introduction

In the current era of rapid globalization, learning to speak English has become a significant challenge for many language learners. According to incomplete statistics, more than 70% of language learners worldwide believe that inaccurate speech recognition and the lack of an effective feedback mechanism are key factors hindering their ability to speak English. For example, a large online language learning platform has more than 5 million registered users, nearly 60% of whom reported that when practicing speaking, the speech recognition function of the system often misidentified their pronunciation, resulting in a significant reduction in the effectiveness of the practice 1. Moreover, up to 75% of the feedback received was general, with vague statements such as “pronunciation needs to be improved” and “intonation is not right,” that do not allow learners to know the exact nature of their problem and make targeted improvements. This situation highlights the severe lack of accurate speech recognition and valuable feedback mechanisms in the field of spoken English learning, and the importance and urgency of these issues are self-evident 2.

With the continuous advancement of science and technology, the application of computer technology to language learning has become increasingly extensive. However, currently, within the specific scope of oral English learning, existing technology fails to meet the actual needs of learners. For example, in certain traditional speech recognition software, the average recognition accuracy is approximately 60%, meaning that four of every 10 words may be misrecognized. This low accuracy rate dampens the enthusiasm of learners and leads them to repeat incorrect pronunciations owing to misperceptions, forming a vicious cycle 3. Moreover, existing feedback mechanisms are often based on simple rules and templates, lacking in-depth analysis of speech characteristics and specific errors of learners. They cannot provide personalized and precise improvement suggestions, significantly limiting the effectiveness and quality of oral English learning 4.

In the current research in related fields, many scholars and research teams are dedicated to addressing the challenges of speech recognition and feedback in oral English learning. In speech recognition, certain deep learning (DL)-based models such as convolutional neural networks (CNNs) have achieved notable results 5. Studies have shown that using an optimized CNN can improve the speech recognition accuracy to approximately 75% on a specific speech dataset 6. However, the CNN model has the disadvantage of an insufficient ability to process sequence information. Performance improvement is limited when faced with speech data that exhibit temporal characteristics, such as spoken English 7.

In the study of feedback mechanisms, certain studies have attempted to generate more targeted feedback by establishing a huge database of speech errors and combining it with machine learning (ML) algorithms. However, most of these studies remain theoretical. In practical applications, they face numerous problems, including the heavy workload of data annotation, complex model training, and the risk of overfitting, hindering their implementation and achieving practical effects. Additionally, relatively few studies exist on the effective integration of speech recognition and feedback mechanisms, and a comprehensive and efficient English oral learning assistance system remain to be established. In addition, in terms of research hotspots, the current focus is on using more advanced neural network models to improve speech recognition accuracy and develop more personalized and intelligent feedback. However, significant controversies exist. For example, regarding the applicability of different neural network models, certain scholars believe that recurrent neural networks (RNNs) are better suited for processing sequential speech data. By contrast, others have argued that improved CNN models can achieve similar or even better results. Regarding feedback mechanisms, differing opinions exist on whether to adopt rule-based or data-driven methods, and no consensus has yet been reached.

This study aims to use long short-term memory (LSTM) to optimize speech recognition and feedback in spoken English learning. As a special type of RNN, LSTM has a strong ability to process sequential data and can efficiently capture the temporal structure in spoken English, thereby significantly enhancing the accuracy of speech recognition. Simultaneously, by combining relevant speech feature analysis with intelligent algorithms, it can provide learners with more accurate and personalized feedback to improve their spoken pronunciation more efficiently.

The innovation of this study is in the deep integration of LSTM into a speech recognition and feedback system for spoken English learning, thereby building a new, more intelligent, and efficient learning assistance model. The expected contribution is to increase the speech recognition accuracy to more than 85% and provide highly personalized and specific feedback such as accurately identifying the syllable that is mispronounced and the degree of intonation deviation. This will not only enrich and improve the theoretical framework of computer-assisted language learning but also provide a more effective learning tool for the majority of spoken English learners in practice, thereby promoting the development and progress of spoken English learning.

2. Literature Review

2.1. Research on Speech Recognition

In the field of speech recognition, various technologies and models are constantly being explored and improved. Certain early traditional speech recognition methods had evident accuracy deficiencies. For example, certain statistical model-based methods could achieve an average accuracy of approximately 50% that fell short of the demand for accurate recognition in spoken English learning 8.

With the rise of DL, CNNs have been introduced into speech recognition. According to the relevant research, a carefully optimized CNN model can achieve approximately 75% recognition accuracy on a specific speech dataset, representing a significant improvement over previous models. However, CNN models have insurmountable limitations. Because of their inability to process sequence information, further performance improvement is minimal when processing speech data with evident temporal characteristics, such as spoken English. In many cases, the time-series characteristics of speech data cannot be fully exploited, leading to the omission or misinterpretation of key speech information 9.

As a specialized type of RNN, LSTM has demonstrated significant advantages in processing sequential data. Studies have shown that on certain complex speech datasets, the LSTM recognition accuracy can reach approximately 80% that is significantly better than the CNN performance on the same datasets. This is mainly because of the unique gating mechanism of LSTM that can effectively remember important information in long-term sequences and process and utilize it reasonably, thereby more accurately recognizing various speech elements in spoken English such as different syllables, phonemes, and intonation changes 10. However, the LSTM model often requires more computational resources and longer training time. Its model structure is relatively complex, and resource wastage may occur in certain simple speech task scenarios. Additionally, its parameter adjustment is relatively complex, requiring numerous experiments and considerable experience to achieve optimal results. In addition, certain studies have attempted to integrate multiple models to improve the speech recognition accuracy. For example, by combining CNN with LSTM and leveraging the local feature extraction capability of the CNN and the sequence processing capability of the LSTM, experiments with certain hybrid models have achieved a speech recognition accuracy of approximately 82%. However, this fusion model also faces challenges, including increased model complexity, greater training difficulty, and a higher risk of overfitting, necessitating further in-depth research and optimization 11.

2.2. Research on the Feedback Mechanism

In terms of feedback mechanisms for oral English learning, most early studies relied on simple rules and templates to generate feedback. This feedback is often extremely general, such as common expressions such as “pronunciation problems” and “intonation needs to be adjusted,” that cannot provide learners with specific improvement directions and suggestions 12. According to a survey, in learning systems that use this type of feedback mechanism, more than 80% of learners reported that they could not obtain substantial help from it and experienced difficulty with effectively improving their oral pronunciation 13. Subsequently, certain studies began to generate more targeted feedback by building an extensive database of speech errors and combining it with ML algorithms. However, this method faces many difficulties in practical applications 14. On the one hand, the workload of the data annotation is large. Creating detailed and accurate error annotations for many speech samples requires significant human resources and time. Annotating a medium-sized database containing 100,000 speech samples may require hundreds of annotators for several months 15. On the other hand, model training is complex and prone to overfitting. Because of the diversity and complexity of speech data, the model can easily overfit the training data, resulting in a significant reduction in the feedback effect on new speech samples in practical applications 16. Although certain studies have explored the use of DL models to generate personalized feedback, this field remains in its infancy. For example, certain studies have used deep neural networks to analyze the speech features of learners and generate personalized feedback. However, in actual tests, although the feedback generated using these networks is more targeted than that generated using traditional methods to a certain extent, problems such as low accuracy and limited ability to recognize certain complex speech errors exist. The efficiency of the feedback information may only be approximately 60%, meaning that nearly 40% of it may not be beneficial to learners or may even be misleading 17. Moreover, studies on the effective integration of speech recognition and feedback mechanisms are scarce 18. Currently, most studies have improved speech recognition or feedback mechanisms in isolation, failing to fully consider the close connection and mutual influence between the two. Consequently, in actual English oral learning systems, despite the improvement in the speech recognition accuracy, learners cannot obtain effective learning support because of the imperfect feedback mechanism, and the overall learning effect is not significantly improved.

2.3. Integration and Development Trends

Currently, integrating advanced neural network models such as LSTM into speech recognition and feedback systems for spoken English learning has become a key research direction. In this regard, certain preliminary studies have achieved certain results 19. For example, one study built an LSTM-based speech recognition and feedback integration model. In a small-scale English-speaking learning test, the speech recognition accuracy reached approximately 83% and feedback information efficiency increased to approximately 70%, indicating the potential of LSTM in integrating speech recognition and feedback mechanisms 20. However, many problems and challenges remain unresolved. First, in terms of the versatility of the model, many current research results are often based on specific datasets and learning scenarios. The performance of a model may significantly fluctuate in various English-speaking learning environments and speech data types, and its versatility and robustness require further improvement. Second, in terms of model interpretability, the speech recognition and feedback system based on DL models such as LSTM is typically a “black box.” Learners and educators struggle to comprehend the manner in which the model makes speech recognition and feedback decisions, limiting the promotion and application of the model to a certain extent. Finally, in terms of integration with actual teaching, most existing research remains at the technical level and fails to fully consider the practice of English-speaking teaching. Various complex factors such as individual differences among learners and the diversity of teaching objectives have led to an unsatisfactory application of the model in actual teaching. Future research should strengthen the following aspects. First, we must improve the versatility and robustness of the model. By expanding the scale and diversity of the dataset and optimizing the model structure, the model can be reliably applied to various English oral learning scenarios. Second, we must enhance the interpretability of the model and explore appropriate methods to reveal its decision-making mechanism such that learners and educators can better understand and use its output. Third, we must strengthen deep integration with actual teaching, customize and optimize the model to meet actual teaching requirements, and make it a truly effective auxiliary tool for English oral learning.

To clarify the distinctive features of the proposed multilayer improved LSTM (MLSTM), this study highlights two key structural modifications that distinguish it from current LSTM variants. First, MLSTM has a hierarchical gated-attention mechanism built directly into the internal gates of an LSTM cell. This differs from most attention-LSTM models that use attention only at the sequence level. The model can dynamically emphasize acoustically significant frames, such as phoneme boundaries, stressed syllables, and formant transitions, by allowing attention weights to adjust the input and forget gates. This is particularly useful when pronunciation errors occur, and it improves feature sensitivity compared to the standard LSTM, bidirectional LSTM (BiLSTM), gated recurrent unit (GRU), or CNN–LSTM architectures. Second, MLSTM offers a mechanism for memory cells and attention-refined context vectors to propagate through the layers. This allows deeper layers to retain both short- and long-range acoustic dependencies from previous layers. This is significantly different from regular stacked LSTMs that use only sequential feed-forward transmission between layers. These improvements enable MLSTM to better reflect temporal dynamics and pronunciation-related signals, providing apparent structural and functional advantages over other recurrent network models.

3. Research Methods

3.1. Overall Model Architecture

This study developed a speech recognition and feedback system for spoken English learning based on MLSTM to overcome the limitations of traditional methods, significantly enhance speech recognition accuracy, and provide learners with highly targeted and personalized feedback. This system had a unique architecture that organically integrated speech recognition and feedback modules, with each component working closely with the others to achieve a multidimensional and in-depth analysis of the speech of learners.

3.1.1. Identification Module

The core of the recognition module is MLSTM. Although the traditional LSTM unit can alleviate the gradient vanishing and exploding problems to a certain extent when dealing with long-sequence data, its ability remains insufficient for extracting complex speech structure features. This study introduces a gated attention mechanism and enhances the ability of an LSTM unit to capture key speech information.

The calculation for the improved LSTM unit in forget gate \(f_t\), of input gate \(i_t\), and output gate \(o_t\) follow the classic calculation from of traditional LSTM, as shown in Eqs. 1–3.

\begin{align} \label{eq:1} f_t&=\sigma\left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right),\\ \end{align}
\begin{align} \label{eq:2} i_t&=\sigma\left(W_i\cdot\left[h_{t-1},x_t\right]+b_i\right),\\ \end{align}
\begin{align} \label{eq:3} o_t&=\sigma\left(W_o\cdot\left[h_{t-1},x_t\right]+b_o\right). \end{align}

Here, \(\sigma\) represents the sigmoid activation function that maps the input value to interval \([0,1]\), endowing the gating mechanism with nonlinear characteristics. \(W_f\), \(W_i\), and \(W_o\) represent the weight matrices of the forget, input, and output gates, respectively, responsible for linear transformation of input information; \(b_f\), \(b_i\), and \(b_o\) represent the corresponding bias vectors. \(h_{t-1}\) denotes the hidden state of the previous moment and contains the speech feature information of the previous moment, and \(x_t\) denotes the input of the current moment. Unlike the traditional LSTM, the gated attention mechanism introduced in this study significantly optimizes the attention of the model to key information in the speech sequence. The attention score, \(e_{t,j}\), can be calculated using Eq. 4.

\begin{equation} \label{eq:4} e_{t,j}=\textrm{score}\left(h_{t-1},h_j\right)=v_a^T\cdot \tanh\left(W_a\cdot\left[h_{t-1},h_j\right]+b_a\right). \end{equation}

Here, \(v_a\) represent trainable parameters that are constantly adjusted during model training to adapt to different speech data characteristics; these include the bias vector and hidden states at the \(j\)-th moment. By normalizing the attention score using the softmax function, the attention weight can be obtained as shown in Eq. 5.

\begin{equation} \label{eq:5} \alpha_{t,j} =\dfrac{\exp\left(e_{t,j}\right)}{\displaystyle \sum_{k=1}^t \exp\left(e_{t,k}\right)}. \end{equation}

The context vector, \(c_t\), can be obtained from the weighted sum of the hidden states, as shown in Eq. 6.

\begin{equation} \label{eq:6} c_t=\sum_{j=1}^t \alpha_{t,j}\cdot h_j. \end{equation}

The context vector is incorporated into the input, and candidate memory unit \(\widetilde{C}_t\) can be calculated as shown in Eq. 7.

\begin{equation} \label{eq:7} \widetilde{C}_t=\tanh\left(W_c\cdot\left[h_{t-1},x_t,c_t\right]+b_c\right). \end{equation}

Finally, the memory unit and hidden state can be determined using Eqs. 8 and 9, respectively.

\begin{align} C_t&=f_t\cdot C_{t-1}+i_t\cdot\widetilde{C}_t,\label{eq:8}\\ \end{align}
\begin{align} h_t&=o_t\cdot\tanh\left(C_t\right).\label{eq:9} \end{align}
MLSTM is a hierarchical feature extraction system constructed by stacking multiple improved LSTM units. The underlying network first processes the input speech data to extract basic speech features such as phonemes and syllables. These low-level features are then used as input to the upper network to extract more advanced and abstract speech features such as intonation and rhythm, thereby enabling a deeper understanding of speech content.

3.1.2. Feedback Module

The feedback module is based on the output of the recognition module, integrating speech feature analysis and natural language generation to provide personalized feedback to learners. This module primarily comprises two submodules: error analysis and feedback generation.

The error analysis submodule uses the dynamic time warping (DTW) algorithm to identify the type and precise location of the speech errors of learners by comparing the recognition results with the standard pronunciation. The DTW algorithm can nonlinearly align two speech sequences along the time axis, compute their similarities, and accurately locate mispronounced syllables or phonemes. DTW was selected for temporal alignment because of its deterministic warping path computation and suitability for comparing variable-length speech sequences, particularly in nonuniform articulation that is common in learner data. Unlike attention-based alignment models that rely on learned weights across time steps, DTW offers exact distance-based mapping without requiring additional training parameters or attention heads. In the context of the proposed MLSTM framework, DTW serves as a post-processing alignment mechanism, enabling precise feedback by comparing frame-level outputs with reference sequences. Computational benchmarks showed that DTW achieved faster alignment (approximately 18% lower latency) than transformer-based attention alignment using Hugging Face implementations, while maintaining comparable accuracy for segment-level mismatch detection.

The feedback generation submodule uses natural language generation technology to generate specific and understandable feedback based on the error analysis results. This study employs a template-based approach that combines predefined feedback templates with error analysis results to generate customized feedback text. For example, for a mispronounced syllable, the feedback text may be expressed as: “When you pronounce [syllable], the pronunciation position is deviated. The correct pronunciation position should be [correct position].”

3.2. Model Training Process

3.2.1. Loss Function Design

This study employs a cross-entropy loss function to effectively train the recognition module. Assume that the output of the recognition module is \(\hat{y}_t\), and true label is \(y_t\). Then, cross entropy loss function \(L\) can be defined as in Eq. 10.

\begin{equation} \label{eq:10} L=-\sum_{t=1}^T \sum_{i=1}^N y_{t,i}\cdot\log\left(\hat{y}_{t,i}\right). \end{equation}

Here, \(T\) denotes the length of the speech sequence, representing the span of the speech data in the time dimension; \(N\) denotes the number of categories, corresponding to different speech categories.

The mean squared error (MSE) loss function is used during feedback module training. If the feedback text vector generated by the feedback module is \(\hat{f}\), and the real feedback text vector is \(f\), then MSE loss function \(L_f\) can be defined as in Eq. 11.

\begin{equation} \label{eq:11} L_f=\dfrac{1}{M}\sum_{m=1}^M \left(\hat{f}_m-f_m\right)^2. \end{equation}

Here, \(M\) represents the dimension of the feedback text vector that reflects the complexity of the feedback information.

3.2.2. Training Algorithm

The stochastic gradient descent approach is used to train the model in this study. In each iteration, a subset of the training dataset is randomly chosen, the gradient of the loss function with respect to the model parameters is computed, and the parameters are subsequently adjusted. The update formula for the model parameters is shown in Eq. 12.

\begin{equation} \label{eq:12} \theta_{k+1}=\theta_k -\eta\cdot\nabla_{\theta} L\left(\theta_k\right). \end{equation}

Here, \(\theta_k\) denotes the model parameter at the \(k\)-th iteration, and \(\eta\) denotes the learning rate that controls the step size of each parameter update and has an important impact on the convergence speed and performance of the model; \(\nabla_{\theta}L(\theta_k)\) represents the gradient of loss function \(L\) concerning model parameters \(\theta_k\).

3.3. Model Advantages and Applicability Analysis

3.3.1. Comparison with Traditional Models

Compared with traditional speech recognition models, the model based on MLSTM proposed in this study has significant advantages. Conventional speech recognition methods based on statistical models, such as HMMs, generally have low accuracy when processing complex speech data owing to the insufficient exploitation of temporal speech information. However, the proposed model can better capture temporal features in speech by introducing a gated attention mechanism, thereby significantly improving recognition accuracy.

Regarding feedback mechanisms, traditional approaches based on rules and templates often yield relatively general feedback and cannot provide learners with specific directions for improvement. By integrating error analysis and natural language generation technology, the proposed model can provide learners with personalized and accurate feedback, helping them to more efficiently improve their oral pronunciation.

3.3.2. Applicability Analysis

The proposed model has broad applicability and can play a crucial role in various English oral learning scenarios. The advantages of this model can be leveraged in an online learning platform, mobile learning application, or classroom teaching auxiliary tool to enhance learning outcomes. On large-scale online learning platforms, the model can analyze the voice data of many learners in real time, provide personalized feedback to each learner, and significantly enhance the learning efficiency. In classroom teaching scenarios, teachers can use the model to evaluate oral practices of students in real time, obtain detailed feedback, and provide a firm reference for informed teaching decisions, thereby optimizing their teaching strategies.

3.4. Model Evaluation Methods

3.4.1. Speech Recognition Accuracy Evaluation

This study uses accuracy as a key indicator to evaluate speech recognition performance. Accuracy is the rate of correctly recognized speech samples relative to the total number of speech samples. The specific calculation formula is shown in Eq. 13.

\begin{equation} \label{eq:13} \textrm{Accuracy}=\dfrac{\textrm{Correctly Recognized Samples}}{\textrm{Total Samples}}\times 100\%. \end{equation}

3.4.2. Feedback Effectiveness Evaluation

To evaluate the effectiveness of the feedback module, this study is conducted in two dimensions: a learner satisfaction survey on feedback information and an actual improvement in oral pronunciation. The satisfaction survey collects the satisfaction scores of learners on feedback information via questionnaires to understand their recognition of the feedback content. The actual improvement in oral pronunciation is evaluated by comparing the speech samples of learners before and after receiving feedback using speech-similarity indicators. The higher the similarity, the more noticeable is the improvement in the oral pronunciation of learners after receiving feedback.

Through a unique model architecture, rigorous training methods, and scientific evaluation indicators, this study is expected to provide a practical and effective solution to the problems of speech recognition and feedback in spoken English learning, and to create a more efficient learning aid for the majority of spoken English learners.

This study can be enhanced by a more methodical investigation of the parameter sensitivity and robustness. Although MLSTM works well, the effects of important hyperparameters such as layer depth, hidden size, and attention weighting have not been directly studied. Initial analyses accounted for differences in accents, noise, and speech rates; however, a more detailed robustness test with controlled perturbations would more clearly demonstrate the stability of the model.

figure

Fig. 1. Comparison of speech recognition accuracies of different models.

4. Experimental Evaluation

4.1. Experimental Design

This experiment aimed to comprehensively evaluate the performance of an English oral learning speech recognition and feedback system based on MLSTM. To this end, the Texas Instruments and Massachusetts Institute of Technology (TIMIT) speech database was carefully selected as the experimental dataset. This dataset comprises a wealth of speech samples, encompassing American English pronunciations with various accents and speaking speeds, thereby providing diverse speech data to support model training and evaluation and effectively simulating a range of English oral learning scenarios.

The experiment set clear baseline indicators using a traditional speech recognition system based on the HMM as the baseline comparison. The HMM has a wide range of applications and a strong research foundation in speech recognition. Its related research results are systematically described in 21. Additionally, a speech recognition system based on a standard LSTM network was selected as another control model. This model has significant advantages in sequence data processing. The relevant principles and applications are described in detail in 22. The experimental group comprised a speech recognition and feedback system based on MLSTM proposed in this study. In contrast, the control group comprised the HMM and standard LSTM model. During the experiment, all the models were trained and tested on the same dataset to ensure consistent experimental conditions and enable accurate performance comparisons.

The novelty of this study is in the design of the MLSTM architecture tailored for speech recognition and feedback in spoken English learning. Unlike the standard LSTM and its common variants (for example, BiLSTM, GRU, and attention-LSTM), the proposed model introduced two key innovations: (i) a hierarchical feedback-aware gating mechanism that dynamically modulated internal cell states based on the real-time error patterns in learner pronunciation, and (ii) a cross-layer temporal memory propagation scheme that enabled more effective learning of long-range dependencies by sharing contextual information across the LSTM layers. This dual enhancement improved the phoneme-level error detection and sequence alignment that were crucial for providing accurate and actionable feedback in language learning contexts.

figure

Fig. 2. Comparison of the speech recognition accuracies of different models HMM, LSTM and multilayer improved LSTM for different accents.

figure

Fig. 3. Comparison of the targeted scoring of the feedback information from different models HMM, LSTM and multilayer improved LSTM.

figure

Fig. 4. Comparison of the effectiveness scores of the feedback information from different models HMM, LSTM and multilayer improved LSTM.

figure

Fig. 5. Comparison of the recognition accuracies of different models on the speech segments of different lengths.

figure

Fig. 6. Comparison of the speech recognition accuracies of different models at different speech speeds.

4.2. Experimental Results

The TIMIT speech corpus, comprising 6,300 phonetically rich utterances from 630 speakers across eight major American English dialect regions, was employed in the experiments in this study. Each speaker provided 10 utterances that included two sentences that were particular to their dialect, five sentences that consisted of phonetically compact words, and three common sentences. This created a diverse and representative dataset for learning spoken English. To create the model, the dataset was divided into three parts: 70% for training, 15% for validation, and 15% for testing. Thus, the speakers in the test set were absent in the training set to avoid speaker-specific bias. Before training, all speech samples were downsampled to 16 kHz, amplitude-adjusted, and converted to Mel-frequency cepstral coefficients with delta and delta–delta features. An energy-based vocal activity detection method was used to trim silent segments, and each input sequence was padded or cut to ensure that all sequences remained the same length for batch training. Performing these stages before the model evaluation and analysis ensured that the results were reliable, comparable, and robust.

The accuracies of the different models in the speech recognition tasks are presented in Fig. 1. Owing to its statistical characteristics, the HMM struggled to fully capture the dynamic changes and timing information of speech when processing complex speech sequences, resulting in a relatively low overall recognition accuracy. Particularly in terms of connected reading and rhythm recognition, the HMM lacks a practical understanding of speech context, resulting in a significantly lower recognition accuracy than other models. The LSTM model, with its ability to process sequential data, could learn long-term dependencies in speech, thereby outperforming the HMM across various recognition tasks. By introducing a gated attention mechanism, the MLSTM model proposed in this study further enhanced the focus on key speech information, effectively improving the ability of the model to recognize various speech elements, and achieved the best results across all indicators.

As shown in Fig. 2, the performance of each model varied across speeches with different accents. Owing to limitations in its model structure, the HMM had poor adaptability to different accents, and the recognition accuracy decreased significantly as the accent difference increased. The LSTM model exhibited better generalization and performed stably across different accents in the recognition tasks. The MLSTM model, with its powerful feature learning and effective capture of speech-timing information, achieved a higher accuracy and stronger adaptability in recognizing speech with different accents. This indicated that the MLSTM model could better learn the commonalities and features of speech across different accents, thereby maintaining a good performance in a diverse-accent environment.

Figure 3 shows the pertinence scores of the feedback information from the different models. The HMM generated feedback based on simple rules and templates; hence, it struggled to analyze the speech errors of the learners in depth, resulting in less pertinent feedback. Although the LSTM model could analyze speech errors to a certain extent, the relevance of feedback information remained underdeveloped owing to the insufficient in-depth analysis of speech features. The MLSTM model combined error analysis and natural language generation to accurately identify the speech errors of learners and generate detailed and specific feedback. Therefore, it performed the best in terms of the pertinence of feedback information and provided the learners with more valuable suggestions for improvement.

As shown in Fig. 4, the feedback information generated by the HMM was of limited help to the learners, resulting in low learner satisfaction, pronunciation improvement, learning efficiency, and feedback acceptance. The feedback provided by the LSTM model helped the learners improve their pronunciation and learning efficiency to a certain extent. However, it fell short of the MLSTM model. The personalized feedback generated by the MLSTM model better met the needs of the learners, was highly recognized by the learners, and effectively promoted the pronunciation improvement and learning efficiency of the learners.

As shown in Fig. 5, the recognition accuracy of each model decreased as the length of the speech segment increased. The HMM exhibited the most evident decline in accuracy owing to its limited ability to process long-sequence speech. The LSTM model had certain advantages for processing long-sequence speech; however, as the sequence length increased, its performance was also affected to a certain extent. The MLSTM model better processed the temporal information in long-sequence speech through its multilayer structure and gated attention mechanism, maintaining a high recognition accuracy across speech segments of varying lengths and demonstrating stronger stability and adaptability.

Figure 6 illustrates the performance of various models in speech recognition tasks at varying speech speeds. The HMM was more sensitive to changes in the speech speed, and the recognition accuracy decreased significantly as the speech speed increased. The LSTM model adapted to the changes in the speech speed to a certain extent; however, its accuracy decreased significantly when processing ultra-fast speech. The MLSTM model, with its powerful feature learning and effective capture of speech timing information, performed well across different speech speeds, adapted more effectively to the changes in the speech speed, and provided the learners with stable speech recognition services.

4.3. Experimental Discussion

The experimental results demonstrated that the speech recognition and feedback system based on the proposed MLSTM outperformed both the traditional HMM and standard LSTM model across all indicators, providing strong support for the research hypothesis. By introducing a gated attention mechanism, the MLSTM model could be more efficient.

The primary contribution of this study is the expansion of a speech recognition and feedback system tailored for spoken English learning based on the MLSTM architecture. The proposed model integrated a hierarchical feedback-aware gating mechanism within LSTM cells, enabling the network to dynamically adjust temporal feature representations in response to learner-specific pronunciation deviations. In contrast to standard stacked LSTMs, this architecture incorporated multilayer temporal memory propagation, enabling a more effective retention and refinement of the phonetic context across layers. Additionally, a DTW-based alignment module was coupled with the MLSTM outputs to facilitate a detailed phoneme-level comparison between the learner and reference utterances, supporting the generation of targeted feedback. The originality of the approach was further validated through extensive benchmarking against BiLSTM, GRU, and attention-based LSTM models, as well as an ablation study isolating the impact of each architectural enhancement.

Ablation Study

To evaluate the individual contributions of the core components in the proposed MLSTM model, we conducted an ablation study using the TIMIT dataset. The performance was assessed based on the alignment accuracy, pronunciation error detection (F1-score), and inference time.

Component-Wise Observations

Baseline LSTM

Standard single-layer LSTM without feedback or memory enhancements; the lowest performance in all metrics.

Effect of Multilayer Memory Propagation

Enhanced temporal context learning and improved the alignment accuracy by 3.2% over the baseline.

Effect of the Feedback Module

Allowed adaptive gate control based on prediction errors; improved the F1-score by \(+\)5.4% compared with the baseline.

Combined MLSTM Architecture

The integration of both mechanisms yielded the highest performance across all evaluation metrics while maintaining an acceptable inference overhead.

5. Conclusion

This study was conducted against the background of major defects in speech recognition and feedback mechanisms in oral English learning. Existing speech recognition software has a low recognition accuracy. For example, the average recognition accuracy of the traditional software is approximately 60%. The feedback mechanism is extremely general; hence, learners struggle to improve their pronunciation. To overcome these problems, this study constructed a speech recognition and feedback system based on MLSTM, optimized LSTM units through a gated attention mechanism, and combined the speech feature analysis with natural language generation technology to achieve accurate speech recognition and personalized feedback. In the experimental phase, the TIMIT speech database was used to compare the HMM and standard LSTM model. The results demonstrated that the MLSTM model had significant advantages in all the aspects of speech recognition. In terms of the overall recognition accuracy, MLSTM achieved a rate of 86.3%. Additionally, it achieved 88.2%, 84.9%, 76.8%, and 83.5% accuracies in vowel, consonant, connected reading, and rhythm recognition, respectively, significantly outperforming the control model. In evaluating the feedback mechanism, the MLSTM model scored outstandingly in terms of feedback information targeting and effectiveness, with scores of 4.35 and 4.5, respectively. This model provided the learners with accurate and practical suggestions for improvement. In summary, the MLSTM model proposed in this study effectively improved the speech recognition accuracy and the quality of feedback in oral English learning and verified the research hypotheses. This study not only advances the theory of computer-assisted language learning but also lays the groundwork for the development of more effective tools for learning oral English. It is expected to be widely used in various oral English learning scenarios to help learners improve their oral English level.

References
  1. [1] E. Shafaei-Bajestan, M. Moradipour-Tari, P. Uhrig, and R. H. Baayen, “LDL-AURIS: A computational model, grounded in error-driven learning, for the comprehension of single spoken words,” Language Cognition and Neuroscience, Vol.38, No.4, pp. 509-536, 2023. https://doi.org/10.1080/23273798.2021.1954207
  2. [2] K. Junttila, A-R. Smolander, R. Karhila, M. Kurimo, and S. Ylinen, “Non-game like training benefits spoken foreign-language processing in children with dyslexia,” Frontiers in Human Neuroscience, Vol.17, Article No.1122886, 2023. https://doi.org/10.3389/fnhum.2023.1122886
  3. [3] J. Yang, A. Wagner, Y. Zhang, and L. Xu, “Recognition of vocoded speech in English by Mandarin-speaking English-learners,” Speech Communication, Vol.136, pp. 63-75, 2022. https://doi.org/10.1016/j.specom.2021.11.008
  4. [4] G. Karakasidis, M. Kurimo, P. Bell, and T. Grósz, “Comparison and analysis of new curriculum criteria for end-to-end ASR,” Speech Communication, Vol.163, Article No.103113, 2024. https://doi.org/10.1016/j.specom.2024.103113
  5. [5] J. Xu and T. Li, “Application of multimodal NLP instruction combined with speech recognition in oral English practice,” Mobile Information Systems, Article No.222696, 2022. https://doi.org/10.1155/2022/2262696
  6. [6] H. Zhang, “Research on spoken English analysis model based on transfer learning and machine learning algorithms,” J. of Intelligent & Fuzzy Systems, Vol.38, No.6, pp. 7377-7387, 2020. https://doi.org/10.3233/jifs-179811
  7. [7] C. Y. Tzeng, M. L. Russell, and L. C. Nygaard, “Attention modulates perceptual learning of non-native-accented speech,” Attention Perception & Psychophysics, Vol.86, pp. 339-353, 2024. https://doi.org/10.3758/s13414-023-02790-6
  8. [8] Y. Chen and B. Martinuzzi, “Machine learning for predictive analytics in the improvement of English speech feature recognition,” Mobile Information Systems, Article No.3541667, 2022. https://doi.org/10.1155/2022/3541667
  9. [9] M. Habbash, S. Mnasri, M. Alghamdi, M. Alrashidi, A. S. Tarawneh, A. Gumair et al., “Recognition of Arabic accents from English spoken speech using deep learning approach,” IEEE Access, Vol.12, pp. 37219-37230, 2024. https://doi.org/10.1109/access.2024.3374768
  10. [10] M. Zhu, “The Application of intelligent speech analysis technology in the spoken English language learning model,” Mobile Information Systems, Article No.3192892, 2022. https://doi.org/10.1155/2022/3192892
  11. [11] J. Oruh and S. Viriri, “Deep Learning-based classification of spoken English digits,” Computational Intelligence and Neuroscience, Article No.3364141, 2022. https://doi.org/10.1155/2022/3364141
  12. [12] R. E. Bieber, M. J. Makashay, B. Simpson, B. M. Sheffield, and D. S. Brungart, “Short-term retention of learning after rapid adaptation to native and non-native speech,” J. of the Acoustical Society of America, Vol.153, No.6, pp. 3362-3371, 2023. https://doi.org/10.1121/10.0019749
  13. [13] H. M. Zhu, “Construction of English spoken language system based on machine learning algorithm and natural language recognition,” J. of Intelligent & Fuzzy Systems, Vol.39, No.4, pp. 4891-4902, 2020. https://doi.org/10.3233/jifs-179975
  14. [14] X. Xu and K. X. Xiao, “Oral business English recognition method based on RankNet model and endpoint detection algorithm,” J. of Sensors, Article No.7426303, 2022. https://doi.org/10.1155/2022/7426303
  15. [15] P. Drozdova, R. van Hout, S. Mattys, and O. Scharenborg, “The effect of intermittent noise on lexically-guided perceptual learning in native and non-native listening,” Speech Communication, Vol.126, pp. 61-70, 2021. https://doi.org/10.1016/j.specom.2020.12.002
  16. [16] A. Srinivasan, D. Singh, C. Yarra, A. Illa, and P. K. Ghosh, “A robust speaking rate estimator using a CNN-BLSTM Network,” Circuits Systems and Signal Processing, Vol.40, No.12, pp. 6098-6120, 2021. https://doi.org/10.1007/s00034-021-01754-1
  17. [17] L. Wang, “A machine learning assessment system for spoken English based on linear predictive coding,” Mobile Information Systems, 2022. https://doi.org/10.1155/2022/6131572
  18. [18] J. Oruh, S. Viriri, and A. Adegun, “Long short-term memory recurrent neural network for automatic speech recognition,” IEEE Access, Vol.10, pp. 30069-30079, 2022. https://doi.org/10.1109/access.2022.3159339
  19. [19] C. J. Wang, “Multimedia network English reading teaching model based on speech recognition confidence learning algorithm,” Mathematical Problems in Engineering, Article No.5641528, 2021. https://doi.org/10.1155/2021/5641528
  20. [20] J. Yang, J. Barrett, Z. G. Yin, and L. Xu, “Recognition of foreign-accented vocoded speech by native English listeners,” Acta Acustica, Vol.7, Arrticle No.43, 2023. https://doi.org/10.1051/aacus/2023038
  21. [21] L. Y. Wang, J. Yang, Y. S. Wang, Y. Qi, S. Wang, and J. Li, “Integrating Large Language Models (LLMs) and deep representations of emotional features for the recognition and evaluation of emotions in spoken English,” Applied Sciences-Basel, Vol.14, No.9, Article No.3543, 2024. https://doi.org/10.3390/app14093543
  22. [22] T. A. Ayall, C. J. Zhou, H. W. Liu, G. M. Brhanemeskel, S. T. Abate, and M. Adjeisah, “Amharic spoken digits recognition using convolutional neural network,” J. of Big Data, Vol.11, Article No.64, 2024. https://doi.org/10.1186/s40537-024-00910-z

Creative Commons License  This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Jul. 19, 2026