Research Paper:
CxLSTM: Deploying xLSTM-Based Custom Keyword Spotting to Edge Devices
Haitao Cao
, Tianyou Xiao, Chao Chang
, Luyao Teng
, Xi Liu
, and Dan Deng

School of Information Engineering, Guangzhou Polytechnic University
No.1342 Shiliang Road, Shawan Street, Panyu District, Guangzhou, Guangdong 511483, China
Corresponding author
Keyword spotting (KWS) serves as the critical entry point for voice interaction in intelligent systems. Traditional KWS systems rely on predefined keywords with limited customization, whereas existing custom keyword spotting (CKWS) approaches incur heavy computational overhead, limiting their deployment on resource-limited edge devices. To address these challenges, we present CxLSTM, an optimized extended LSTM module with only 57.6K parameters, 318.6 KB static memory, 593.1 KB peak memory, and 3.54 MACs per inference. Based on CxLSTM, we propose an on-device CKWS method and perform extensive experiments. In offline evaluation, the proposed method obtains an equal error rate (EER) of 0.27 and an area under the ROC curve (AUC) of 0.78 on the public Google Speech Commands V2 dataset. Superior performance is further validated on a private Mandarin dataset, yielding an EER of 0.06 and an AUC of 0.98. Furthermore, in real-world tests for Mandarin CKWS across three scenarios, the proposed method achieves a recognition accuracy above 95% on Android devices, with a false alarm rate below 0.75/24 h and an average latency of 2.1 ms. When deployed on STM32 microcontrollers, it maintains an accuracy of over 92.3%, with moderately increased false alarms (15.8/24 h) and latency (264.7 ms). Despite performance degradation on low-power microcontrollers due to limited hardware resources, the proposed CKWS method still enables efficient voice interaction on edge devices.
Custom keyword spotting with CxLSTM
- [1] O. Friha, M. A. Ferrag, B. Kantarci, B. Cakmak, A. Ozgun, and N. Ghoualmi-Zine, “Llm-based edge intelligence: A comprehensive survey on architectures, applications, security and trustworthiness,” IEEE Open J. of the Communications Society, Vol.5, pp. 5799-5856, 2024. https://doi.org/10.1109/OJCOMS.2024.3456549
- [2] S. Anusha, M. Ezhilvandan, S. Gayathri, V. Sivaraman et al., “Emerging trends in edge AI for Industry 4.0 and 5.0: Technologies, applications, and challenges,” AI Innovations in Service and Tourism Marketing, pp. 211-228, 2024. https://doi.org/10.4018/979-8-3693-7909-7.ch010
- [3] Y. Sone and J. Woo, “Design of a Human-Centric Robotic System for User Support Based on Gaze Information,” J. Adv. Comput. Intell. Intell. Inform., Vol.29, No.4, pp. 796-802, 2025. https://doi.org/10.20965/jaciii.2025.p0796
- [4] Z. Zhang, R. Li, and K. Chen, “A Parallel CNN-Transformer Framework for Speech Age Recognition,” J. Adv. Comput. Intell. Intell. Inform., Vol.29, No.5, pp. 1137-1144, 2025. https://doi.org/10.20965/jaciii.2025.p1137
- [5] C. Mao and S. Liu, “A Study on Speech Recognition by a Neural Network Based on English Speech Feature Parameters,” J. Adv. Comput. Intell. Intell. Inform., Vol.28, No.3, pp. 679-684, 2024. https://doi.org/10.20965/jaciii.2024.p0679
- [6] M. Li, “A Lightweight Architecture for Query-by-Example Keyword Spotting on Low-Power IoT Devices,” IEEE Trans. on Consumer Electronics, Vol.69, No.1, pp. 65-75, 2023. https://doi.org/10.1109/TCE.2022.3213075
- [7] Y. Wei, Z. Gong, S. Yang, K. Ye, and Y. Wen, “EdgeCRNN: an edge-computing oriented model of acoustic feature enhancement for keyword spotting,” J. of Ambient Intelligence and Humanized Computing, Vol.13, No.3, pp. 1525-1535, 2022. https://doi.org/10.1007/s12652-021-03022-1
- [8] L. Lei, G. Yuan, H. Yu, D. Kong, and Y. He, “Multilingual Customized Keyword Spotting Using Similar-Pair Contrastive Learning,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, Vol.31, pp. 2437-2447, 2023. https://doi.org/10.1109/TASLP.2023.3284523
- [9] M. Rusci and T. Tuytelaars, “Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems,” Interspeech 2023, pp. 2768-2772, 2023. https://doi.org/10.21437/Interspeech.2023-1904
- [10] Q. Shao, J. Hou, Y. Hu, Q. Wang, L. Xie, and X. Lei, “Target speaker extraction for customizable query-by-example keyword spotting,” 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conf. (APSIPA ASC), pp. 672-678, 2021.
- [11] Z. Yang, S. Sun, J. Li, X. Zhang, X. Wang, L. Ma, and L. Xie, “CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework based on Cascaded Transducer-Transformer,” Interspeech 2022, pp. 1681-1685, 2022. https://doi.org/10.21437/Interspeech.2022-10258
- [12] H. Yan, Q. He, and W. Xie, “CRNN-CTC based mandarin keywords spotting,” ICASSP 2020-2020 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7489-7493, 2020. https://doi.org/10.1109/ICASSP40776.2020.9054618
- [13] S. Li and H. Zhang, “Keyword spotting based on CTC and Similarity Matching for Chinese Speech,” 2023 IEEE/ACIS 23rd Int. Conf. on Computer and Information Science (ICIS), pp. 79-84, 2023. https://doi.org/10.1109/ICIS57766.2023.10210227
- [14] T. N. Sainath and C. Parada, “Convolutional neural networks for small-footprint keyword spotting,” Interspeech 2015, pp. 1478-1482, 2015. https://doi.org/10.21437/Interspeech.2015-352
- [15] G. Chen, C. Parada, and T. N. Sainath, “Query-by-example keyword spotting using long short-term memory networks,” 2015 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 5236-5240, 2015. https://doi.org/10.1109/ICASSP.2015.7178970
- [16] S. Ö. Arik, M. Kliegl, R. Child, J. Hestness, A. Gibiansky, C. Fougner, R. Prenger, and A. Coates, “Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting,” Interspeech 2017, pp. 1606-1610, 2017. https://doi.org/10.21437/Interspeech.2017-1737
- [17] S. Majumdar and B. Ginsburg, “MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition,” Interspeech 2020, pp. 3356-3360, 2020. https://doi.org/10.21437/Interspeech.2020-1058
- [18] B. Kim, S. Chang, J. Lee, and D. Sung, “Broadcasted Residual Learning for Efficient Keyword Spotting,” Interspeech 2021, pp. 4538-4542, 2021. https://doi.org/10.21437/Interspeech.2021-383
- [19] S. Choi, S. Seo, B. Shin, H. Byun, M. Kersner, B. Kim, D. Kim, and S. Ha, “Temporal Convolution for Real-Time Keyword Spotting on Mobile Devices,” Interspeech 2019, pp. 3372-3376, 2019. https://doi.org/10.21437/Interspeech.2019-1363
- [20] J. Huang, W. Gharbieh, Q. Wan, H. S. Shim, and H. C. Lee, “QbyE-MLPMixer: Query-by-Example Open-Vocabulary Keyword Spotting using MLPMixer,” Interspeech 2022, pp. 5200-5204, 2022. https://doi.org/10.21437/Interspeech.2022-11080
- [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, pp. 6000-6010, 2017.
- [22] A. Berg, M. O’Connor, and M. T. Cruz, “Keyword Transformer: A Self-Attention Model for Keyword Spotting,” Interspeech 2021, pp. 4249-4253, 2021. https://doi.org/10.21437/Interspeech.2021-1286
- [23] K. Ding, M. Zong, J. Li, and B. Li, “LETR: A Lightweight and Efficient Transformer for Keyword Spotting,” 2022 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7987-7991, 2022. https://doi.org/10.1109/ICASSP43922.2022.9747295
- [24] N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” arXiv preprint, arXiv:2001.04451, 2020. https://doi.org/10.48550/arXiv.2001.04451
- [25] Y. Li, G. Yuan, Y. Wen, J. Hu, G. Evangelidis, S. Tulyakov, Y. Wang, and J. Ren, “EfficientFormer: Vision Transformers at MobileNet Speed,” Advances in Neural Information Processing Systems, Vol.35, pp. 12934-12949, 2022. https://doi.org/10.52202/068431-0940
- [26] M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, “xLSTM: Extended Long Short-Term Memory,” Advances in Neural Information Processing Systems, Vol.37, pp. 107547-107603, 2024. https://doi.org/10.52202/079017-3417
- [27] S. Yadav, S. Theodoridis, and Z.-H. Tan, “AxLSTMs: learning self-supervised audio representations with xLSTMs,” Interspeech 2025, pp. 3524-3528, 2025. https://doi.org/10.21437/Interspeech.2025-426
- [28] B. Alkin, M. Beck, K. Pöppel, S. Hochreiter, and J. Brandstetter, “Vision-LSTM: xLSTM as Generic Vision Backbone,” The Thirteenth Int. Conf. on Learning Representations, 2025.
- [29] Z. Liu, T. Li, and P. Zhang, “RNN-T based open-vocabulary keyword spotting in mandarin with multi-level detection,” 2021 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 5649-5653, 2021. https://doi.org/10.1109/ICASSP39728.2021.9413588
- [30] S.-G. Leem, I.-C. Yoo, and D. Yook, “Multitask Learning of Deep Neural Network-Based Keyword Spotting for IoT Devices,” IEEE Trans. on Consumer Electronics, Vol.65, No.2, pp. 188-194, 2019. https://doi.org/10.1109/TCE.2019.2899067
- [31] E. van der Westhuizen, H. Kamper, R. Menon, J. Quinn, and T. Niesler, “Feature learning for efficient ASR-free keyword spotting in low-resource languages,” Computer Speech & Language, Vol.71, Article No.101275, 2022. https://doi.org/10.1016/j.csl.2021.101275
- [32] K. R, V. K. Kurmi, V. Namboodiri, and C. V. Jawahar, “Generalized Keyword Spotting using ASR embeddings,” Interspeech 2022, pp. 126-130, 2022. https://doi.org/10.21437/Interspeech.2022-10450
- [33] Y. Yang, B. Du, Y. Zhang, W. Wang, and Y. Li, “BBS-KWS: The Mandarin Keyword Spotting System Won the Video Keyword Wakeup Challenge,” arXiv preprint, arXiv:2112.01757, 2021. https://doi.org/10.48550/arXiv.2112.01757
- [34] E. T. Mekonnen, A. Brutti, and D. Falavigna, “End-to-end low resource keyword spotting through character recognition and beam-search re-scoring,” 2022 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 8182-8186, 2022. https://doi.org/10.1109/ICASSP43922.2022.9746012
- [35] B. Labrador, G. Zhao, I. L. Moreno, A. S. Scarpati, L. Fowl, and Q. Wang, “Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spotting,” 2023 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2023. https://doi.org/10.17023/r23c-4m17
- [36] A. Parnami and M. Lee, “Few-Shot Keyword Spotting With Prototypical Networks,” Proc. of the 2022 7th Int. Conf. on Machine Learning Technologies (ICMLT ’22), pp. 277-283, 2022. https://doi.org/10.1145/3529399.3529443
- [37] J. Lin, K. Kilgour, D. Roblek, and M. Sharifi, “Training keyword spotters with limited and synthesized speech data,” 2020 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7474-7478, 2020. https://doi.org/10.1109/ICASSP40776.2020.9053193
- [38] M. Xu and X.-L. Zhang, “Depthwise Separable Convolutional ResNet with Squeeze-and-Excitation Blocks for Small-Footprint Keyword Spotting,” Interspeech 2020, pp. 2547-2551, 2020. https://doi.org/10.21437/Interspeech.2020-1045
- [39] M. Sun, A. Raju, G. Tucker, S. Panchapagesan, G. Fu, A. Mandal, S. Matsoukas, N. Strom, and S. Vitaladevuni, “Max-pooling loss training of long short-term memory networks for small-footprint keyword spotting,” 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 474-480, 2016. https://doi.org/10.1109/SLT.2016.7846306
- [40] M. Mazumder, C. Banbury, J. Meyer, P. Warden, and V. J. Reddi, “Few-Shot Keyword Spotting in Any Language,” Interspeech 2021, pp. 4214-4218, 2021. https://doi.org/10.21437/Interspeech.2021-1966
- [41] Y. Xi, T. Tan, W. Zhang, B. Yang, and K. Yu, “Text adaptive detection for customizable keyword spotting,” 2022 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 6652-6656, 2022. https://doi.org/10.1109/ICASSP43922.2022.9746647
- [42] B. Wei, M. Yang, T. Zhang, X. Tang, X. Huang, K. Kim, J. Lee, K. Cho, and S.-U. Park, “End-to-End Transformer-Based Open-Vocabulary Keyword Spotting with Location-Guided Local Attention,” Interspeech 2021, pp. 361-365, 2021. https://doi.org/10.21437/Interspeech.2021-1335
- [43] H. Cao, X. Liu, Z. Tan, Z. Yang, and X. Qin, “Custom Mandarin Keyword Spotting with Extended Long Short-Term Memory,” IAENG Int. J. of Computer Science, Vol.51, No.12, Article No.1933, 2024.
- [44] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” Interspeech 2019, pp. 2613-2617, 2019. https://doi.org/10.21437/Interspeech.2019-2680
- [45] T. Devries and G. W. Taylor, “Improved Regularization of Convolutional Neural Networks with Cutout,” arXiv preprint, arXiv:1708.04552, 2017. https://doi.org/10.48550/arXiv.1708.04552
- [46] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint, arXiv:1804.03209, 2018. https://doi.org/10.48550/arXiv.1804.03209
- [47] A. Mesaros, T. Heittola, A. Diment, B. Elizalde, A. Shah, E. Vincent, B. Raj, and T. Virtanen, “DCASE 2017 Challenge Setup: Tasks, Datasets and Baseline System,” Proc. of the Detection and Classification of Acoustic Scenes and Events 2017 Workshop (DCASE2017), pp. 85-92, 2017.
- [48] N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” Workshop on Detection and Classification of Acoustic Scenes and Events, 2019. https://doi.org/10.33682/006b-jx26
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.