JACIII Vol.27 No.6 pp. 1096-1107
doi: 10.20965/jaciii.2023.p1096

Research Paper:

Real-Time Dynamic Gesture Recognition Algorithm Based on Adaptive Information Fusion and Multi-Scale Optimization Transformer

Guangda Lu*,**, Wenhao Sun*,**,† ORCID Icon, Zhuanping Qin*,**, and Tinghang Guo*,**

*School of Automation and Electrical Engineering, Tianjin University of Technology and Education
No.1310 Dagu South Road, Hexi District, Tianjin 300222, China

**Tianjin Key Laboratory of Information Sensing & Intelligent Control
No.1310 Dagu South Road, Hexi District, Tianjin 300222, China

Corresponding author

April 4, 2023
July 9, 2023
November 20, 2023
dynamic gesture recognition, Transformer, optical flow, information fusion

Gesture recognition is a popular technology in the field of computer vision and an important technical mean of achieving human-computer interaction. To address problems such as the limited long-range feature extraction capability of existing dynamic gesture recognition networks based on convolutional operators, we propose a dynamic gesture recognition algorithm based on spatial pyramid pooling Transformer and optical flow information fusion. We take advantage of Transformer’s large receptive field to reduce model computation while improving the model’s ability to extract features at different scales by embedding spatial pyramid pooling. We use the optical flow algorithm with the global motion aggregation module to obtain an optical flow map of hand motion, and to extract the key frames based on the similarity minimization principle. We also design an adaptive feature fusion method to fuse the spatial and temporal features of the dual channels. Finally, we demonstrate the effectiveness of model components on model recognition enhancement through ablation experiments. We conduct training and validation on the SCUT-DHGA dynamic gesture dataset and on a dataset we collected, and we perform real-time dynamic gesture recognition tests using the trained model. The results show that our algorithm achieves high accuracy even while keeping the parameters balanced. It also achieves fast and accurate recognition of dynamic gestures in real-time tests.

Cite this article as:
G. Lu, W. Sun, Z. Qin, and T. Guo, “Real-Time Dynamic Gesture Recognition Algorithm Based on Adaptive Information Fusion and Multi-Scale Optimization Transformer,” J. Adv. Comput. Intell. Intell. Inform., Vol.27 No.6, pp. 1096-1107, 2023.
Data files:
  1. [1] A. Carfì and F. Mastrogiovanni, “Gesture-based human–machine interaction: Taxonomy, problem definition, and analysis,” IEEE Trans. on Cybernetics, Vol.53, No.1, pp. 497-513, 2023.
  2. [2] X. Lu et al., “Development of a wearable gesture recognition system based on two-terminal electrical impedance tomography,” IEEE J. of Biomedical and Health Informatics, Vol.26, No.6, pp. 2515-2523, 2022.
  3. [3] S. Shin et al., “Hand gesture recognition using EGaIn-silicone soft sensors,” Sensors, Vol.21, No.9, Article No.3204, 2021.
  4. [4] G. Benitez-Garcia et al., “Improving real-time hand gesture recognition with semantic segmentation,” Sensors, Vol.21, No.2, Article No.356, 2021.
  5. [5] G. Krishnan et al., “Spatio-temporal continuous gesture recognition under degraded environments: Performance comparison between 3D integral imaging (InIm) and RGB-D sensors,” Optics Express, Vol.29, No.19, pp. 30937-30951, 2021.
  6. [6] D. C. Silpani, K. Suematsu, and K. Yoshida, “A feasibility study on hand gesture intention interpretation based on gesture detection and speech recognition,” J. Adv. Comput. Intell. Intell. Inform., Vol.26, No.3, pp. 375-381, 2022.
  7. [7] K. M. Vamsikrishna, D. P. Dogra, and M. S. Desarkar, “Computer-vision-assisted palm rehabilitation with supervised learning,” IEEE Trans. on Biomedical Engineering, Vol.63, No.5, pp. 991-1001, 2016.
  8. [8] C. I. Patel et al., “Histogram of oriented gradient-based fusion of features for human action recognition in action video sequences,” Sensors, Vol.20, No.24, Article No.7299, 2020.
  9. [9] W. Rawat and Z. Wang, “Deep convolutional neural networks for image classification: A comprehensive review,” Neural Computation, Vol.29, No.9, pp. 2352-2449, 2017.
  10. [10] J. Yu, M. Qin, and S. Zhou, “Dynamic gesture recognition based on 2D convolutional neural network and feature fusion,” Scientific Reports, Vol.12, No.1, Article No.4345, 2022.
  11. [11] J. Lin et al., “Adaptive local spatiotemporal features from RGB-D data for one-shot learning gesture recognition,” Sensors, Vol.16, No.12, Article No.2171, 2016.
  12. [12] Y. Liu et al., “Dynamic gesture recognition algorithm based on 3D convolutional neural network,” Computational Intelligence and Neuroscience, Vol.2021, Article No.4828102, 2021.
  13. [13] Y.-X. Wang et al., “Multitask touch gesture and emotion recognition using multiscale spatiotemporal convolutions with attention mechanism,” IEEE Sensors J., Vol.22, No.16, pp. 16190-16201, 2022.
  14. [14] M. Abavisani, H. R. V. Joze, and V. M. Patel, “Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training,” 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1165-1174, 2019.
  15. [15] A. Vaswani et al., “Attention is all you need,” Proc. of the 31st Int. Conf. on Neural Information Processing Systems (NIPS’17), pp. 6000-6010, 2017.
  16. [16] A. Galassi, M. Lippi, and P. Torroni, “Attention in natural language processing,” IEEE Trans. on Neural Networks and Learning Systems, Vol.32, No.10, pp. 4291-4308, 2021.
  17. [17] K. Han et al., “A survey on vision transformer,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.45, No.1, pp. 87-110, 2023.
  18. [18] W. Sun et al., “Regional time-series coding network and multi-view image generation network for short-time gait recognition,” Entropy, Vol.25, No.6, Article No.837, 2023.
  19. [19] Y. Liu et al., “A survey of visual transformers,” IEEE Trans. on Neural Networks and Learning Systems, 2023.
  20. [20] “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 9992-10002, 2021.
  21. [21] Z. Wu, C. Shen, and A. van den Hengel, “Wider or deeper: Revisiting the ResNet model for visual recognition,” Pattern Recognition, Vol.90, pp. 119-133, 2019.
  22. [22] Y.-H. Wu et al., “P2T: Pyramid pooling transformer for scene understanding,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 2022.
  23. [23] S. S. Beauchemin and J. L. Barron, “The computation of optical flow,” ACM Computing Surveys, Vol.27, No.3, pp. 433-466, 1995.
  24. [24] Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” Proc. of 16th European Conf. on Computer Vision (ECCV 2020), pp. 402-419, 2020.
  25. [25] J. Chung et al., “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv:1412.3555, 2014.
  26. [26] S. Jiang et al., “Learning to estimate hidden motions with global motion aggregation,” 2021 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 9752-9761, 2021.
  27. [27] Z. Rasheed and M. Shah, “Detection and representation of scenes in videos,” IEEE Trans. on Multimedia, Vol.7, No.6, pp. 1097-1105, 2005.
  28. [28] M. Mentzelopoulos and A. Psarrou, “Key-frame extraction algorithm using entropy difference,” Proc. of the 6th ACM SIGMM Int. Workshop on Multimedia Information Retrieval (MIR’04), pp. 39-45, 2004.
  29. [29] Y. Zhuang et al., “Adaptive key frame extraction using unsupervised clustering,” Proc. 1998 Int. Conf. on Image Processing (ICIP98), Vol.1, pp. 866-870, 1998.
  30. [30] C. Liu et al., “Dynamic-hand-gesture authentication dataset and benchmark,” IEEE Trans. on Information Forensics and Security, Vol.16, pp. 1550-1562, 2021.
  31. [31] W. Song et al., “TDS-Net: Towards fast dynamic random hand gesture authentication via temporal difference symbiotic neural network,” 2021 IEEE Int. Joint Conf. on Biometrics (IJCB), 2021.
  32. [32] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv: 2010.11929, 2020.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Nov. 24, 2023