In recent years, achieving sustainable development goals has required of personalized support systems that can adapt to diverse users through natural human–system interactions. Understanding the nonverbal information of users is essential for facilitating these interactions. In this study, we focus on human gestures as a fundamental nonverbal modality that conveys user emotions and intentions. Therefore, we propose a gesture analysis system based on the relative positions of human joints and arm orientations. Using an RGB-D camera, we acquire human skeletal information and develop a gesture recognition system. Within this system, two analytical approaches are investigated: dynamic time warping-based classification method and a neural network-based classification methods. Through experimental evaluation of a small-scale dataset, we analyze the behavioral characteristics and recognition tendencies of each approach under controlled conditions. In addition, we present several cases demonstrating the effectiveness of the proposed system and discuss its applicability.

1. [1] C. Wong, E. Yang, X.-T. Yan, and D. Gu, “An overview of robotics and autonomous systems for harsh environments,” 2017 23rd Int. Conf. on Automation and Computing (ICAC), 2017. https://doi.org/10.23919/IConAC.2017.8082020
2. [2] S. Yamamoto, J. Woo, W. H. Chin, K. Matsumura, and N. Kubota, “Interactive information support by robot partners based on informationally structured space,” J. Robot. Mechatron., Vol.32, No.1, pp. 236-243, 2020. https://doi.org/10.20965/jrm.2020.p0236
3. [3] R. Jahanmahin, S. Masoud, J. Rickli, and A. Djuric, “Human-robot interactions in manufacturing: A survey of human behavior modeling,” Robotics and Computer-Integrated Manufacturing, Vol.78, Article No.102404, 2022. https://doi.org/10.1016/j.rcim.2022.102404
4. [4] T. Kanda, M. Kamasima, M. Imai, T. Ono, D. Sakamoto, H. Ishiguro, and Y. Anzai, “A humanoid robot that pretends to listen to route guidance from a human,” Autonomous Robots, Vol.22, pp. 87-100, 2007. https://doi.org/10.1007/s10514-006-9007-6
5. [5] B. B. Lonkar, A. Kuthe, R. Shrivastava, and P. Charde, “Design and implement smart home appliances controller using IOT,” Information Systems and Management Science: Conf. Proc. of 3rd Int. Conf. on Information Systems and Management Science (ISMS) 2020, pp. 105-117, 2022. https://doi.org/10.1007/978-3-030-86223-7_11
6. [6] S. K. Roy, G. Singh, R. L. Gruner, B. L. Dey, S. Shabnam, S. S. Muhammad, and M. Quaddus, “Relationship quality in customer-service robot interactions in Industry 5.0: An analysis of value recipes,” Information Systems Frontiers, 2023. https://doi.org/10.1007/s10796-023-10445-y
7. [7] Y. Wang, H. Wu, G. Tian, G. Liu, F. Lu, and Y. Wang, “Fast search strategy for robots in dynamic home environment,” J. Adv. Comput. Intell. Intell. Inform., Vol.26, No.3, pp. 315-324, 2022. https://doi.org/10.20965/jaciii.2022.p0315
8. [8] P. Bull and J. P. Doody, “Gesture and body movement,” J. A. Hall and M. L. Knapp (Eds.), “Nonverbal Communication,” pp. 205-228, 2013. https://doi.org/10.1515/9783110238150.205
9. [9] S. Kita, “Why do people make gesture?,” Cognitive Studies, Vol.7, No.1, pp. 9-21, 2000 (in Japanese). https://doi.org/10.11225/jcss.7.9
10. [10] W.-F. Hsieh, E. Sato-Shimokawara, and T. Yamaguchi, “Investigation of robot expression style in human-robot interaction,” J. Robot. Mechatron., Vol.32, No.1, pp. 224-235, 2020. https://doi.org/10.20965/jrm.2020.p0224
11. [11] J. Woo, T. Sato, and Y. Ohyama, “Development of a human-centric system using an IoT-based socially embedded robot partner,” J. Robot. Mechatron., Vol.35, No.3, pp. 859-866, 2023. https://doi.org/10.20965/jrm.2023.p0859
12. [12] C. Matsumoto, Y. Sone, J. Woo, and Y. Ohyama, “Development of a gesture analysis system for human system interaction,” 2024 Joint 13th Int. Conf. on Soft Computing and Intelligent Systems and 25th Int. Symp. on Advanced Intelligent Systems (SCIS&ISIS), 2024. https://doi.org/10.1109/SCISISIS61014.2024.10760070
13. [13] Y. Sone, C. Matsumoto, J. Woo, and Y. Ohyama, “Development of a control support system for smart homes using the analysis of user interests based on mixed reality,” ROBOMECH J., Vol.12, Article No.2, 2025. https://doi.org/10.1186/s40648-024-00288-w
14. [14] J. Woo and J. Hu, “System for analyzing user interest based on eye gaze responses to enhance empathy with users,” J. Adv. Comput. Intell. Intell. Inform., Vol.29, No.3, pp. 641-648, 2025. https://doi.org/10.20965/jaciii.2025.p0641
15. [15] J. Woo and J. Hu, “System for analyzing user interest based on eye gaze responses to enhance empathy with users,” J. Adv. Comput. Intell. Intell. Inform., Vol.29, No.3, pp. 641-648, 2025. https://doi.org/10.20965/jaciii.2025.p0641
16. [16] Y. Sone and J. Woo, “Design of a human-centric robotic system for user support based on gaze information,” J. Adv. Comput. Intell. Intell. Inform., Vol.29, No.4, pp. 796-802, 2025. https://doi.org/10.20965/jaciii.2025.p0796
17. [17] M. Servi, A. Profili, R. Furferi, and Y. Volpe, “Comparative evaluation of Intel RealSense D415, D435i, D455 and Microsoft Azure Kinect DK Sensors for 3D vision applications,” IEEE Access, Vol.12, pp. 111311-111321, 2024. https://doi.org/10.1109/ACCESS.2024.3441238
18. [18] A. Hussain, H. Shakeel, F. Hussain, N. Uddin, and T. L. Ghouri, “Unity game development engine: A technical survey,” Univ. Sindh J. Inf. Commun. Technol., Vol.4, No.2, pp. 73-81, 2020.
19. [19] M. Tomasello, “Origins of Human Communication,” MIT Press, 2010.
20. [20] M. Müller, “Dynamic time warping,” Information Retrieval for Music and Motion, pp. 69-84, 2007. https://doi.org/10.1007/978-3-540-74048-3_4
21. [21] F. Petitjean, A. Ketterlin, and P. Gançarski, “A global averaging method for dynamic time warping, with applications to clustering,” Pattern Recognition, Vol.44, No.3, pp. 678-693, 2011. https://doi.org/10.1016/j.patcog.2010.09.013
22. [22] X. Huang, Y. Ye, L. Xiong, R. Y. Lau, N. Jiang, and S. Wang, “Time series k-means: A new k-means type smooth subspace clustering for time series data,” Information Sciences, Vol.367, pp. 1-13, 2016. https://doi.org/10.1016/j.ins.2016.05.040
23. [23] M. Cui et al., “Introduction to the k-means clustering algorithm based on the elbow method,” Accounting, Auditing and Finance, Vol.1, No.1, pp. 5-8, 2020. https://dx.doi.org/10.23977/accaf.2020.010102
24. [24] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee et al., “MediaPipe: A framework for building perception pipelines,” arXiv:1906.08172, 2019. https://doi.org/10.48550/arXiv.1906.08172
25. [25] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan, “uWave: Accelerometer-based personalized gesture recognition and its applications,” Pervasive and Mobile Computing, Vol.5, No.6, pp. 657-675, 2009. https://doi.org/10.1016/j.pmcj.2009.07.007
26. [26] X. Zhang, X. Chen, Y. Li, V. Lantz, K. Wang, and J. Yang, “A framework for hand gesture recognition based on accelerometer and emg sensors,” IEEE Trans. on Systems, Man, and Cybernetics – Part A: Systems and Humans, Vol.41, No.6, pp. 1064-1076, 2011. https://doi.org/10.1109/TSMCA.2011.2116004
27. [27] F. Farhadi-Niaki, J. Gerroir, A. Arya, S. A. Etemad, R. Laganiére, P. Payeur, and R. Biddle, “Usability study of static/dynamic gestures and haptic input as interfaces to 3D games,” Proc. of the 6th Int. Conf. on Advances in Computer-Human Interactions, pp. 315-323, 2013.
28. [28] Z. Tang, J. Ma, W. Liu, P. Zhang, and F. Lv, “Sliding window dynamic time-series warping-based ultrasonic guided wave temperature compensation and defect monitoring method for turnout rail foot,” IEEE Trans. on Ultrasonics, Ferroelectrics, and Frequency Control, Vol.69, No.9, pp. 2681-2695, 2022. https://doi.org/10.1109/TUFFC.2022.3195933
29. [29] J. Lee, T. Ahmed, T. Rikakis, and P. Turaga, “Automatic temporal segmentation for post-stroke rehabilitation: A keypoint detection and temporal segmentation approach for small datasets,” Proc. of the Winter Conf. on Applications of Computer Vision, pp. 21-29, 2025. https://doi.org/10.1109/WACVW65960.2025.00008
30. [30] H. Liu, A. Panahi, D. Andrews, and A. Nelson, “An fpga-based upper-limb rehabilitation device for gesture recognition and motion evaluation using multi-task recurrent neural networks,” IEEE Sensors J., Vol.22, No.4, pp. 3605-3615, 2022. https://doi.org/10.1109/JSEN.2022.3141659
31. [31] E. Rahimian, S. Zabihi, A. Asif, D. Farina, S. F. Atashzar, and A. Mohammadi, “Hand gesture recognition using temporal convolutions and attention mechanism,” 2022-2022 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 1196-1200, 2022. https://doi.org/10.1109/ICASSP43922.2022.9746174