Research Paper:
Home Appliance Operation via 3D Keypoint Based Gesture Detection in Body-Relative Command Spaces
Masae Yokota*,†, Soichiro Majima*, Yushin Mochizuki*, Sarthak Pathak**
, and Kazunori Umeda**
*Precision Engineering Course, Graduate School of Science and Engineering, Chuo University
1-13-27 Kasuga, Bunkyo-ku, Tokyo 112-8551, Japan
†Corresponding author
**Department of Precision Mechanics, Faculty of Science and Engineering, Chuo University
Tokyo, Japan
In this paper, we propose a flexible device control method using personalized command spaces that function as buttons on a virtual remote control that follows the user. By performing two different gestures in each space, the users can control various devices in a room. This system is implemented through multiple cameras and 3D human keypoint tracking. We experimentally evaluated the influence of command spaces arrangement on gesture recognition and determined the recognition accuracies for different gestures in each command space. The system demonstrated high usability, with even inexperienced users achieving high gesture recognition accuracy.
- [1] R. Faizrakhmanov, A. Platunov, and M. R. Bahrami, “Smart home user interface: Overview,” 2023 Int. Conf. on Industrial Engineering, Applications and Manufacturing (ICIEAM), pp. 595-600, 2023. https://doi.org/10.1109/ICIEAM57311.2023.10139050
- [2] H. Nakayama, N. Yabuki, H. Inoue, Y. Sumi, and T. Tsukutani, “A control system for electrical appliances using eye-gaze input,” 2012 Int. Symp. on Intelligent Signal Processing and Communications Systems, pp. 410-413, 2012. https://doi.org/10.1109/ISPACS.2012.6473521
- [3] S. Sugimura and K. Hoshino, “Wearable hand pose estimation for remote control of a robot on the moon,” J. Robot. Mechatron., Vol.29, No.5, pp. 829-837, 2017. https://doi.org/10.20965/jrm.2017.p0829
- [4] T. Nagayasu, H. Asano, K. Terabayashi, and K. Umeda, “Improvement of an intelligent room that detects hand waving motion for operation of home appliances,” SICE Annual Conf. 2011, pp. 821-826, 2011.
- [5] K. Irie, M. Takahashi, K. Terabayashi, H. Ogishima, and K. Umeda, “Skin color registration using recognition ofwaving hands,” J. Robot. Mechatron., Vol.22, No.3, pp. 262-272, 2010. https://doi.org/10.20965/jrm.2010.p0262
- [6] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” 2017 IEEE Conf. on Computer Vision and Pattern Recognition, pp. 7291-7299, 2017. https://doi.org/10.1109/CVPR.2017.143
- [7] M. Yokota, S. Majima, S. Pathak, and K. Umeda, “Intuitive arm-pointing based home-appliance control from multiple camera views,” 2023 32nd IEEE Int. Conf. on Robot and Human Interactive Communication (RO-MAN), pp. 179-184, 2023. https://doi.org/10.1109/RO-MAN57019.2023.10309557
- [8] T. Kano, T. Kawamura, H. Asano, T. Nagayasu, and K. Umeda, “Hand waving in command spaces: A framework for operating home appliances,” Advanced Robotics, Vol.32, No.18, pp. 999-1006, 2018. https://doi.org/10.1080/01691864.2018.1515661
- [9] S. Yan, Y. Ji, and K. Umeda, “A system for operating home appliances with hand positioning in a user-definable command space,” 2020 IEEE/SICE Int. Symp. on System Integration (SII), pp. 366-370, 2020. https://doi.org/10.1109/SII46433.2020.9025978
- [10] R. Hartley, “Multiple view geometry in computer vision,” Cambridge University Press, 2003.
- [11] Y. Mochizuki, M. Yokota, S. Pathak, and K. Umeda, “Visualisable and adjustable command spaces for gesture-based home appliance operation system via hololens2,” The 2025 IEEE/SICE Int. Symp. on System Integrations (SII), pp. 1405-1410, 2025. https://doi.org/10.1109/SII59315.2025.10870954
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.