Research Paper:
Greeting Gesture Classification Using Machine Learning Based on Politeness Perspective in Japan
Angga Wahyu Wibowo*, , Kurnianingsih** , Azhar Aulia Saputra* , Eri Sato-Shimokawara* , Yasufumi Takama* , and Naoyuki Kubota*
*Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan
Corresponding author
**Politeknik Negeri Semarang
Jl. Prof. H. Soedarto, SH Tembalang, Semarang, Indonesia
Understanding traditional culture is important. Various methods are used to achieve better cross-cultural understanding, and certain researchers have studied human behavior. However, behavior does not always represent a culture. Therefore, our study aims to understand Japanese greeting culture by classifying it through machine learning. Following are our study contributions. (1) The first study to analyze cultural differences in greeting gestures based on the politeness level of Japanese people by classifying them. (2) Classify Japanese greeting gestures eshaku, keirei, saikeirei, and waving hand. (3) Analyze the performance results of machine and deep learning. Our study noted that bowing and waving were the behaviors that could symbolize the culture in Japan. In conclusion, first, this is the first study to analyze the eshaku, keirei, saikeirei, and waving hand greeting gestures. Second, this study complements several human activity recognition studies that have been conducted but do not focus on behavior representing a culture. Third, according to our analysis, by using a small dataset, SVM and CNN methods provide better results than k-nearest neighbors (k-NN) with Euclidean distance, k-NN with DTW, logistic regression and LightGBM in classifying greeting gestures eshaku, keirei, saikeirei, and waving hand. In the future, we will investigate other behaviors from different perspectives using another method to understand cultural differences.
- [1] H. Gaspard, Y. Jiang, H. Piesch, B. Nagengast, N. Jia, J. Lee, and M. Bong, “Assessing Students’ Values and Costs in Three Countries: Gender and Age Differences Within Countries and Structural Differences Across Countries,” Learning and Individual Differences, Vol.79, Article No.101836, 2020. https://doi.org/10.1016/j.lindif.2020.101836
- [2] S. Zhao, D. Kirk, S. Bowen, D. Chatting, and P. Wright, “Supporting the Cross-Cultural Appreciation of Traditional Chinese Puppetry Through a Digital Gesture Library,” J. on Computing and Cultural Heritage, Vol.12, No.4, 2019. https://doi.org/10.1145/3341882
- [3] H. Wu, J. Gai, Y. Wang, J. Liu, J. Qiu, J. Wang, and X. Zhang, “Influence of Cultural Factors on Freehand Gesture Design,” Int. J. of Human-Computer Studies, Vol.143, Article No.102502, 2020. https://doi.org/10.1016/j.ijhcs.2020.102502
- [4] A. Y. Kwon, C. D. Vallotton, M. Kiegelmann, and K. H. Wilhelm, “Cultural Diversification of Communicative Gestures Through Early Childhood: A Comparison of Children in English-, German-, and Chinese-Speaking Families,” Infant Behavior and Development, Vol.50, pp. 328-339, 2018. https://doi.org/10.1016/j.infbeh.2017.10.003
- [5] E. Nicoladis, J. Nagpal, P. Marentette, and B.Hauer, “Gesture Frequency Is Linked to Story-Telling Style: Evidence from Bilinguals,” Language and Cognition, Vol.10, No.4, pp. 661-664, 2018. https://doi.org/10.1017/langcog.2018.25
- [6] D. McNeill, “Gesture in linguistics,” J. D. Wright (Ed.) “Int. Encyclopedia of the Social & Behavioral Sciences,” Elsevier, pp. 109-120, 2015. https://doi.org/10.1016/B978-0-08-097086-8.53050-5
- [7] R. Li, J. Lee, W. Woo, and T. Starner, “Kissglass: Greeting Gesture Recognition Using Smart Glasses,” Proc. of the Augmented Humans Int. Conf. (AHs’20), 2020. https://doi.org/10.1145/3384657.3384801
- [8] M. Bâce, S. Staal, G. Sörös, and G. Corbellini, “Collocated Multi-User Gestural Interactions with Unmodified Wearable Devices,” Augmented Human Research, Vol.2, No.1, Article No.6, 2017. https://doi.org/10.1007/s41133-017-0009-z
- [9] A. Melnyk and P. Hénaff, “Physical Analysis of Handshaking Between Humans: Mutual Synchronisation and Social Context,” Int. J. of Social Robotics, Vol.11, No.4, pp. 541-554, 2019. https://doi.org/10.1007/s12369-019-00525-y
- [10] S. Aina, K. V. Sholesi, A. R. Lawal, S. D. Okegbile, and A. I. Oluwaranti, “Gesture Recognition System for Nigerian Tribal Greeting Postures Using Support Vector Machine,” Malaysian J. of Computing, Vol.5, No.2, Article No.609, 2020. https://doi.org/10.24191/mjoc.v5i2.10347
- [11] G. Trovato, M. Zecca, M. Do, Ö. Terlemez, M. Kuramochi, A. Waibel, T. Asfour, and A. Takanishi, “A novel greeting selection system for a culture-adaptive humanoid robot,” Int. J. of Advanced Robotic Systems, Vol.12, Issue 4, 2015. https://doi.org/10.5772/60117
- [12] T. Osugi and J. I. Kawahara, “The Spill-Over Effect of Formal Bowing Motion on Subjective Facial Attractiveness” Japanese Psychological Research, Vol.65, No.1, pp. 37-47, 2021. https://doi.org/10.1111/jpr.12347
- [13] M. Amri, “Ojigi: The Ethics of Japanese Community’s Nonverbal Language,” Proc. of the Social Sciences, Humanities; Education Conf. (SoSHEC 2019), 2019. https://doi.org/10.2991/soshec-19.2019.9
- [14] G. Gusnawaty, L. Lukman, A. Nurwati, A. Adha, N. Nurhawara, and A. Edy, “Strategy of Kinship Terms as a Politeness Model in Maintaining Social Interaction: Local Values Towards Global Harmony,” Heliyon, Vol.8, No.9, Article No.e10650, 2022. https://doi.org/10.1016/j.heliyon.2022.e10650
- [15] Z. Ye, “The Politeness Bias and the Society of Strangers,” Language Sciences, Vol.76, Article No.101183, 2019. https://doi.org/10.1016/j.langsci.2018.06.009
- [16] P. G.-C. Blitvich and M. Sifianou, “IM/politeness and Discursive Pragmatics,” J. of Pragmatics, Vol.145, pp. 91-101, 2019. https://doi.org/10.1016/j.pragma.2019.03.015
- [17] M. Hendon, L. Powell, and H. Wimmer, “Emotional Intelligence and Communication Levels in Information Technology Professionals,” Computers in Human Behavior, Vol.71, pp. 165-171, 2017. https://doi.org/10.1016/j.chb.2017.01.048
- [18] P. Xie and L. Deng, “Simulated analysis of modeling of driving behavior characteristics based on satellite positioning data,” J. Adv. Comput. Intell. Intell. Inform., Vol.23, No.1, pp. 114-118, 2019. https://doi.org/10.20965/jaciii.2019.p0114
- [19] C. Hofmann, C. Patschkowski, B. Haefner, and G. Lanza, “Machine Learning Based Activity Recognition to Identify Wasteful Activities in Production,” Procedia Manufacturing, Vol.45, pp. 171-176, 2020. https://doi.org/10.1016/j.promfg.2020.04.090
- [20] D. Katagami, Y. Ikeda, and K. Nitta, “Behavior Generation and Evaluation of Negotiation Agent Based on Negotiation Dialogue Instances,” J. Adv. Comput. Intell. Intell. Inform., Vol.14, No.7, pp. 840-851, 2010. https://doi.org/10.20965/jaciii.2010.p0840
- [21] Q. Xu, W. Zheng, Y. Song, C. Zhang, X. Yuan, and Y. Li, “Scene Image and Human Skeleton-Based Dual-Stream Human Action Recognition,” Pattern Recognition Letters, Vol.148, pp. 136-145, 2021. https://doi.org/10.1016/j.patrec.2021.06.003
- [22] Y. Fuse, H. Takenouchi, and M. Tokumaru, “A Robot in a Human–Robot Group Learns Group Norms and Makes Decisions Through Indirect Mutual Interaction With Humans,” J. Adv. Comput. Intell. Intell. Inform., Vol.24, No.1, pp. 169-178, 2020. https://doi.org/10.20965/jaciii.2020.p0169
- [23] Y. Li, W. F. Hsieh, E. Sato-Shimokawara, and T. Yamaguchi, “Expression and Identification of Confidence Based on Individual Verbal and Non-Verbal Features in Human-Robot Interaction,” J. Adv. Comput. Intell. Intell. Inform., Vol.23, No.6, pp. 1089-1097, 2019. https://doi.org/10.20965/jaciii.2019.p1089
- [24] K. Ohkura, T. Yasuda, and Y. Matsumura, “Generating Cooperative Collective Behavior in Swarm Robotic Systems,” J. Adv. Comput. Intell. Intell. Inform., Vol.17, No.5, pp. 699-706, 2013. https://doi.org/10.20965/jaciii.2013.p0699
- [25] S. Hoshino and K. Niimura, “Optical Flow for Real-Time Human Detection and Action Recognition Based on CNN Classifiers,” J. Adv. Comput. Intell. Intell. Inform., Vol.23, No.4, pp. 735-742, 2019. https://doi.org/10.20965/jaciii.2019.p0735
- [26] Z. Chen, X. Ma, Z. Peng, Y. Zhou, M. Yao, Z. Ma, C. Wang, Z. Gao, and M. Shen, “User-Defined Gestures for Gestural Interaction: Extending from Hands to Other Body Parts,” Int. J. of Human-Computer Interaction, Vol.34, No.3, pp. 238-250, 2018. https://doi.org/10.1080/10447318.2017.1342943
- [27] S. Shao, N. Kubota, K. Hotta, and T. Sawayama, “Behavior Estimation Based on Multiple Vibration Sensors for Elderly Monitoring Systems,” J. Adv. Comput. Intell. Intell. Inform., Vol.25, No.4, pp. 489-497, 2021. https://doi.org/10.20965/jaciii.2021.p0489
- [28] S. Shao and N. Kubota, “A Fuzzy Inference-Based Spiking Neural Network for Behavior Estimation in Elderly Health Care System,” J. Adv. Comput. Intell. Intell. Inform., Vol.23, No.3, pp. 528-535, 2019. https://doi.org/10.20965/jaciii.2019.p0528.
- [29] H. Igarashi, Y. Adachi, and K. Takahashi, “Adaptive Cooperation for Multi Agent Systems Based on Human Social Behavior,” J. Adv. Comput. Intell. Intell. Inform., Vol.16, No.1, pp. 139-146, 2012. https://doi.org/10.20965/jaciii.2012.p0139
- [30] P. Li, Q. Fei, Z. Chen, X. Yao, and Y. Zhang, “Characteristic Behavior of Human Multi-Joint Spatial Trajectory in Slalom Skiing,” J. Adv. Comput. Intell. Intell. Inform., Vol.26, No.5, pp. 801-807, 2022. https://doi.org/10.20965/jaciii.2022.p0801
- [31] K. Zhang, Y. Maeda, and Y. Takahashi, “Cooperative Behavior Learning Based on Social Interaction of State Conversion and Reward Exchange Among Multi-Agents,” J. Adv. Comput. Intell. Intell. Inform., Vol.15, No.5, pp. 606-616, 2011. https://doi.org/10.20965/jaciii.2011.p0606
- [32] K. Zhang, Y. Maeda, and Y. Takahashi, “Group Behavior Learning in Multi-Agent Systems Based on Social Interaction Among Agents,” J. Adv. Comput. Intell. Intell. Inform., Vol.15, No.7, pp. 896-903, 2011. https://doi.org/10.20965/jaciii.2011.p0896
- [33] K. Sakai, F. D. Libera, Y. Yoshikawa, and H. Ishiguro, “Generation of Bystander Robot Actions Based on Analysis of Relative Probability of Human Actions,” J. Adv. Comput. Intell. Intell. Inform., Vol.21, No.4, pp. 686-696, 2017. https://doi.org/10.20965/jaciii.2017.p0686
- [34] N. Kubota, T. Obo, and H. Liu, “Human Behavior Measurement Based on Sensor Network and Robot Partners,” J. Adv. Comput. Intell. Intell. Inform., Vol.14, No.3, pp. 309-315, 2010. https://doi.org/10.20965/jaciii.2010.p0309
- [35] Z. Benhaili, Y. Balouki, and L. Moumoun, “A Hybrid Deep Neural Network for Human Activity Recognition Based on IOT Sensors,” Int. J. of Advanced Computer Science and Applications, Vol.12, No.11, 2021. https://doi.org/10.14569/IJACSA.2021.0121129
- [36] P. Tokas, “Machine Learning Based Text Neck Syndrome Detection Using Microsoft Kinect Sensor,” Materials Today: Proc., Vol.80, pp. 3751-3756, 2023. https://doi.org/10.1016/j.matpr.2021.07.373
- [37] Y. J. Luwe, C. P. Lee, and K. M. Lim, “Wearable sensor-based human activity recognition with ensemble learning: A Comparison Study,” Int. J. of Electrical and Computer Engineering (IJECE), Vol.13, No.4, Article No.4029, 2023. https://doi.org/10.11591/ijece.v13i4.pp4029-4040
- [38] H. Los, G. S. Mendes, D. Cordeiro, N. Grosso, H. Costa, P. Benevides, and M. Caetano, “Evaluation of Xgboost and LGBM Performance in Tree Species Classification with Sentinel-2 Data,” 2021 IEEE Int. Geoscience and Remote Sensing Symp. (IGARSS), pp. 5803-5806, 2021. https://doi.org/10.1109/IGARSS47720.2021.9553031
- [39] D. K. Basuki, A. A. Saputra, N. Kubota, and K. Wada, “Joint Angle-Based Activity Recognition System for Paro Therapy Observation,” IFAC-PapersOnLine, Vol.56, No.2, pp. 1145-1151, 2023. https://doi.org/10.1016/j.ifacol.2023.10.1718
- [40] Ö. F. İnce, I. F. Ince, M. E. Yıldırım,, J. S. Park, J. K. Song, and B. W. Yoon, “Human Activity Recognition with Analysis of Angles Between Skeletal Joints Using a RGB-Depth Sensor,” ETRI J., Vol.42, No.1, pp. 78-89, 2020. https://doi.org/10.4218/etrij.2018-0577
- [41] J. Zhou and T. Komuro, “An Asymmetrical-Structure Auto-Encoder for Unsupervised Representation Learning of Skeleton Sequences,” Computer Vision and Image Understanding, Vol.222, Article No.103491, 2022. https://doi.org/10.1016/j.cviu.2022.103491
- [42] Y.-H. Lee, C.-P. Wei, T.-H. Cheng, and C.-T. Yang, “Nearest-Neighbor-Based Approach to Time-Series Classification,” Decision Support Systems, Vol.53, No.1, pp. 207-217, 2012. https://doi.org/10.1016/j.dss.2011.12.014
- [43] Z. Geler, V. Kurbalija, M. Ivanovic, and M. Radovanovic, “Time-Series Classification with Constrained DTW Distance and Inverse-Square Weighted k-NN,” 2020 Int. Conf. on Innovations in Intelligent Systems and Applications (INISTA), 2020. https://doi.org/10.1109/INISTA49547.2020.9194639
- [44] S. Ghodsi, H. Mohammadzade, and E. Korki, “Simultaneous Joint and Object Trajectory Templates for Human Activity Recognition from 3-D data,” J. of Visual Communication and Image Representation, Vol.55, pp. 729-741, 2018. https://doi.org/10.1016/j.jvcir.2018.08.001
- [45] L. Lo Presti and M. La Cascia, “3D Skeleton-Based Human Action Classification: A Survey,” Pattern Recognition, Vol.53, pp. 130-147, 2016. https://doi.org/10.1016/j.patcog.2015.11.019
- [46] M. R. Widyanto, S. N. Endah, and K. Hirota, “Human Behavior Classification Using Thinning Algorithm and Support Vector Machine,” J. Adv. Comput. Intell. Intell. Inform., Vol.14, No.1, pp. 28-33, 2010. https://doi.org/10.20965/jaciii.2010.p0028
- [47] A. A. Saputra, A. R. Besari, and N. Kubota, “Human joint skeleton tracking using multiple kinect azure,” 2022 Int. Electronics Symp. (IES), pp. 430-435, 2022. https://doi.org/10.1109/IES55876.2022.9888532
- [48] P. C. Huu, L. Q. Khanh, and L. T. Ha, “Human Action Recognition Using Dynamic Time Warping and Voting Algorithm (1),” VNU J. of Science: Comp. Science & Com. Eng., Vol.30, Issue 3, 2014.
- [49] S. K. Yadav, K. Tiwari, H. M. Pandey, and S. A. Akbar, “Skeleton-Based Human Activity Recognition Using CONVLSTM and Guided Feature Learning,” Soft Computing, Vol.26, No.2, pp. 877-890, 2022. https://doi.org/10.1007/s00500-021-06238-7
- [50] J. K. Aggarwal and L. Xia, “Human Activity Recognition from 3D Data: A Review,” Pattern Recognition Letters, Vol.48, pp. 70-80, 2014. https://doi.org/10.1016/j.patrec.2014.04.011
- [51] S. Gaglio, G. L. Re, and M. Morana, “Human Activity Recognition Process Using 3-D Posture Data,” IEEE Trans. on Human-Machine Systems, Vol.45, No.5, pp. 586-597, 2015. https://doi.org/10.1109/THMS.2014.2377111
- [52] E. Cippitelli, S. Gasparrini, E. Gambi, and S. Spinsante, “A Human Activity Recognition System Using Skeleton Data from RGBD Sensors,” Computational Intelligence and Neuroscience, Vol.2016, 2016. https://doi.org/10.1155/2016/4351435
- [53] G. Hu, B. Cui, and S. Yu, “Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition,” IEEE Trans. on Multimedia, Vol.22, No.9, pp. 2207-2220, 2020. https://doi.org/10.1109/TMM.2019.2953325
- [54] B. M. V. Guerra, S. Ramat, R. Gandolfi, G. Beltrami, and M. Schmid, “Skeleton Data Pre-Processing for Human Pose Recognition Using Neural Network,” 2020 42nd Annual Int. Conf. of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4265-4268, 2020. https://doi.org/10.1109/EMBC44109.2020.9175588
- [55] G. Hu, B. Cui, and S. Yu, “Skeleton-Based Action Recognition with Synchronous Local and Non-Local Spatio-Temporal Learning and Frequency Attention,” 2019 IEEE Int. Conf. on Multimedia and Expo (ICME), pp. 1216-1221, 2019. https://doi.org/10.1109/ICME.2019.00212
- [56] I. Rodríguez-Moreno, J. M. Martínez-Otzeta, I. Goienetxea, I. Rodriguez-Rodriguez, and B. Sierra, “Shedding Light on People Action Recognition in Social Robotics by Means of Common Spatial Patterns,” Sensors, Vol.20, No.8, Article No.2436, 2020. https://doi.org/10.3390/s20082436
- [57] L.-Y. Hu, M.-W. Huang, S.-W. Ke, and C.-F. Tsai, “The Distance Function Effect on k-Nearest Neighbor Classification for Medical Datasets,” SpringerPlus, Vol.5, No.1, Article No.1304, 2016. https://doi.org/10.1186/s40064-016-2941-7
- [58] S. Celebi, T. T. Temiz, A. S. Aydin, and T. Arici, “Gesture Recognition Using Skeleton Data with Weighted Dynamic Time Warping,” Proc. of the Int. Conf. on Computer Vision Theory and Applications, 2013. https://doi.org/10.5220/0004217606200625
- [59] H. Basly, W. Ouarda, F. E. Sayadi, B. Ouni, and A. M. Alimi, “CNN-SVM Learning Approach Based Human Activity Recognition,” Lecture Notes in Computer Science, pp. 271-281, 2020. https://doi.org/10.1007/978-3-030-51935-3_29
- [60] H. Qian, Y. Mao, W. Xiang, and Z. Wang, “Recognition of Human Activities Using SVM Multi-Class Classifier,” Pattern Recognition Letters, Vol.31, No.2, pp. 100-111, 2010. https://doi.org/10.1016/j.patrec.2009.09.019
- [61] C. Schuldt, I. Laptev, and B. Caputo, “Recognizing Human Actions: A Local SVM Spproach,” Proc. of the 17th Int. Conf. on Pattern Recognition (ICPR 2004), Vol.3, pp. 32-36, 2004. https://doi.org/10.1109/ICPR.2004.1334462
- [62] Y. Jung, “Multiple Predictingk-Fold Cross-Validation for Model Selection,” J. of Nonparametric Statistics, Vol.30, No.1, pp. 197-215, 2018. https://doi.org/10.1080/10485252.2017.1404598
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.